Current & relevant configuration:

• Standard Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)
• Microsoft SQL Server 2005 – 9.00.4207.00 (X64) = SP3 CU1; latest MS build is 9.00.4305 SP3 CU10
• 31.9GB physical RAM, 4092MB page file on c:; windows recommends 49,143MB; my normal recommendation is 2.5x physical = 81,920MB
• SQL Server configured for max memory 27,000MB ~= 26.36GB; not to use AWE; “Lock Pages in Memory” is disabled as x64/default

 Train of events:

• last logged normal operation which was a transaction log backup as per schedule
• Log message “A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 0 seconds. Working set (KB): <size>, committed (KB): <size>, memory utilization: 0%.”
• server rebooted
• server starting services
• SQL recovery complete – server ready to accept and process connections; service resumed
• SQL reports database mirroring is disconnected
  • Error: 9642, Severity: 16, State: 3.
  • An error occurred in a Service Broker/Database Mirroring transport connection endpoint, Error: 8474, State: 11. (Near endpoint role: Target, far endpoint address: ”)
• MSSQL restart resolves mirroring disconnect – details attached
• Full DR resumed

 Investigations into root cause of stop:

• Given the current configuration of page file it is understandable the server ran out of memory.
• Research (ref1, ref2 & ref3) into the memory page out error suggests the server basically ran out of memory
  • ref1 correlates that other people have had stops due to the same cause; refers ref2
  • ref2 gives background regarding memory working sets for SQL and concurs that a SQL server is simply not just the memory for sqlservr.exe
  • ref3 (KB918483) explains how to prevent the OS operations that lead to the stop
    • This article then refers a blog post (ref4) form the CSS engineers who have given background on memory trim’s; there are three (3) types of trim:
      • Signalled trim – does not generate our error message
      • Self trim – the OS requires additional RAM for pages not in memory, the “amount to trim can be ~1/4 the current working set size” – therefore we don’t have a page file large enough for this trim
      • Hard trim – system memory is very low and the “trim can be the entire working set” – again, we do not have a large enough page file
      • Therefore it is essential we increase the page file size.
    • ref4 gives several signals as to the cause of the memory trim. Those of immediate interest are as follows:
      • system file cache size (ref5)
      • “TCP Chimney” with Broadcom network adapters
    • ref5 discusses the performance counters that indicate whether there is an underlying OS issue with system file cache size

 Next Steps:

• Increase page file size; 2.5x is my own blanket rule of thumb, but given the drive capacities we may had to increase to a lower value initially and monitor:
  • A trusted source of information (ref6 and ref7) has the following guidelines for provisioning the page file size:
    • Based on the “Page File Bytes Peak” value (3.10E+10 bytes) monitored on and the 70% rule indicates a page file requirement of 20.1GB.
• Check Broadcom Network Adapter drivers are latest
• PerfMon and monitor the system file cache size issue

So, where did I start: credit is due to this post for the 90 seconds turn around: Why This SQL Server DBA is Learning Powershell. So, rather than “Hello World”, my first script this time is as follows:

# Execute, on servers in $args[0], script in $args[1]

$dt = new-object System.Data.DataTable

foreach ($svr in get-content $args[0])
{
  $con = "server=$svr;database=master;Integrated Security=sspi"

  $cmd = get-content $args[1]

  $da = new-object System.Data.SqlClient.SqlDataAdapter ($cmd, $con)
  $da.fill($dt) | out-null
  $svr
}

# $dt | Format-Table -autosize

$output_csv = $args[1] + ".csv"
$dt | export-csv -path $output_csv –noType

I’ll post further later, just wanted to get something out…which then means I will be motivated to blog/post further!

Since the whitsundays there was the Lego christmas do, NYE in Melbourne and the start of the Sydney to Hobart race, then we decided to put this years photo’s in a annual album rather than under the World Trip which include Kaz’s 30th on the hawkesbury, Good Vibrations and Easter!

• how to query a mail server (MTA) from telnet
• sample batch scripts
• serialize the XML dll for CLR sprocs

and a app store for winmo that only serves free apps!

From reading on how best to do this, I am starting with method 2.

The course was amazing. Steve was the instructor/skipper and we were also aboard with Adam (who was preparing for his yachtmasters) and Blake (who was taking his day skipper). The five of us had a great time sailing around one of the most amazing sailing spots in the world – both day and NIGHT!

Once we completed our course, with knowledge on on to sail, eight knots and a desperate need for a proper shower, we headed to Hamiliton Island to meet the gang. Well, the place was stunning and the shower amazing! Em had booked a sailing excursion for us all so we eventually got to swim in the green waters, snokkled on a reef, swam with a turtle then played cricket on Whitehaven beach, messages below:

We all then returned to Sydney for a long weekend of sighseeing for Dan, meeting the family (Towells & Lyon) and as usual plenty of drinking and eating for us all! All in all, a wicked holiday which also served as half my notice as I have a new job at Yahoo!7

Not to mention a stroll at the seaside (bondi) and ice-cream at coogee! Although it was a fleeting visit it was very enjoyable

The weather once we got back was amazing and we settled in for a long, long summer, had a couple of weekends on the beach, played plenty of golf and restarted midweek supper club! There are a serious number of restaurants just around our suburb so we have taken to task at getting around them and true to form we are progressing nicely!

Couple of weeks ago we headed to Melbourne for a long weekend. We left Sydney in beautiful 30 degree sunshine and turned up to a brisk 11 degrees Melbourne! The weekend was awesome, as usual plenty of top quality food, flowing wine and great times! It was AFL Grand Final weekend in Melbourne, given tickets were selling in excess of $1000 each we opted for a bar in St Kilda, the home suburb of one of the finalists! Unfortunately St Kilda didn’t win, but the atmosphere and fun were top notch!

Back to Sydney we have had a few weeks of rain which, given the recent dust storm, have been welcome by most so Karen will have to wait a couple of weeks for a return to the beach!  My birthday fell midweek so we opted for supper club on my actual birthday (probably one of the top three restaurant experiences to-date in Oz) and celebrations with all on the Sunday of a long weekend. The Sunday was then the turn of the rugby league grand final which resulted in a day long drinking accumulating in everyone back to ours! I don’t think the neighbours were too impressed given we were complaining the music wasn’t loud enough at 2am!!!

Anyway, wedding plans are coming along nicely – stag & hen plans are in the air and the finer detail for the wedding being discussed.