Wednesday, October 30, 2013

Intro to using the Windows Debugger

Windows Debugger can be downloaded as part of the Windows SDK. I would install both the 32 and 64bit versions.

I used it in the past when diagnosing why a website was performing poorly during peak periods. Users would hit the website url, wait for 30+ seconds and then the site would appear quickly. Even the home page which did nothing (i.e. no db hits), was slow to appear.

I got a memory dump from the server and using the analysis approach below could see that all available IIS worker threads were in use and all were busy rendering. The server was starved of threads and user requests were ending up in a queue waiting to be processed.

I tweaked the IIS settings and added more worker threads. The problem went away. Thanks Windows Debugger!

How do you to get a memory dump?

1. Open a command prompt
2. Get the list of worker process ids:
  • %systemroot%\system32\inetsrv\APPCMD list wps
    • Your worker process should be in the list
3. Change to the Debugging Tools directory. On my machine that was at:
  • cd C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x86 4. Create a directory for the memory dump. I went with c:\dumps
5. Run the following, substituting your process id for xxxx:
  • ADPlus -Hang -p xxxx -quiet -o c:\dumps. You'll end up with a .dmp file in the c:\dumps folder.

Opening the memory dump in WinDBG

1. Run WinDBG. You'll see something similar to the following:

2. Goto File -> Open Crash Dump. You should then see something similar to the following:

3. Set the symbol path:
  • .sympath srv*c:\Symbols*
  • .symfix
  • .reload 

4. Load the .Net CLR
  • .Net 3.5: .loadby sos mscorwks
  • .. or for .Net 4.0: .loadby sos clr

Working out what each of the threads are doing

1. Get the list of threads in the memory dump
  • !Threads -special 

2. Switch to a ThreadPoolWorker thread, e.g. thread 11:
  • ~11s 

3. Get the stacktrace for this thread
  • !clrstack 

Using the stacktrace, you can then see what the thread is doing. Switch then to a different thread to see what it is doing. Using this approach I could work out that all threads were busy doing normal activities and that the website just needed more worker threads.

MVC HTTP Headers for CDN caching

On a recent project I added upgraded our systems to use a Content Delivery Network (CDN). The project supplied content to users and we wanted to offload the 'heavy-lifting' of the content supply away from our PROD server onto a more robust delivery network.

The mobile clients would request the content from the CDN. If the CDN hadn't cached the content yet, it would grab it from our PROD server, cache it, and respond back to the clients. For this caching to work, though, some HTTP Header magic was required.

Which headers should you watch out for?

  • Cache control should be public
    • For the CDN to cache the content on behalf of your origin server, set the Cache-control=public
  • Expires Date
    • To stop the CDN regularly requested the same static content from your PROD server, set the Expires-Date setting to be in the far future (e.g. > 1 year)
  •  Vary 
    • Watch out for the 'Vary' header. MVC3 had initially defaulted to adding the "vary=*" http header which stopped the CDN from caching the data. 

A useful RFC to refer to is: RFC 2616 13.6:
A Vary header field-value of * always fails to match and subsequent requests on that resource can only be properly interpreted by the origin server.

.. or if you prefer the MSDN page:

If you set Vary="NONE", this gets outputted as "Vary=*", which isn't very useful. Either get rid of the Vary header or set it to something broad such as "Vary: Accept-Encoding"

How to implement this in MVC?

[OutputCache(Location = OutputCacheLocation.Any, Duration = 31536000, VaryByParam = "")]
public ActionResult GetMyCDNContent(string someSortOfIdenfier)
OutputCacheLocation.Any will be outputted as: Cache-Control: public
Duration = 31536000 will be outputted as:
  • Cache-Control: max-age=31536000
  • Expires: {a date a year from today}
  • VaryByParam="" will not output any Vary= header
  • This is what we needed; the content was going to be the same for all clients.

Here's how it ends up looking in Fiddler:

You might want something different if you have alternate caching needs for different encodings, such as support for gzip and non-gzip caching:
[OutputCache(Location = OutputCacheLocation.Any, Duration = 31536000, VaryByParam = "", VaryByHeader = "Accept-Encoding" )]
public ActionResult GetMyCDNContent(string someSortOfIdenfier)

Here's how it ends up looking in Fiddler: