How can I debug a memory leak in a service

Question

I have a windows service program that is running on one of our servers. It has a memory leak of some kind within the program. When it first starts running it is using 20,000k memory. Within a few days it is up to 400,000k of memory. It is a fairly simple program that tracks running processes and writes the information to a database.

I am unsure on how I can debug a program running on a remote machine to find out where the memory leak is coming from. What methods would you suggest I use in order to try and track this down?

Update
The program has a lot of exception handling in place already. However, it has no logging whatsoever. What information should I be logging in order to track down the memory leak?

Answer 1

There are a number of ways that come to mind:

• Run the service locally, but not as a service. Create your own lightweight host environment such as a simple console app. You can now attach memory profiling tools (the one built into certain versions of Visual Studio, etc).
• You can do Core Dumps when the memory is large. This is non-trivial and you'll need some tools from the Windows SDK.
• Do you have any logging in place? If so, you could compare what's happening in your logs against some PerfMon data.

Answer 2

Services that run 24x7 are very hard to debug for memory leaks (and also for memory corruption).

There are two lines of attack to achieve memory leak free services:

A. Debug to fight it out

1. Some tools that can readily provide insight in this are Valgraind, IBM Rational Purify, memwatch and there are many more so on. Here is a good article that talks about such tools: http://www.linuxjournal.com/article/6556

2. Run individual modules in a long run; this is kind of unit test but not mock methods. Run a rather meaningful application that uses minimum units first, once you establish that component A is leak free, add in the application B and then C and so on. This will allow isolated

3. Run different type of loads - first without any meaning full work then other specific type of transaction/work items and so on. Basically try to identify different potential path - basically isolate different code coverage lines that identifies

4. Log all memory allocations (this is hard to analyse) whether it is due to unnecessary allocations, or forgotten deletions.

5. Focus on running data - for example packets in network stack or, items generated per transactions, items generated (or forget to delete) per unit of work. What you should do is that identify exact memory usage - and track it per transaction by transaction. And you can again check if certain type of work produces noticeable leaks against otherwise.

6. Track all small structures, string carefully; mostly such leaks are better caught by code reviews than by tracking memory usage because they are kind of very minor leaks.

B. Design to prevent system

1. Use local memory as much as possible (basically don't try to keep things heap - but not if there is a perform penalty).

2. Define clear allocation and deallocation responsibility. As a general rule, the application must allocate and consequently de-allocate and library doesn't have any responsibility in some case.

3. classify running data structures vs. infra. For example basic objects that are expected to run across the full life cycle vs. objects that gets generated and dies with every transaction/operation etc. The first class (infra) must be all allocated before practical operation starts and during the run time, only work specific data structures allocated; i know that is not always possible or must - but it helps to see that once all infra element are allocated, the memory should be stable

4. Create system design as modular as possible. Every module should have a very simple abstract of data exchange and/or object creation deletion

5. Clean up memory on exception. Not removing memory on failed allocations or creators is one of the difficult thing to fight,because everything works for 99% of time, but after 5 days you see some leaks which has no explanation!

6. Life pools - one of the beauty of Apache web server is that you don't do any free! What it does is that all infra objects (such as core objects, plug-ins etc.) are created a prior. If there is leak in every transaction, over time it will kill memory. What Apache does, is that it creates different life pool - per transaction, medium term and long term pools. Crux is, for the per transaction pool you can just keep alocating. Once transaction is over -the whole bunch is freed at once. Read this reference that gives you details on how this work. http://www.fmc-modeling.org/category/projects/apache/amp/4_6Memory_resource.html

7. Similarly you can also design system (or parts) with static pools. However, this might be difficult for many systems. But such a thing is quite common in embedded systems. see here for details: http://wyw.dcweb.cn/static_mem_pool.htm

Finally, Remember nothing is 100% leak free!

Answer 3

The way I've approached this problem was to look for threads and objects that weren't being properly disposed/released. This may be due to odd programming errors or unexpected data causing processes to lock up, network device problems and so forth. The problem is that these can sometimes be hard to detect or duplicate when you're running outside of the production environment.

To help track down these problems, provided test cases and/or memory profiling on your development systems don't reveal them right off, you'll need to have good logging and exception handling in place. Watch especially for situations where a thread/object sticks around longer than it should.

Answer 4

You can use Visual Leak Detector on Windows to check for memory leaks.

Answer 5

I find cppcheck a very helpful tool as well. It performs static code analysis and can give you hints, where memory leaks might occurr.

EDIT: Did not realize the question was targeting .Net, so my advice may be not so helpful

Answer 6

It's hard to debug in production try running locally. I use Debug Viewer in conjunction with a profiler.

I would recommend ANTS, not sure if it's free right now. I used a 30 day demo version and it will give a detailed analysis when you run the application with list of susceptible parts which you can actually break down into for finding out the specific cause. I think there are plenty of free profilers available too. I found it very difficult to debug WPF leaks than conventional winform leaks.

Hope it ain't a WPF. Good luck though.