Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm thinking of how we do JVM memory monitor in a low overhead way in production environment even under busy hour.

Suppose I have two tomcat app server in production, load balance set up behind them. If I can see the jvm memory statistics I can tell load balance to stop sending the request to the server which will encounter OOM issue. Do this make sense? Jconsole or VisualVM eats more performance resourse is not my choice.

share|improve this question
    
The Java Simon framework might be a look worth. –  khmarbaise Apr 19 '12 at 15:26
    
possible duplicate - stackoverflow.com/questions/242958/best-tools-to-monitor-tomcat –  jasonk Apr 19 '12 at 22:20
add comment

4 Answers

Others have provided suggestions on how to monitor memory usage ...

Suppose I have two tomcat app server in production, load balance set up behind them. If I can see the jvm memory statistics I can tell load balance to stop sending the request to the server which will encounter OOM issue. Do this make sense?

Sort of. But it isn't necessarily the best way to solve your problem.

Lets backtrack to the root of the problem ... the OOMEs. In the context of Tomcat, OOMEs are likely to be caused by one of the following:

  • memory leaks in your application, (or possibly Tomcat itself),
  • attempting to process too many requests in parallel on each Tomcat, or
  • individual requests that need too much memory during processing.

To solve your problem, you first need to find out which of these is happening ... because the solution is different for each of them.

1) To see if this is a memory leak, you need to use a memory analysis tool to examine the long-term memory usage patterns. This will probably show a sawtooth pattern ... which is normal. What you need to look for is the the level of the bottoms of the "teeth" trending upwards over time. That indicates that something is creating garbage that cannot be collected; i.e. a memory leak.

If you have a memory leak, then the best solution is to figure out what part of your code is responsible, and fix it. Anything else ... including load balancing ... is a bandaid solution and could lead to worse problems down the track.

2) Having eliminated memory leaks, you need to figure out if the problem is that you are processing too many requests at once. I'm not sure of the best way to do that, but if this is the problem (or you suspect it is) then there are a few possible solutions:

  • Adjust the Tomcat server configuration to reduce the number of worker threads.

  • If your requests are I/O bound, then another possibility would be to look at the asynchronous request handling support available in recent versions of the Servlet spec - see http://docs.oracle.com/javaee/7/tutorial/doc/servlets012.htm. But that will be more work.

3) If the problem turns out to be that certain requests are using too much memory, then you need to figure out how to detect those requests before hand and "deal with them". Both detecting and dealing with these requests could be difficult ... and it is hard to advise without details of your application. But a couple of pragmatic solutions are:

  • Forward the anomalous requests to another server with a big heap ... where the OOMEs won't interfere with "normal" requests.

  • Increase the heap size. If you have enough physical memory, running with a larger heap could actually make your Tomcat servers more efficient ... as well as avoiding OOMEs.


In summary, rather trying to load balance to avoid OOMEs, I suggest you figure out why you are getting OOMEs ... and try to deal with the cause of the OOMEs directly.

share|improve this answer
add comment

Maybe jvmtop is worth a look for you.

It shows you in a "top-like" manner on a per-jvm basis monitoring metrics like memory consumption, cpu utilization, thread counts etc.

share|improve this answer
add comment

If monitoring externally, I'd recommend using an OS level monitoring tool such as ps on Unix to give you memory consumption stats and set up process alerts. You can and should use JMX to monitor heap utilization but remember off heap allocations via direct byte buffers or JNI libraries are invisible to JMX heap monitoring and may lead to even more catastrophic failures than OOM.

share|improve this answer
add comment

JMX would be the answer (Jolokia being a JMX interface).

You might want to also look at - http://stackoverflow.com/questions/242958/best-tools-to-monitor-tomcat

share|improve this answer
add comment

protected by gnat Mar 22 at 8:16

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.