Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We're implementing a new centralized monitoring solution (Zenoss). Incorporating servers, networking, and Java programs is straightforward with SNMP and JMX.

The question, however, is what are the best practices for monitoring and managing custom C++ applications in large, heterogenous (Solaris x86, RHEL Linux, Windows) environments?

Possibilities I see are:

  1. Net SNMP
    • Advantages
      1. single, central daemon on each server
      2. well-known standard
      3. easy integration into monitoring solutions
      4. we run Net SNMP daemons on our servers already
    • Disadvantages:
      1. complex implementation (MIBs, Net SNMP library)
      2. new technology to introduce for the C++ developers
  2. rsyslog
    • Advantages
      1. single, central daemon on each server
      2. well-known standard
      3. unknown integration into monitoring solutions (I know they can do alerts based on text, but how well would it work for sending telemetry like memory usage, queue depths, thread capacity, etc)
      4. simple implementation
    • Disadvantages:
      1. possible integration issues
      2. somewhat new technology for C++ developers
      3. possible porting issues if we switch monitoring vendors
      4. probably involves coming up with an ad-hoc communication protocol (or using RFC5424 structured data; I don't know if Zenoss supports that without custom Zenpack coding)
  3. Embedded JMX (embed a JVM and use JNI)
    • Advantages
      1. consistent management interface for both Java and C++
      2. well-known standard
      3. easy integration into monitoring solutions
      4. somewhat simple implementation (we already do this today for other purposes)
    • Disadvantages:
      1. complexity (JNI, thunking layer between native C++ and Java, basically writing the management code twice)
      2. possible stability problems
      3. requires a JVM in each process, using considerably more memory
      4. JMX is new technology for C++ developers
      5. each process has it's own JMX port (we run a lot of processes on each machine)
  4. Local JMX daemon, processes connect to it
    • Advantages
      1. single, central daemon on each server
      2. consistent management interface for both Java and C++
      3. well-known standard
      4. easy integration into monitoring solutions
    • Disadvantages:
      1. complexity (basically writing the management code twice)
      2. need to find or write such a daemon
      3. need a protocol between the JMX daemon and the C++ process
      4. JMX is new technology for C++ developers
  5. CodeMesh JunC++ion
    • Advantages
      1. consistent management interface for both Java and C++
      2. well-known standard
      3. easy integration into monitoring solutions
      4. single, central daemon on each server when run in shared JVM mode
      5. somewhat simple implementation (requires code generation)
    • Disadvantages:
      1. complexity (code generation, requires a GUI and several rounds of tweaking to produce the proxied code)
      2. possible JNI stability problems
      3. requires a JVM in each process, using considerably more memory (in embedded mode)
      4. Does not support Solaris x86 (deal breaker)
      5. Even if it did support Solaris x86, there are possible compiler compatibility issues (we use an odd combination of STLPort and Forte on Solaris
      6. each process has it's own JMX port when run in embedded mode (we run a lot of processes on each machine)
      7. possibly precludes a shared JMX server for non-C++ processes (?)

Is there some reasonably standardized, simple solution I'm missing?

Given no other reasonable solutions, which of these solutions is typically used for custom C++ programs?

My gut feel is that Net SNMP is how people do this, but I'd like other's input and experience before I make a decision.

share|improve this question

5 Answers 5

I'm picking up on this as we recently went through the very same process like you: We were looking for a lightweight, non-blocking, open-source solution which allows exposing and subsequent remote monitoring of metrics from within C/C++ services (we have around ~3000).

SNMP came closest but the integration into the source and the monitoring system is a pain and not suitable for our real-time processes.

In the end, we decided to develop a new solution called CMX which uses shared memory technology and made it open-source. You can check it out here : www.cern.ch/cmx.

share|improve this answer

If you're on Windows you tend to write to the event log, and then use a WMI or similar process to read the events. If you want monitoring, you add performance monitor counters to your app and let perfmon read them. Both are system services in Windows.

On Linux, it obviously tends to be more flexible, but I've always seen nagios style monitors implemented, with a custom socket sending data to a nagios style server.

That all said, I have seen several places where SMNP is used, and frankly, I can't see a reason why you wouldn't use it - especially if you're running a completely heterogenous environment.

share|improve this answer

i am not familiar with these products you name but for windows i monitor memory consumption using perfmon, there are some special counters, like non paged pool faults, who show you if your program contains memory leaks, they might be little and thus take a long time to monitor but in my opinion this a simple checking method.

On windows you can do a lot using perfmon, even remotely Or make use of WMI to attach to the same counters, and do some automation on it (in wmi) to perform actions.

share|improve this answer

I am not super familiar with the c++ side of things but in Java we extensively use CodaHale metrics in conjunction with Graphite. CodaHale stores metrics on a per instance basis in the local memory of the instance then uses a background thread to flush metrics to a graphite server every minute (configurable). In graphite we can aggregate across instances as well as identify faulty instances. If you do not want the complexity of maintaining a graphite cluster you can use HostedGraphite.

This setup means no single point of failure for metrics aggregation or reporting as (time based aggregation happens on the nodes themselves and the reporting aggregation across happens in a distributed graphite cluster (or hosted graphite).

Lastly, you can use Seyren to provide alerts on top of the monitoring data.

share|improve this answer

I'm not super familiar with Zenoss but when I used to used nagios for this sort of thing we'd make the c/c++ process listen on a socket and write a custom nagios plugin which would hand over diagnostic and status information.

First step is to choose the lib you want to use to make your process listen.. Something like C++ Socket Library will do for that. Nothing complicated there.. just make the process listen.

Then you have to define the response your process will send given a particular stimulus. This really meant (at least with nagios) defining the 'service' and then sending the process the signal that corresponded to that service. The simplest thing you can do is create a 'process ping' just see if you can successfully connect to the running process. If you do than the custom nagios plugin knows at least the process is still alive.

There's much more sophisticated stuff you can do but the idea is simple enough. You can write your own little lib of process listening code encapsulated within objects and pull it into your custom c++ stuff in a standardized manner whenever you build one (or all) your executables

My understanding is Zenoss can do this too.

Probably since Zenoss is python then you'll write your custom plugin for it using something like Twisted for connecting to your listening c++ executable.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.