This is long question I know. I have a few options for what I want. I don't want to know "this is obviously the best," but maybe any pros or cons I may be missing about the options, as well as any options I'm unaware of.
We have a java application seperately run as multuple daemons, each a seperate JVM. Right now SQL is used to store data and effecticvely our 'shared memory', but luckily not much is dynamic once in memory.
We will soon be dynamically creating and destroying connections and that will mean dynamic changes happening semi-fast and need something better then a SQL database. We think all the dynamic stuff will happen in only one daemon, but we aren't certain.
We will need a faster method of saving and loading this state data, going to disk each time is just too slow. The current database needs an overhaul for this change anyways, so we're open to modification or tossing of it as well. We can re-learn all the state we keep in memory; but it's a slow process. Persisting some or all of the data to a DB between runs to save time learning state at bootup is nice, if we can be certain it gets persisted right.
Ability to easily query a 'database' to see current state for debugging is a major plus.
The solutions I have considered, and some of the pros & cons are:
1) some sort of hibernate solution combined with an in-memory sql database. I haven't used hibernate yet (really want to!), but I think it can address persistance and avoid stale objects between JVM's for us and sql will still be close to the speed of other options if in memory? I don't know how much the overhead for this is or how it scales to multuple JVM's and possible (semi?) fast changes to data. Presumable this is slower then all the other options, but think it's fast enough for our needs?
2) Have each JVM store it's own internal state without any shared system. Use reliable intercommunication (probably JVM) to try to keep state consistent. Most important state is in one daemon, so there won't be that much sharing of dynamic state, but still need to address inconsistent state and dataraces when it does occure. I'm uncertain of how we would want to record persistant state to allow fast reboot even if daemon dies unexpectedly. Also, this solution would require implementing some quick messages to allow fetching a description of memory state to help debugging. It would offer the highest performance once we get it right, but probably the most time consuming to implement (I anticipate obscure bugs due to data-race and inconsistent data?).
3) Use a system to allow some shared memory between JVMs. Quick research suggested MappedByteBuffer and Ecache as possible options, there could be more. Only looked into this a little so far so don't fully know pro's and cons yet. Slightly afraid of creating a single point of failure if our memory is all shared.
4) Some sort of alternative I don't know about or possible merging of two of these idea.
