A WebApp writes a stream of data coming over a network to disk as key/value pair and then reads & send it over network again after few milliseconds in 99% of case. In 1% of cases write/read can be few hours to days apart (really don't know).
Latency is strict "no" in the app and we have to serve 100% of customers therefore I am planning to put data into RAM and later read and send it over network and only write to disk if the data is not read from RAM within a certain prefixed interval.
What could be possible solution to this problem?
P.S - Memcached looks like a good solution for using RAM instead of disk but Can memcached read/write of disk after time_to_live?
I also looking at Couchbase but I don't want to write to disk in all cases as 99% write/read happens in the matter of milliseconds.
Is it possible that after time_to_die Memcached could write the key/value to Couchbase instead of removing it from RAM and read it on demand?
After spending last few trying to explore various options. The following looks like a possible solution
- Write the data stream simultaneously to the RAM (using Memcached) and disk using (GlusterFS).
- In case the data is demanded by the application layer check it first in the RAM (Memcached) if it return the data ok, if it does not check the data on disk GlusterFS.
- Remove data stream from RAM if data is demanded by application or when time_to_live is reached whichever is earlier.
Suggestion most welcome.