Semi-securely storing data in a remote key-value store, ensuring minimal data corruption/tampering

Question

I'm bashing around a few ideas for storing usage data (similar to google analytics) for some Javascript products that we are working on.

Unfortunately as yet we don't quite know 100% what data we want to store and only have a basic idea of the metrics we want to glean from it.

My thinking is that basically we want to store a set of key-value pairs grouped by a unique instance_id (there may well be flaws in this logic:)). With that in mind I knocked up a (very) quick prototype in monkrb, the grand total of which is below:

  get "/store" do
    key = params[:key]
    value = params[:value]
    instance_id = params[:instance_id]
    Ohm.redis.hset instance_id, key, value
  end

While this should work, I'm somewhat worried that an inquiring mind could easily pollute our data with random sets and key-values, doubled by the fact that this is all going to be called in Javascript so anything we do to secure or encrypt it is going to be pretty transparent to anyone who cares to view source.

A lot of it should be easily filtered out, as come reporting time we'll have an idea which keys we are interested in (and know which ones we've created) so we can just junk any sets that contain keys which we know nothing about (though again there's the risk that additional keys have been added to an otherwise valid set..)

Following down this path here are my questions:

1. What steps can I take to make it as hard as possible to add completely false data? (handshaking, obfuscation, signing, etc)
2. What can I do to try and detect any junk data? (bonus for a rough idea of how to implement too)
3. What sort of checks should I have in place to stop an existing instance_id being guessed and messed about with? (so any historical data can't get polluted)

It should be noticed that this is mostly academic at the present stage, as I know 6 lines of ruby does not an application make, I'm just interested in the scenario and would like to hear others approaches and thoughts on where one could go with this.

Answer 1

I'm assuming that you'll be periodically passing the analytics back to the server periodically (user clicked a button, user flipped to this tab, user performed a search, etc). If this is not the case please correct my assumption.

With respect to the instance ID:

1. Construct it by concatenating some incrementing value (GUIDs work well for this) with a secret. The GUID becomes the instance ID. If you don't use the secret then the increment can be guessed. Store the current ID for that user somewhere persistent and attach it to their session.
2. Next, keep a counter for each POST of the analytics data, and also attach that to the session. Concatenate this with the GUID and secret.
3. Encrypt the result and pass that to the page.
4. When an analytics POST is made, require the encrypted token. Check that the unencrypted GUID is valid, the secret is correct, and that the counter passed in matches the current counter attached to the session.
5. Update the increment each time the analytics are POSTed from the client, and return a new GUID + secret + incremented counter for the next POST.
6. Keep the current GUID + secret + counter in a private scoped variable so it can't be changed, and use anonymous functions as the callbacks for the AJAX.
7. If you're exceptionally paranoid, make the counter a hashed "GUID + secret" as well so the sequential changes in the token aren't fodder for cracking the encryption... ;-)

This way the instance ID is never in the open, the instance ID (or a future one) can't be guessed and spoofed because of the secret, a user can't replay a particular instance ID to the server, and if the user attempts to use a valid running session to spoof an analytics POST they will only be able to get one bogus request in before the real analytics code fails due to an out of sync instance ID. Once the session is closed that instance ID will be closed off as well.

Of course if the user is really intent on griefing you he'll just replace the entire analytics routine on the client with one that allows public access and still does the instance ID reflection, but seriously, it's just analytics (unless this has to do with ad counts or something, in which case your only real option is to count on the server, not the client)...

With respect to validation, it's difficult to advise without knowing what kinds of things you'll be keeping track of. Key checks are easy, but knowing that the data is ok is not. If you get an invalid instance ID on an analytics POST, you'll at least know that that instance ID's data is now suspect.

Note on GUIDs: in Oracle at least, SYS_GUID returns sequential GUIDs when generated in the same database session. This is why adding a secret is important for validation, because it is possible to guess them (however unlikely).

Edit: Now that I'm thinking more about this, it's probably sufficient to just hash the instance ID + secret + counter instead of encrypting, and match it back up to the instance ID attached to the session since you're already storing that somewhere. It will still change each time so it can't be reused and you won't have to muck about with the encryption.