Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I have a proof of concept application that uses Azure tables to associate DNA sequences to "something".

Table 1 is the master table. It uniquely lists every DNA sequence. The PK is a load balanced hash of the RK. The RK is the unique encoded value of the DNA sequence.

Additional tables are created per subject. The PK is a load balanced hash, and the RK is the unique value of the DNA sequence. Assume that the quantity of RKs here is many order of magnitudes smaller than the Master table. Each subject has a list of N DNA sequences that have one reference in the Master table, where N is > 100,000.

It is possible for many tables to reference the same DNA sequence, but in this case only one entry will be present in the Master table.

My Azure dilemma:

I need to lock the reference in the Master table as I work with the data. I need to handle timeouts, and prevent other threads from overwriting my data as one C# thread is working with the information. Other threads need to realise that this is locked, and move onto other unlocked records and do the work.

Ideally I'd like to get some progress report of how my computation is going, and have the option to cancel the process (and unwind the locks).

Question

What is the best approach for this?

I'm looking at these code snippets for inspiration:

http://blogs.msdn.com/b/jimoneil/archive/2010/10/05/azure-home-part-7-asynchronous-table-storage-pagination.aspx

http://stackoverflow.com/q/4535740/328397

share|improve this question
    
Just my two cents, if you're concerned about state and not trying to implement the state maintenance in the database itself, you might take a look at Ela ( code.google.com/p/elalang ), it's a pureish functional language on the CLR so it should be able to execute in azure. Granted the CLR isn't pure so neither is it completely, but it'll be far more natural to minimize state in than C# as it doesn't have mutable data structures/assignable variables –  Jimmy Hoffa Sep 12 '12 at 4:03
    
Well, I don#t know anything about Azure tables, but can't you make some kind of snapshot of the data your C# process needs before it starts to process the data? Making the snapshot will need a lock, but if you have an idea how to implement that fast, you won't need any locks during the main processing stage. –  Doc Brown Sep 12 '12 at 7:24
    
@DocBrown - I need to maintain state across many nodes, and even if a node were to restart. –  makerofthings7 Sep 12 '12 at 13:49
    
@JimmyHoffa Perhaps I could use that, or Azure App Fabric caching to maintain state across all nodes, but that option adds a price I'm not yet willing to pay. I think using Azure Table will give me an architecture that can use a memory cache sometime in the future. –  makerofthings7 Sep 12 '12 at 13:51
    
Why do you need to lock the master table? You're not doing any updates to it... If you have a number of worker threads then having a work queue seems like a better approach, or at worst some lock pool that exists in a distributed cache; they're pretty good with that sort of thing. –  Telastyn Nov 3 '12 at 19:08

1 Answer 1

While I've not worked with Azure, only on-premise SQL Server, it seems like this is really a matter of asynchronous data access/concurrency.

Why not simply have each thread keep track in a centralized location (e.g. another table) which set of data it's working with. Then, subsequent threads can ignore any records that are already "checked out" to another thread. Once the thread is finished working with the data it has "checked out", it would then remove the record from the centralized location.

Also, you could include the UTC timestamp of when the record was "checked out", so that if something where to fail or otherwise timeout, then the "check out" record (i.e. the lock) could be cleared by the next thread, or even a seperate worker process.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.