How to update display during large processing job

Question

I have an ASP.NET C# application that needs to go off and iterate through a directory of .xml files and do some processing on the data inside them. This is easy enough, but i need to display the result after each .xml is processed, how can this be done?

What im really asking here is how do you process a lot of data in a loop that needs to be displayed back to the client even though the server processing has not fully completed. Is this even possible?

I could start the big process in a seperate thread and start a js timeout to postback and essentially poll the process but this doesn't sound great and would probably require using static variables to maintain state.

Would it be correct to use SessionState / ViewState / QueryString to persist the status of which file ive processed and do away with the loop altogether? Therefore seperate the large job into seperate requests?

Answer 1

The bottom line here is that if you are a server, you can't maintain a persistent connection with your clients. This means that your solution has got to be based on some sort of polling. I would structure it like this:

1. Client clicks on "Start Processing" or similar.
2. As a result, there is a Javascript AJAX call to a WebMethod on the server.
3. WebMethod begins the processing, and thenreturns the result of processing the first file when ready.
4. Client receives the response to the service call containing the result of processing the first file, displays this response, and then requests the next result.
5. Server returns the next result of the processing, and the loop continues.

The caveat to this, naturally, is dependent on how the latency of your client/server connection compares to the average time taken to process each file. Also, do you have to show the user the result of each processing, or is it OK if one is skipped because the processing loop is iterating faster than the display update loop. You could always tie them together, so that the server does not process the next file until it receives a request to do so from the client.

Answer 2

Contrary to @Mike's answer, you actually can maintain a "sort" of persistent connection with your clients. With HTML5 and WebSockets (or Flash), you can. However, there is a really simple way to do this if you don't mind a 3rd party service.

We just implemented a real-time chat system in a membership site using Pusher.com's service. They provide a JavaScript library which maintains a persistent connection to their server, and then allow you to post messages to their server, which are instantly delivered to all of the JavaScript clients, allowing the display to update.

Check it out: http://pusher.com/

Answer 3

I would say that you should not do long processing causing the user to wait for long time in an ASP.NET application to start with.

I suggest that you do the processing in batch and have the result stored on some kind of data store (file or database).

If the information changes frequently (say, you keep getting XML files), you could set a trigger/watcher on the directories to monitor arriving XML files, process the new ones and update the results (in batch). You could also build an ETL job for that.