I'm looking for good practices or any advice regarding file access transaction mechanisms. We will have multiple instances of an application spread over a redundant network (cloud) watching a directory on a 100% available NAS. I'm also looking for any alternative architecture that may be more appropriate.
- We will have thousands of mobile devices accessing the system every minute. The devices produce a binary file (picture for example) that is sent to a RESTful service without transformation. The device must do the job quickly to preserve battery.
- Since RESTful service must take the input as fast as possible, the binary file is currently written directly as a file in a 100% availability NAS.
- If an error occurs (file not being written correctly), the client is notified and can try again.
We must build a worker service that will watch the file folder for new files to process them. The worker will convert them to another format then move them to another folder on the NAS. We will use as many instances as possible to handle load.
My main concern is about file access. The RESTful service is writing to a temporary file with an extension that the worker service is ignoring (.tmp), then after the last byte is written, the file is renamed. The process is quick, but we can't afford the case where 2 or more worker services access the same file at the same time.
We need some transaction mechanism or any other entreprise architecture pattern.
Is there any kind of file system transaction mechanism that scales?