I am writing an application that will crawl web pages and analyze data, and I'm wondering if I can host this as a cloud-based service.
The main reason I'm looking at cloud hosting is that users may process thousands of pages at once, and I need to return data as quickly as possible. The user will have software installed on their computer that will communicate with this application.
If, for example, each page takes 10 seconds to process, is it possible to set up auto scaling so that a user can process 1, 10 or 10,000 pages in the same amount of time (ie: 10 seconds) by simply adding and removing instances as needed? Note: Each process requires little RAM/CPU resources, so it will be possible to process multiple pages in a single instance but for this example, I'll simplified it to 1 instance per process.
If so, which cloud-based service is best suited for this task?