If you cannot have the information "pushed" to you, you have to build your own "push".
That is, you should have one application that does your FTP polling and the other, not directly connected to the first one, that retrieves the new data and treats it. Your second application should "subscribe" to the first one in order to receive a notification for the availability of a new file. This way, you should use very few resources since your main application will only launch/work when receiving a notification. Disconnecting into at least two applications is also more flexible since with the notifier/subscriber architecture any other new application you write would only need to subscribe to your polling service to be able to work with it.
Also, to use even less resources, you can poll with an increasing time frame. For example, let's say your polling time frame is between a minimum of 2 and 10 minutes. You start with 2 minutes. If the polling application does not find a new file, the time is increased by 1 minute, until you reach a maximum of 10 minutes. If the polling application finds a new file, the time is set back to 2 minutes. You can even imagine a decreasing time frame...
Finally, to improve your architecture even more, you can even break the problem into 3 applications: one for the polling, one for retrieving the data and storing it somewhere (like a cache), and another application that treats the data (and therefore has no relationship whatsoever with the "source" of the data, be it FTP, HTTP, or a web service)
Edit: below, an example of an architecture based on web applications and web services for my first proposal (two applications).
The Poller Service can be a web application composed of:
One Web Service composed of two methods:
- Subscribe for allowing a client application/web service to register for receiving
information that a new file is available. One of the parameters of this web method should be the HTTP address where this "client" could be reached at.
- Unsubscribe for not receiving anything anymore.
One service triggered every x minutes for polling an FTP for new files. When a new file is found, the service calls every subscribed "client" Web Service's NewFile web method (which I describe right below)
Now, a client application can be really anything but it should have a web service facade with at least the following web method:
NewFile, for allowing the poller to push the information on the availability of the new file. Within this web method the poller may also send some more information on the file such as the size, when it was modified and of course its exact location.
When this application starts, it calls the Subscribe method of the Poller web service.
- When this application exits, it calls the Unsubscribe method of the Poller web service.
Now, this was an example using web services, but replace web service by REST and web method by URL, and you have another architecture that works.
