I've written a lot of .NET clustered applications over the years (exclusively for Active/Passive clusters) and can share what I've learned.
I've always written my apps as Windows Services and include them as clustered resources in the Windows Cluster Administrator. When the cluster fails over from Node1 to Node2, the Cluster Administrator shuts down the service on Node1 and starts it up on Node2. That's the simple part and Windows takes care of the heavy lifting.
Aside - There is no technical reason why you couldn't write a console app and have it kicked off by the Scheduled Tasks service (which can also be setup as a clustered resource) or some other means. When I first started writing clustered apps the requirements often required the process to be kicked off by the arrival of a file and that seemed to be a better fit for a service than for a console app. For consistency sake, we just continued to write them as services.
The design of your app is going to depend quite a bit on what exactly its task is. But conceptually, you'll need some way to keep track of what unit of work your app is executing so that if the cluster fails over, your app on Node2 knows where to pick up where the app on Node1 left off.
If your app is file based, meaning it is performing some work on files or the data contained in one or more files, those files will need to be on a drive that either fails over or is visible to both nodes. Your app will need to keep track of what file it was working on and potentially what record within the file as well. How it stores this progress is really up to you and what resources are available, but the important thing is that this progress is stored in a manner that will survive the failover (ie, in a DB table or a file on a drive). Storing this progress on a local drive of Node1 will do you no good when the app starts up on Node2, if Node1 is dead.
While you're designing/coding, keep asking yourself - if the cluster failed over at this exact point in the process,
- What data would be lost? (and would it matter?)
- Where would the other node deteremine it should start from? (and is it correct?)
There's no magic to it really, just a lot of thought in the details. Should your app do all of the work in transaction (DB or COM+)? That depends on the complexity of the task and requirements. Can you just tag records in a DB as your app completes them and have the other node work on untagged records? Sure, again it depends on the requirements.