Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I will be putting together an application to automate an external web site/application. In some instances I will need to navigate the site as a user would (some links I need to follow cannot be predicted and must be parsed from a response)

I am already using Html Agility Pack, and am aware of Tidy if that is needed.

Are there any other technologies I should be aware of?

Are there any recommended patterns for being able to quickly adjust in the event that the external web app changes? I’m envisioning encapsulating the validation of responses as some type of strategy or similar pattern that can be easily separated/plugged in as necessary, but any specific suggestions would be great.

share|improve this question

4 Answers 4

The pattern really should stem from the problem but here are some choices:

If you are mainly concerned with ongoing development cost/annoyance: Strategy Pattern (already mentioned) - use this to implement the component level interfaces, and use a dynamic binding methodology to resolve specific strategies (implementations). An IOC container (I like Autofac these days) would work well.

If you are need to support the above plus have a need to scale: Map Reduce (anyone know of a good m+r fx for .net?).

share|improve this answer

Depending on how much the webpage changes this could be tough. You could keep a list of standard regular expressions in a DB and then when one no longer works you can loop through the base set with others until you hit one. Then you'd want to mark that for future use.

share|improve this answer

If you're looking to automate navigation of an external website as a user would, Watin is perfect for that. It will drive a web browser through an object model, and has a wide range of parsing capabilities built upon the DOM (as well as non-DOM abilities you would find in a browser, including scripting).

Here's a link:

Watin.org

share|improve this answer
    
Watin is like your favorite JavaScript library for C#. –  The Muffin Man Jul 28 '11 at 20:52

If you're looking for the ability to quickly make changes to track those in the external web app, then I'd look at scripting. If C# is your language of choice, then look at compiling code on the fly. If you're using a database, the script code can be stored there, otherwise, plain text files would work too.

As to GoF patterns, Strategy is the textbook choice.

The other "pattern" to consider, especially since you mentioned validating responses using them to select the next action, is a finite-state machine. For example, the states for a web-mail client could be "not logged in", "at inbox", "reading message", "editing reply", etc. You'd execute some action depending on the state, then based on the response, select the next state.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.