Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm researching ways to build a web service to periodically traverse a predetermined list of web pages (of another external website) to detect if a page's content has changed from

  1. editing of the page, and
  2. deletion of the page.

The end goal is to have this web service post push-notification events to mobile devices.

FYI, I've searched and read "Questions with similar titles" here.

Thank you for sharing your answers.

share|improve this question
    
Can you share your current research? I feel that your question is too open ended for programmers as it stands. –  Oded Mar 24 '12 at 20:52
    
I don't think you really want a web service, since you're not actually providing a service. This type of program would be more of a periodic task or cron job. –  TMN Mar 24 '12 at 21:40
    
Thanks for your feedback. I was looking at the problem as a web service because this service (to monitor an external website) will then initiate push notifications to an app on a mobile phone. –  Global nomad Mar 25 '12 at 5:44
add comment

2 Answers

What about just reading the page with cURL? Maybe store a checksum of what you read and compare against that next time.

share|improve this answer
    
Thanks for the suggestion. I was thinking of a checksum solution but thought I'll seek alternative approaches before deciding. –  Global nomad Mar 25 '12 at 5:45
add comment

Editing of the page can be checked by:

  1. Check if the HTTP header last-modified is present. If it is, compare it to the previously stored value and you are done.
  2. If the last-modified HTTP header is not present, store the page checksum or checksum of the important part of the page. Since some of the web pages contain randomly generated parts, you want to ignore these parts. You can define the 'important part' of the page using e.g. regular expression. Next time you will compare the stored checksum with new one.

Delete of the page can be checked simply by trying to connect to it.

share|improve this answer
    
Thank you for answering. Very helpful and specific. –  Global nomad Mar 25 '12 at 5:47
    
Many web servers, web applications, and even proxies by default set the last-modified to NOW or don't implement it (properly) at all - so the page is ALWAYS changed. –  Ben DeMott Mar 31 '12 at 10:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.