Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm researching ways to build a web service to periodically traverse a predetermined list of web pages (of another external website) to detect if a page's content has changed from

  1. editing of the page, and
  2. deletion of the page.

The end goal is to have this web service post push-notification events to mobile devices.

FYI, I've searched and read "Questions with similar titles" here.

Thank you for sharing your answers.

share|improve this question
Can you share your current research? I feel that your question is too open ended for programmers as it stands. – Oded Mar 24 '12 at 20:52
I don't think you really want a web service, since you're not actually providing a service. This type of program would be more of a periodic task or cron job. – TMN Mar 24 '12 at 21:40
Thanks for your feedback. I was looking at the problem as a web service because this service (to monitor an external website) will then initiate push notifications to an app on a mobile phone. – Global nomad Mar 25 '12 at 5:44

What about just reading the page with cURL? Maybe store a checksum of what you read and compare against that next time.

share|improve this answer
Thanks for the suggestion. I was thinking of a checksum solution but thought I'll seek alternative approaches before deciding. – Global nomad Mar 25 '12 at 5:45

Editing of the page can be checked by:

  1. Check if the HTTP header last-modified is present. If it is, compare it to the previously stored value and you are done.
  2. If the last-modified HTTP header is not present, store the page checksum or checksum of the important part of the page. Since some of the web pages contain randomly generated parts, you want to ignore these parts. You can define the 'important part' of the page using e.g. regular expression. Next time you will compare the stored checksum with new one.

Delete of the page can be checked simply by trying to connect to it.

share|improve this answer
Thank you for answering. Very helpful and specific. – Global nomad Mar 25 '12 at 5:47
Many web servers, web applications, and even proxies by default set the last-modified to NOW or don't implement it (properly) at all - so the page is ALWAYS changed. – Ben DeMott Mar 31 '12 at 10:21

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.