I am writing a program to scrap some data from the web. The pages are sequential ( 1,2,3 ... ), but I have no idea when will it stops. I combine a prefix and a integer to make a link for the python urllib to parse on it. For example : 'http://some.domain.com/page' + '1' + '.htm'.
So the request would fail if the link is invalid, however, there could be some other error such as network error, connection timed out which will resolve itself. I could retry a couple of times on these error.
Aside from 'Page not found' error, there could be other error such as 'Internal Server Error' which was thrown when the server is down. There error won't fix themselves. I should probably move on or stop the program.
Back to my program. Because I don't know when is the page index are going to end, so I set the end integer to 9999. The links may run out after some hundreds. I should identify this and send a 'break' to it.
What will you do? Gather all the possible error and put them on the exception line and treat them differently? Now I just stop after 10 failures.