I think the ethical question here is really "What will the data be used for?". I'd never write something that harvested e-mail addresses, for instance. Nor would I write something that just reproduced content, even if the license says I can. To me, that feels like littering.
That being said, if the public can get to it, I see no issue with having a program parse the information. You might be interested in studying small world networks and decide to examine link relationships. You might be studying linguistics and trying to figure out how to determine literal phrases from sarcastic ones. Many web sites already offer a convenient way to get the content programmatically, via RSS/ATOM.
On the commercial side, you might be writing something that lets people find a good deal on stuff they want to buy. You might be writing a program that helps a company find mentions of itself on blogs and forums.
Just make sure that you:
- Respect robots.txt
- Identify your bot properly and maintain a page that discloses its exact purpose
- Respect the copyright of others and adhere to their terms
- Don't interfere with the normal operation of someone else's computer (most likely server in this case, but a number of sites are hosted at home)
My razor for solving ethical dilemmas is simply "If in doubt, don't." :)