Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

That is, if you agree that it's OK.

We have a need to maintain an accurate internal record of bank routing numbers, and my boss wants me to set up a process where once a week someone goes to the Federal Reserve's website, clicks on the link to get the list of routing numbers (or the link giving the updates since a particular date), and then manually uploads the resultant text file to an application that will make the update to our data.

I told him that a manual process was not at all necessary, and that I could write a routine that would access the FED's routing numbers in the application that keeps our data updated, and put it on whatever schedule was appropriate. But he is greatly opposed to doing this, and calls it "hacking the Federal Reserve website." I think he's afraid that the FED is going to get after us.

I showed him the FED's robot.txt file, and the only thing it forbids is an automated indexing of pages with extension .cf*:

User-agent: *    # applies to all robots
Disallow:   CF   # disallow indexing of all CF* directories and pages

This says nothing about accessing the same data automatically that you could access manually.

Anyone have a good counterargument to the idea that we'd be "hacking" the FED?

share|improve this question
    
You could ask him to elaborate on how he sees this as hacking as if you each have a different view of the term that could be where the problem is. Don't forget that in getting the application up and running at first there may be many calls to that outside site that may be seen as flooding the server in a sense. –  JB King Jan 6 '11 at 17:14
    
Were you going to screen scrape or? –  Aaron McIver Jan 6 '11 at 17:20
    
Of the two methods on the page of getting data, one for a list of recent changes, and the other for a complete current list, I had two different routines, using a POST and a GET. –  Cyberherbalist Jan 6 '11 at 18:28
add comment

6 Answers

up vote 28 down vote accepted

Point him at this page:

http://www.federalreserve.gov/datadownload/help/default.htm#youShouldKnow

And the Automated Systems section which says it's fine.

"The "Direct download for automated systems" link on the page is the URL that can be used by computer programs to retrieve the contents of the data package without actually using a browsing window. It performs the same function as clicking the "Download file" button but is presented as a URL for those who wish to automate the retrieval of data."

As he sees it, they want to be hacked.

share|improve this answer
    
Thanks for that. They don't appear to provide bank routing numbers here, but it does provide ammunition that they have no problem with automated access in principle. –  Cyberherbalist Jan 6 '11 at 18:10
add comment

When the Fed (or anybody else) publishes data on their webserver, they are giving a copy of the data to the server with the implicit instructions: here, give a copy of this data to anyone who asks for it. Whether your computer displays this data in a browser window or feeds it to a custom application to extract the data makes no difference to the Fed.

The ethical question is "Are we allowed to obtain this data from the fed?". The Fed's anwer is implicitly yes: they've put it on their public website. The ethics don't depend on whether you read it manually or with a custom application program once its sitting in your computer's memory.

There is a minor ethical issue with robots if they query so frequently that they impose an unreasonable load on the Feds webserver. However, that doesn't apply in this case, since either process (manual or automated) will be accessing the Fed's site with the same frequency (once a week). In fact, the automated process will probably impose less load on the Fed's webserver, since it will probably just fetch the appropriate HTML document and not bother making the other requests (CSS, images, javascript, favicon.ico, etc.) that a person accessing the page with a web browser would make.

share|improve this answer
1  
Just because something is on a publicly-accessible website doesn't make it legal/ethical to download it for your own purposes. Check the Terms and Conditions of the website. As others have pointed out, the Feds have explicitly allowed it. –  Barry Brown Jan 6 '11 at 19:26
    
@Barry While some uses of downloaded data are unethical/illegal (for example republishing), that's besides the point the OP Boss has raised. The Boss was only concerned based on how the data was converted (manual extraction versus automatic program), not with how the extracted data was ultimately used. Whether the OP's firm's ultimate use of the Fed's data is ethical/legal is a completely independent issue from whether it converted it manually or automatically (i.e. their use may or may not be ethical/legal, but the answer doesn't change if they use a program to get and convert the data. –  Stephen C. Steel Jan 6 '11 at 21:46
add comment

I find I have the problem where I try to explain how I solve a problem to people more so than explain the results of my solution.

I think if you would have told your boss in the first place that you could automate the process without explaining much further than how long it would take you might be in a better position. To explain how the automating process works: "write a routine that would access the FED's routing numbers in the application that keeps our data updated, and put it on whatever schedule was appropriate" sounds too technical for someone who might not be so inclined. It sounds like hacking to them when in all reality it's a very simple and legal process.

I've been working on explaining the results of a projected solution, rather than how I would accomplish it to my supreriors. I think it might help you as well.

share|improve this answer
    
You make a good point, but as it turns out the boss is a developer and knows exactly what it means to access a website automatically. It's not the technical, but the ethical he's concerned with. –  Cyberherbalist Jan 6 '11 at 18:11
add comment

It seems to me that the Fed Bank has made that data publicly available via a 'web service of sorts'. Why not contact them and get an official Yay/Nay from them to make your boss feel at ease? You might then also be able to agree on a ToS etc.

It looks like @JonHopkins has found a relevant page, be sure to check his answer.

share|improve this answer
    
this may be the simplest solution, would certainly make the boss happy! –  RYFN Jan 6 '11 at 17:10
1  
Nice, but where do you see this web service? Looking at the website in question (fededirectory.frb.org), I don't see one. There's a link to "Download E-Payments Routing Directories", but this is not a web service -- in fact, this is the link he wants to use to get the data. –  Cyberherbalist Jan 6 '11 at 17:15
1  
@Cyberherbalist Nice investigation :), if it's the case that the Fed Bank aren't deliberately making this available then he's best finding out whether he'll get in hot water or not. Maybe the Fed Bank will work with him to create a service ;) –  Martijn Verburg Jan 6 '11 at 17:19
add comment

Your boss sounds like he may not be a techy person. You may be able to convince him that having a program accessing the site, is no different to having a browser accessing a site.

Your argument needs to be based on:

  • The time it would take to write the application to manually upload to.

  • The time it would take to write the application that automatically scrapes the data.

  • The time it would take someone to manually scrape the data, perhaps over the course of one year (as opposed to the automatic one just running).

Using this, you may be able to convince him that the automatic way will save him money in the long run. Also it will run regardless of someone forgetting to do it.

edit: as Martijn says, an official yes/no from the FED would also probably go a long way.

share|improve this answer
    
Oh, he's a developer -- and pretty darned good at it -- but he has an overdeveloped sense of ethics. I tried to tell him that every search engine on the planet was hitting on the FED's site every day and nobody's suing them, but to no avail. –  Cyberherbalist Jan 6 '11 at 17:18
    
The argument can't be based on cost, but rather on whether it's legal and ethical to do the downloading. That's the issue. –  David Thornley Jan 6 '11 at 17:26
    
ah my mistake, sometimes you can see a similar response when people don't know what's technically involved :) –  RYFN Jan 6 '11 at 20:16
add comment

Read up on RESTful web services. Some of them are HTML-based. You can bolster your discussion with examples of other RESTful applications. Your boss may not get RESTful applications, and may not see how the web is designed to do exactly what you're proposing.

Indeed, you can probably find an application already written which does the federal reserve page scraping.

Go here: http://www.gregthatcher.com/financial/default.aspx and see how it's done.

https://addons.mozilla.org/en-US/firefox/addon/196149/

Since both of these do the same thing, it shows that you're in good company. Other folks are already doing what you're proposing.

share|improve this answer
3  
That's not the question he's asking :) –  Martijn Verburg Jan 6 '11 at 17:07
    
I think the answer is helpful -- he's showing that there's already a precedence for this. Although I do agree that contacting the FED about an existing web service is the way to go. –  Mark Freedman Jan 6 '11 at 17:19
    
@Mark Freedman: The @Jon Hopkins answer, saves trying to find a person to contact. –  S.Lott Jan 6 '11 at 17:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.