Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm building an API for an ad serving platform that will allow you to request tracker data for ad campaigns. Campaigns often exceed hundreds of millions of requests, which means there will be many terabytes worth of data. Therefore we need to prevent API consumers from requesting too much data at once (such that the request times out), but I'm not sure what the best practice is for doing so is.

Options I've already identified are:

  1. add an extra parameter to the request that indicates which section of the data is desired
  2. truncate the data and somehow tell the client that they need to use more specific filters
  3. respond with HTTP status code 413 (but this appears to be for large request bodies, not responses)
  4. switching to a streaming API (like twitter's streaming APIs)

But my question is, what is the standard practice / proper response for this kind of situation?

Note: DoS attacks aren't much of a concern since this will not be a public API

share|improve this question

migrated from serverfault.com Sep 3 at 19:45

This question came from our site for professional system and network administrators.

1  
or make the error part of the API, –  ratchet freak Sep 3 at 19:56
    
2) seems like a bad idea because the client programmer may overlook the flag of "incomplete data". If you fail to provide what the client requests, make it clear that you are not providing it (fail hard, and fail early). I would vote for 3) or better, ratchet suggestion. –  SJuan76 Sep 3 at 19:59
    
recommended reading: Why is asking a question on “best practice” a bad thing? –  gnat Sep 3 at 20:04
    
@gnat would it be more appropriate to just ask what solutions others have successfully implemented? –  Griffin Sep 3 at 20:53
1  
@gnat done. Thanks for the feedback –  Griffin Sep 3 at 21:04

3 Answers 3

Return the harshest, unfriendliest result possible in the event of a malformed request (one that returns more data than your metering allows is malformed). I suggest returning a 4** error code. Then, also provide paging parameters, so that users may request pages. oData has this feature, for instance. Do not truncate the data silently, under any circumstances.

Consulting with customers is a bad idea. They are going to tell you to do whatever possible to minimize errors, which is a bad engineering approach. This is your decision, take it by the horns and do the right thing.

An example of a paginated api is oData:

http://www.odata.org/documentation/odata-version-2-0/uri-conventions/

share|improve this answer
    
+1. 412, 413, 416, 417 are correct responses. –  Residuum Sep 3 at 21:28
    
can you give an example API that batches / paginates the results? –  Griffin Sep 3 at 22:09
    
@Griffin edited to reflect an example –  Chris McCall Sep 4 at 19:05

To expand on what @joshin4colours said, I think you have a false dichotomy (trichotomy?). Why not provide all three solutions? Maybe the default is to return a 413 but with other flags you can either get some of what you want with an embedded error in the data and/or provide a way of batching the data.

It really depends on what your specific customer/consumer of the API expects and how they want to use your API. Are they ever going to want a 413? Should the default response include some data and indicate how much more there is? Maybe. You could also put yourself in the client's shoes and think about what they would want, i.e. what would be useful to them.

What I've usually done is give the first batch of data with an idea of how much more there is. Returning a 413 isn't very friendly, but maybe that's want you want in some cases. From what I've experienced, usually there's a default batch size but people can ask for a certain batch size up to some limit.

Also, you could consider aggregation or sampling to reduce batch size. For example, I want 50,000 results as a random sample of 5,000,000 matching records. There are different ways to slice and dice depending on how statistically significant you want your results to be.

share|improve this answer
    
right, consulting the actual customer(s) is always a good idea. In the mean time I'd like to explore what solutions have worked for others. –  Griffin Sep 3 at 21:01

Not sure about a best practice, but in our case we do have parameters in our API that are set to some kind of maximum value (think Integer.MAX_VALUE from Java). These parameters are often not available to the UI/client-side of the application, only to server-side calls.

Basically, the approach would be to set a maximum on records returned by your request. Seems to work well, particularly when data doesn't need to be organized or paginated in any way.

If a client (human or otherwise) need more than this maximum, you may want to consider increasing it or batching your data somehow.

share|improve this answer
1  
and at the very least document the maxes when they leak through the abstraction –  ratchet freak Sep 3 at 20:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.