Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I need to create a REST web service to manage user submitted images and displaying them all in a website. There are multiple websites that are going to use this service to manage and display images. The requirements are to have 5 pre-defined image sizes available.

The 2 options I see are the following:

  1. The web service will create the 5 images, store them in the file system and and store the URL's in the database when the user submits the image. When the image is requested, the web service will return an array of URLs.

    I see this option to be a little hard on the hard drive. The estimates are 10,000 users per site, and lets say, 100 sites. The heavy processing will be done when the user submits the image and each image is going to be pulled from the File System.

  2. The web service will store just the image that the user submits in the file system and it's URL in the database. When the user request images, the web service will get the info from the DB, load the image on memory, create its 5 instances and return an object with 5 image arrays (I will probably cache the arrays).

    This option is harder on the processor and memory. The heavy processing will be done when the images get requested.

    A plus I see for option 2 is that it will give me the option to rewrite the URL of the image and make them site dependent (prettier) than having a image repository for all websites. But this is not a big deal.

What do you think of these options? Do you have any other suggestions?

share|improve this question
    
General rule of thumb: store thumbnails in the database, store full-size images in the file system. Where the cutoff-point is is anyone's guess. Radical idea: Generate and serve the intermediate sizes from the full-size picture as they are requested, and cache them for awhile in the file system. –  Robert Harvey Mar 19 '12 at 23:20
    
@RobertHarvey cool! storing some of the images in the db will could help to share the load between db server and file server. That is, if I choose the first option right? About your radical idea, are you suggesting, storing thumbnails in the DB, full size image in the File System and dynamically generating intermediate sizes? (btw, the 5 sizes will always be requested at the same time) –  ismaelf Mar 20 '12 at 0:22
add comment

2 Answers

up vote 0 down vote accepted

Definitely use a file store, whether that's a classic CDN, a cloudy one, or a roll-your-own. I also wouldn't bother storing all 5 image sizes. Create the scaled images on the fly, at first access (i.e. if the DB query for a particular image and size returns NULL for the URL, scale the larger image down, and return the newly created URL, then store it for future references).

Don't bother too much about hard drives. There's already a very effective file cache in every relevant OS, and you're going to be hard-pressed to beat that. Furthermore, you'd planning to service a million users. You can quote storage at a reasonable $1.00 per user and have plenty of money to play with.

Don't worry about the URL pretty-ness. Our filestore is known as download.<OurCompanyName>.com, even though it's actually hosted by a well-known CDN company.

share|improve this answer
    
Thanks! certainly hard drive space is cheaper and less critical than memory/processor! and cache also enters the equation. Hmmm Yes, I think I'll do that, I will dynamically create the other images and stored them when they get requested. –  ismaelf Mar 20 '12 at 20:42
add comment

I would rather host them on image servers and link to them from databases. Databases aren't very good at handling huge blobs.

For example, if you were using AWS suite, I would say maintain your data (upto 64kb) in DynamoDB and put the actual image in S3. (You can use CDN if geographical distribution/huge availability is necessary). Depending on your database, I would find one such solution.

And yes - ensure the FS is as available as S3, and that you have thought about long term persistence / backup of these as well. When the db gets backed up / restored, the FS has to restore to the same point - else you will see very weird bugs. This is way harder to implement in practice, if you plan to build this store on your own.

share|improve this answer
    
Thanks for your reply! I will consider a CDN. What do you think would be the best way for the web service to return the images, dynamically re-sizing images and returning byte arrays? or saving all sizes in the CDN and return all the images url's ? –  ismaelf Mar 20 '12 at 1:27
    
Storage is quite cheap compared to the latency for generating these files in most scenarios - so I would go with generating all sizes and storing them. Unless you don't care about latency - then you could generate them on the run, but do look up the math on CPU time vs storage space. If you indeed go with on demand generation, consider the fact that you could add a cache for the most recent, there by reducing the actual generations considerably for repeat invocations. If your access pattern is a long tail (like most are), this would be most beneficial, at the cost of an extra layer of caching. –  Subu Subramanian Mar 20 '12 at 2:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.