Understanding streams

Question

I have started learning streaming APIs and I found one of the good documentations here. There was a comparison given by author to demonstrate the effectiveness of the streams.

var http = require('http');
var fs = require('fs');

var server = http.createServer(function (req, res) {
    fs.readFile(__dirname + '/data.txt', function (err, data) {
        res.end(data);
    });
});
server.listen(8000);

It is told that in the example above, for every request the whole file would be read and stored in memory which may create problems for large number of concurrent connections. Sounds good!

Now, for the solution in the second example:

var http = require('http');
var fs = require('fs');

var server = http.createServer(function (req, res) {
    var stream = fs.createReadStream(__dirname + '/data.txt');
    stream.pipe(res);
});
server.listen(8000);

it is told that since res is itself is a stream hence we can read the file via stream and pipe the result into res which would not create the memory issue.

Question

Will the browser would keep the connection open util the whole file is read since according to me the browser knows about HTTP only hence how would it handle the stream scenario. Also, wouldn't it take longer to stream the file vs sending whole file at a time?

Answer 1

wouldn't it take longer to stream the file vs sending whole file at a time?

Welcome to the time space tradeoff. Reading a whole file into memory (say as an array or string) is called slurping. It can be a good or bad idea depending on file size, available memory, and how many times you do it at the same time.

The alternative is line by line processing (or chunk processing for binary files). As smaller parts of the file come in they are acted on and then forgotten. This keeps the memory footprint small but increases accessing overhead. That's the tradeoff that you must balance.

Streams can do either. When you offer access as a stream you're saying, "I don't know which you're about to do but here's where to get your data". Once you have a stream you can dump it all in memory, act on smaller parts, or hand it to something else as a stream.