I've used a software suite that is installed in offices and on remote vessels. The installations communicate back and forth, and they do that by using a simple proprietary file format that looks something like this:
/SHIP:16 MILES=45213 /ORDER:22943 STATUS=OPEN TOTAL=447.84 URGENCY=HIGH /ORDERLINES:22943 ITEM=3544 QUANTITY=1 PRICE=299.99 ITEM=11269 QUANTITY=5 PRICE=29.57
Recently, I've been writing a piece of software for a customer that saves information in the same kind of flat file format.
When the file is opened, the lines are iterated over, and "stuff happens" to the lines (i.e. they're inserted into a database, or whatever).
But it got to me on to thinking, how would this kind of file scale? (I like things being able to scale)
I could of course
gzip it; but how does a file format evolve from being something basic like to this, to being monolithic? What typical practises are employed when making a file format for a new piece of software? How are they typically built?