Sign up ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free.

Question should be clear from its title. For example Apache saves its access and error logs in files instead of RDBMS no matter on how large or small scale it is being utilized.

For RDMS we just have to write SQL queries and it will do the work while for files we must decide a particular format and then write regex or may be parsers to manipulate them. And those might even fail in particular circumstances if great care was not paid.

Yet everyone seems to prefer filesystem for maintaining the logs. I am not biased against any of these methods but I would like to know why it is practiced like this. Is it speed or maintainability or something else?

share|improve this question
So how would you log DB errors (db unavailable for example) if your logging system logs to a DB? – Marjan Venema Jul 12 '11 at 12:13
@Marjan How would I log Filesystem errors if it fails?! – Yasir Jul 12 '11 at 12:17
Quite true, but if that fails, chances are your DB is inaccessible as well... After all, where / how would it write to its tables without the file system? – Marjan Venema Jul 12 '11 at 12:30
@Yasir: Send all log messages to a syslog server before logging to the filesystem :) – staticx Jul 12 '11 at 12:35
@MarjanVenema the what if game is pointless. What if the local disk is full, your logging will fail but app and os can keep going. If you're logging to a remote DB server though you'll still be able to log. There are pros and cons to either store for log messages, and which is best depends on what you're trying to get out of logging. Sorry, I'll let the herd get back to file log is the one true way. – Andy Jul 25 at 14:51

7 Answers 7

up vote 21 down vote accepted

A) Too many things can fail with the database, and logging this failures is important too.

B) Unless you have a database system allowing autonomous transactions (or no transactions at all), logging would require a seperate connection so a rollback or commit in logging doesn't interfere with rollback or commit in the application.

C) Many things worth logging happen during startup, i.e. possibly before the database connection has been established.

share|improve this answer

I've seen logs written to the DB before (and sometimes you get configurable options for logging, where trace goes to file, errors to DB, fatals to Windows Event log).

The main reasons are speed and size, enabling some tracing can produce vast, vast qualtities of logging - I've trawled through log files gigabytes in size. The other main reason is that reading the logs needs to be sequential, there's no real need to query the log, except to find a certain error or entry - and find-in-file works perfectly well for that.

share|improve this answer
But I have a confusion for this. My notepad, wordpad, gedit or notepad++ or any web-browser won't be happy opening a file 4GB in size. The same browser, however, will be able to show me a list of thousand pages, each containing 500 records printed. Right? – Yasir Jul 12 '11 at 12:14
@Yasir because you are using editors that try to load the whole file in memory. Try to use a smarter editor that is able to 'stream' the big file. Vim is a good example. – Chaker Nakhli Jul 12 '11 at 12:21
@Yasir: This is true, but you are trying to optimize the wrong thing. The vast majority of the time, logs are written and never read. So you make the creation of logs very fast because it is the common case. – unholysampler Jul 12 '11 at 12:45
Eh, I've done logging to database before and being able to easily query the log messages was immensely beneficial, especially when we turn on debug level logging to track down a hard to replicate bug. – Andy Jul 25 at 0:13
@Andy its overrated, when you have much logging, notepad++ has some great search capabilities that is quicker and easier to use than DB queries. See the search 'mark lines' option with bookmarks, then cut and paste all bookmarked lines into a new tab, and of course a log file lets you highlight repeated bits of text. – gbjbaanb Jul 25 at 11:09

Speed is one reason; others are:

  • Eliminating points of failure. A filesystem rarely fails under conditions where a DBMS wouldn't, but there are lots and lots of error conditions in databases that simple don't exist in filesystems.
  • Low-tech accessibility. If things go really really bad, you can boot into a rescue shell, or mount the disk on a different system, and still have adequate tools available to inspect log files. If it's a database, you're nowhere without a database server running.
share|improve this answer

First off.

And those might even fail in particular circumstances if great care was not paid.

Database transactions can't fail when you are not careful?

Writing to a text file has a number of benefits, the most important being

  • Text is human readable. Anyone can open up a log file with an basic text editor and see what the messages are. You don't need to understand how the database is organized.
  • Speed. Writing text to disc is much faster that a database service figuring out where the text goes in a database, writing it there, and ensuring the transaction completed.
share|improve this answer
Obviously any and everything may fail if we are not careful. But for this question I was refering to high level programmer. As a simple example, the programmer might want to separate values using a particular character. So his/her regex will work like a charm but will fail when the same character is contained inside a value block. This way he needs to take care of similar possible cases and he doesn't need to think about them if he were saving in DB. Also, can you please see my comment on gbjbaanb's answer? – Yasir Jul 12 '11 at 12:21
And if you are hand writing your SQL, you have the same problem. The difference being the write will fail (or corrupt your data) instead of annoying some developer slightly because his search string brought up some bad results. Yes, there are frameworks that mean you don't have to write SQL, but every extra layer slows down the process. And remember this is just logging. Every cycle you use to log is a cycle you are not using to do real work. – unholysampler Jul 12 '11 at 12:51
@unholysampler Your performance argument is weak, logging can be done very fast and on a background thread to a database, and logging to the f's while potentially faster is still not free either, especially if its not done in the background. – Andy Jul 25 at 14:44

You raise Apache specifically, so I will discuss this in detail.

Apache can be configured to log to a database, although it requires an external plugin to do so. Using such a plugin can make log analysis easier, but only if you intend to write your own log analysis software. Standard off-the-shelf log analysers assume your logs are in files, so you won't be able to use these.

When I was doing this, I also experienced reliability issues: if the database server's write buffer filled up (which can happen with mysql if you use up your file system quota for the user it runs under) it starts queuing up queries until they are able to proceed, at which point Apache starts waiting for it to finish, resulting in hung requests to your web site.

(This issue may now be fixed, of course - it was many years ago that I did this)

share|improve this answer

Complexity. Adding RDBMS will increase complexity of whole system astronomically. And ability to manage complexity is the main thing which distinguishes programmers from source code producers.

share|improve this answer
Could you expand on what you mean about complexity as it relates to logging to a DB versus a file system? From my experience, there hasn't been a significant difference in complexity in a business environment. – Adam Zuckerman Jul 24 at 20:45
Really? SqlLite increases the complexity astronomically? And while a web server normally wouldn't need a DB, many LOB apps are already using one, so there's no additional cost there at all. – Andy Jul 25 at 14:47
@AdamZuckerman of course any RDBMS requires maintenance, prone to corruption, may need special tuning, may be affected by bad configuration, may need special recovery, brings own limitations, has own dependencies, supported platforms, upgrade issues, bugs, licensing and so on. – noonex Jul 25 at 21:29
@Andy first of all, SQLite is not RDBMS in classic seance - it is "embedded RDBMS". And yes - requiring SQLite for logging will increase complexity a lot. – noonex Jul 25 at 21:38
@noonex You're just arbitrary making a distinction between embedded vs full server, when RDBMS doesn't. SqlLite provides ACID compliance, which is really what RDBMS are about. And it increases complexity a lot? I can only imagine you haven't worked on anything but the most trivial of applications. Finally, good job completely ignoring my point about many LOB applications already needed a database anyway. – Andy Jul 26 at 15:04

Is it speed or maintainability or something else?


share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.