Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

The Senario:

You have a file with a string (average sentence worth) on each line. For arguments sake lets say this file is 1Mb in size (thousands of lines).

You have a script that reads the file, changes some of the strings within the document (not just appending but also removing and modifying some lines) and then overwrites all the data with the new data.

The Questions:

  1. Does 'the server' PHP, OS or httpd etc. already have systems in place to stop issues like this (reading/writing half way through a write)?

  2. If it does, please explain how it works and give examples or links to relevant documentation.

  3. If not, are there things I can enable or set-up, such as locking a file until a write is completed and making all other reads and/or writes fail until the previous script has finished writing?

My Assumptions and Other Information:

  1. The server in question is running PHP and Apache or Lighttpd.

  2. If the script is called by one user and is halfway through writing to the file and another user reads the file at that exact moment. The user who reads it will not get the full document, as it hasn't been written yet. (If this assumption is wrong please correct me)

  3. I'm only concerned with PHP writing and reading to a text file, and in particular, the functions "fopen"/"fwrite" and mainly "file_put_contents". I have looked at the "file_put_contents" documentation but have not found the level of detail or a good explanation of what the "LOCK_EX" flag is or does.

  4. The scenario is an example of a worst case scenario where I would assume these issues are more likely to occur, due to the large size of the file and the way the data is edited. I want to learn more about these issues and don't want or need answers or comments such as "use mysql" or "why are you doing that" because I'm not doing that, I just want to learn about file read/writing with PHP and don't seem to be looking in the right places/documentation and yes I understand PHP is not the perfect language for working with files in this way.

share|improve this question
2  
I can tell you from experience that reading from and writing to large files with PHP (1 MB isn't really that large, but still) can be tricky (and slow). You can always lock the file, but it'd probably be easier and safer just to use a database. –  NullUserException Oct 26 '12 at 17:15
    
I know it would be better to use a DB. Please read the question (last paragraph number 4) –  hozza Oct 26 '12 at 18:32
1  
I did read the question; I'm saying it's not a great idea and there are better alternatives. –  NullUserException Oct 26 '12 at 18:35
2  
file_put_contents() is just a wrapper for the fopen()/fwrite() dance, LOCKEX does the same as if you'd call flock($handle, LOCKEX). –  Yannis Rizos Oct 26 '12 at 19:12
1  
@hozza That's why I posted a comment, not an answer. –  NullUserException Oct 26 '12 at 19:30

2 Answers 2

1) No 3) No

There are several issues with the original suggested approach:

Firstly, some UNIX-like systems such as Linux may not have locking support implemented. The OS does not lock files by default. I have seen the syscalls being NOP (no-operation), but that's a few years back, so you need to verify whether a lock set by your instance of the application is respected by another instance. (i.e. 2 concurrent visitors). If the locking is still unimplemented [very likely it is], the OS lets you overwrite that file.

Reading large files line-by-line is not feasible for performance reasons. I suggest using file_get_contents() to load the whole file into memory and then explode() it to get the lines. Alternatively, use fread() to read the file in blocks. The aim is to minimize the number of read calls.

In regards to file locking:

LOCK_EX means an exclusive lock (typically for writing). Only one process may hold an exclusive lock for a given file at a given time. LOCK_SH is a shared lock (typically for reading), More than one process may hold a shared lock for a given file at a given time. LOCK_UN unlocks the file. Unlocking is done automatically in case you use file_get_contents() http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems

Elegant solution

PHP supports data stream filters which are intended for processing data in files or from other inputs. You may want to create one such a filter properly using the standard API. http://php.net/manual/en/function.stream-filter-register.php http://php.net/manual/en/filters.php

Alternative solution (in 3 steps):

  1. Create a queue. Instead of processing one filename, use the database or other mechanism to store unique filenames somewhere in pending/ and processed in /processed. This way nothing gets overwritten. The database will be also useful for storing additional information, such as metadata, reliable timestamps, processing results, and other.

  2. For files up to a few MB, read the whole file into memory and then process it (file_get_contents() + explode() + foreach())

  3. For larger files read the file in blocks (i.e. 1024 Bytes) and process + write in real-time each block as reading (careful about the last line which doesn't end with \n. It needs to be processed in the next batch)

share|improve this answer

In PHP documentation for file_put_contents() you can find in example#2 the usage for LOCK_EX, puting simply:

file_put_contents('somefile.txt', 'some text', LOCK_EX);

The LOCK_EX is a constant with an integer value than can be used on some functions in a bitwise.

There are also a specific function in order to control locking for files: flock() manner.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.