Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I would like to monitor the filesystem in python, so that my application gets warned of the new file addition, file removal, or file change. Once the file is detected, the application starts extracting the contained data through various plugins. The problem is that I am dealing with big files, and when the user starts copying a file from outside into the watched directory, it will be detected, but it will appear as corrupted. Checking for file size between invocations is potentially a good strategy, but it ignores the fact that other generators of the file (such as wget) might have long pauses when the file is not changing in size, and yet is not completed. I don't have control of the file format I am downloading either, so I can't check for an end-of-file mark, because it could not be there.

Is there a cross platform way (or different, platform specific ways for linux and windows) to solve this problem? Can I check if a file is currently open somehow? How is this problem solved in other software?

share|improve this question
    
In windows at least I can say it's possible to query the kernel for handles on a given file that would tell you if anyone had a write handle so you could poll. Linux likely has similar though I don't know about it, also doing this from python may be tricky but I presume python has some capability for FFI that will allow COM calls or etc –  Jimmy Hoffa Jun 7 '13 at 22:42
    
I have not used it but watchdog (pypi.python.org/pypi/watchdog) might be appropriate. It has has the "watchmedo" command you could use to see if it handles partial transfers correctly. –  Le Petit Prince Jun 8 '13 at 19:37

1 Answer 1

Since you know which file is changed but not by which process this is hard but solvable. You can use psutil to handle this.

With psutil you can iterate over all running processes and ask them for their open files. Written as pseudo code:

for process in psutil open processes:
    for file in process all open files:
        check if this your file:
            yes => remember pid
pid is set:
    yes => wait for the process to finish
do your work with the file

This is an easy but not very performing way to handle your problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.