I'm working on a system file scanner that reveals info about various files (e.g. size, last used, duplicates, etc). Currently I'm traversing the file system once just to get a good measure of the files I'll be processing, I then loop through doing the actual processing (size info, hash info, etc). Obviously that immediately creates an entire layer of "extra" processing, but it allows me to use the previously acquired info to provide the user with some "progress data".
I've been looking for a good mechanism to use in order to speed up the process while still showing progress data for end users. I thought about creating separate threads (one for appending files to a stack, and the other for reading from the stack as they became available), but that might get out of hand programmatically quickly.
In the interest of speeding up the initial scan, I currently perform a "find path" (or the equivalent depending on what OS is being used) and grab all of the output. This, however, prevents me from negating entire subfolders (if the user desires) as it simply recursively lists everything. Certain OSes do have command line options for negating directories, etc, but I need a cross-platform solution.
So, aaaaaaall that said, does anyone have any algorithmic suggestions for being speedy while providing quality progress? I'm not fundamentally tied to any specific language. I'm looking more for a higher level view of what needs to occur.