Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Yesterday I came across this question: How can i call robocopy within a python script to bulk copy multiple folders?, and I though it might be a good exercise for multithreading.

I though of spawning as many threads as files needed to be copied, each routine having an exception handling system to prevent the whole copying process from crashing (and log -using mutex on the log file - if there was an error).

My question: Is there a limit on the number of thread you can spawn almost simultaneously? If yes, what is the limiting factor?

My question is focused on PC desktop, but I welcome any answer on different hardware (embedded systems, calculus clusters, etc.).

share|improve this question
1  
There's no reason to make IO multi threaded, since it's blocking. It only provides extra overhead. –  Yam Marcovic Nov 1 '12 at 14:13
    
Bulk copying of files is best done serial and not parallel. At least for reading and writing data from disks. –  Pieter B Nov 1 '12 at 14:52
    
ok, so it is better to do everything in a single background process, with good exception handling/testing ? Is that how anti-viruses do when scanning hard drive for malvare/spyware/etc ? –  georgesl Nov 1 '12 at 15:03

1 Answer 1

up vote 2 down vote accepted

A quick google query revealed this article about limits of processes and threads in Windows. It seems pretty thorough, and there are other articles about pushing the limits in the same series.

I didn't read it, but from what I glanced, on a 32-bit system you will only be able to create a little over 2000 threads, and then you'll run out of memory (or rather, address space, because a 32-bit process has a 2GB address limit). And that's only for empty threads, which don't do anything.

Also, for a copy routine spawning such a huge number of threads would devastate performance. Not only would there be a huge amount of context switching going on, but also the source disk would get thrashed as all the threads tried to read from it at the same time. It might be better for an SSD, but it would still mean a huge IO queue for the system to maintain.

All in all I'm not saying that a file copy operation can't benefit from multithreading, but blindly creating a thread for each file doesn't solve anything. Because in a copy operation it's not the CPU that's the bottleneck - it's the disk (or the network, if you're copying over the network). So any optimization would need to address these problems first.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.