17

What are some good practices in debugging multiprocessing programs in Python?

2 Answers 2

7

True multiprocessing Python programs (as opposed to multi-threaded Python programs which have to deal with the dreaded GIL) are no different from those in any other language. They all have the same basic challenges:

  1. Task assignment and result reporting. Even if they are mostly working on independent datasets, they normally have to go back to the master thread to report results and get new data to work on. This can be a choke point.
  2. Race conditions. Processes are trying to use a one-at-a-time resource and they need to use mutex (or something similar) to keep from stepping all over each other's data. Failure to protect these kinds of resources can lead to really, really painful debug sessions.
  3. Sequentiality. Sometimes you are trying to make something parallel that isn't. The various processes end up waiting for each other to do something and the end result is that you have, for all intents and purposes, taken a sequential program, made it parallel, and it still ends up executing in linear time (or worse).

Although there are development methods that try to avoid each of these problems, at the end of the day you really need to think about what you are doing. I recommend heavy stress testing — far beyond anything you think might happen in real life — so that you have a good chance of hitting those Windows of Opportunity and blowing up in development as opposed to in the middle of a major demo or during production.

We used to use microsecond-timestamped log files and then created a color-coded log viewing app so we could try to visualize what was happening between N process running on M processors. We also tried (and mostly succeeded) in creating a system that would drive off the log files to recreate the context of the crash.

But the best tool is good design and really wicked, nasty people who try to blow your app out of the water. (Hi, dbell!)

28

One thing that I find very helpful, is to use the existing logger in the multiprocessing module. Try this in your main code:

import multiprocessing, logging
mpl = multiprocessing.log_to_stderr()
mpl.setLevel(logging.INFO)

See also: http://docs.python.org/library/multiprocessing.html#logging

Additionally, you can access the current process name using:

cpname = multiprocessing.current_process().name
# print cpname
mylogger.info("{0} is currently doing...".format(cpname))

See: http://docs.python.org/library/multiprocessing.html#multiprocessing.current_process

Apart from that I don't know of anything else except the standard debugging methods like pdb & co.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.