Discovered a bug in Python 2.x/3.x

I have recently discovered a bug in Python (both in the 2.x and 3.x families) and offered
a patch to solve the issue.

When imap() or imap_unordered() are called with the iterable parameter set as a generator function, and when that generator function raises an exception, then the _task_handler thread (running the method _handle_tasks) dies immediately, without causing the other threads to stop and without reporting the exception to the main thread (that one that called imap()).

I saw this issue in Python 2.7.8, 2.7.9 and 3.4.2. Didn’t check other versions, but I assume this is a bug in all Python versions since 2.6.

I reported this bug here and attached examples that reproduce this issue, as well as patches for both Python 2.7 and Python 3.4.

The patches I attached do 2 things:

  1. A deadlock is prevented, wherein the main thread waits forever for the Pool thread/s to finish their execution, while they wait for instructions to terminate from the _task_handler thread which has died. Instead, the exception are caught and handled and termination of the pool execution is performed.
  2. The exception that was raised is caught and passed to the main thread, and is re-thrown in the context of the main thread – hence the user catch it and handle it, or – at the very least – be aware of the issue.

Now I’m waiting for a review.

Update, 2015-03-06: patch was reviewed, tests were added, and it was merged into all
Python branches.

Leave a Reply

Your email address will not be published. Required fields are marked *