Opened 7 years ago

Last modified 8 months ago

#29023 new Cleanup/optimization

running tests in parallel doesn't show exception chain, even with tblib

Reported by: Chris Jerdonek Owned by: nobody
Component: Testing framework Version: dev
Severity: Normal Keywords: parallel, exception, chain, traceback, pickling, tblib
Cc: Chris Jerdonek, Sage Abdullah, Ülgen Sarıkavak Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Running tests in parallel doesn't show more than the first link of an exception chain. For example, with the following test (with tblib installed):

def test(self):
    time.sleep(2)
    try:
        raise ValueError('foo')
    except Exception:
        raise KeyError('bar')

this is what the failure looks like when running tests in parallel:

Traceback (most recent call last):
  File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line 605, in run
    testMethod()
  File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2
    raise KeyError('bar')
KeyError: 'bar'

In contrast, this is what it looks like when running them not in parallel:

Traceback (most recent call last):
  File "/Users/.../myproject/myproject/tests/test_a.py", line 11, in test2
    raise ValueError('foo')
ValueError: foo

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2
    raise KeyError('bar')
KeyError: 'bar'

Side question: why must pickling be used at all? Why can't we simply render the traceback to a string using Python's traceback module, and then send the string back? It seems like that would result in simpler, more understandable code and sidestep many of the difficulties.

(This was reported using Python 3.6.4. and master as of Jan. 15, 2018: 02365d3f38a64a5c2f3e932f23925a381d5bb151 .)

Change History (8)

comment:1 by Chris Jerdonek, 7 years ago

Cc: Chris Jerdonek added

comment:2 by Chris Jerdonek, 7 years ago

One more thing to add: in the case where the full chain isn't being shown, there's also no way for the user to know that anything is missing (without also running in non-parallel mode and comparing the test outputs).

comment:3 by Simon Charette, 7 years ago

Side question: why must pickling be used at all? Why can't we simply render the traceback to a string using Python's traceback module, and then send the string back? It seems like that would result in simpler, more understandable code and sidestep many of the difficulties.

From what I know It's just the default behavior of the multiprocessing module for inter process exception bubbling. I feel like it's a bit harder than that as you still have to construct a traceback object from the string on the parent process side to raise the appropriate exception.

comment:4 by Tim Graham, 7 years ago

Summary: running tests in parallel doesn't show exception chain (even with tblib)running tests in parallel doesn't show exception chain, even with tblib
Triage Stage: UnreviewedAccepted

comment:5 by Chris Jerdonek, 7 years ago

For future reference, here is one possible work-around that I think would be better than the status quo. If an exception raised during a test either (1) fails the "pickleable test," or (2) has a chain of length bigger than one (e.g. non-None __cause__ attribute), then the traceback could be formatted to a string in the child process and a pickleable exception of the form UnpickleableTestException(traceback) could be sent back to the parent process. The end result won't look as clean as the pickled case, but at least the information of the full exception chain should display. Also, in the case of unpickleable exceptions, it should prevent the test run from aborting completely. So it would address two issues.

comment:6 by Chris Jerdonek, 7 years ago

FYI, I implemented a proof-of-concept for the idea I described in my previous comment and posted it here: https://github.com/cjerdonek/django/tree/ticket_29023

In particular, it handles both chained exceptions and non-picklable exceptions, and without aborting the test run in the latter case.

I included example "before" and "after" outputs for both cases of chained exceptions and non-picklable exceptions in the commit message of this commit: https://github.com/cjerdonek/django/commit/0040040ca4fec08a9980a24f08bcad900e2b0f22

comment:7 by Sage Abdullah, 15 months ago

Cc: Sage Abdullah added

comment:8 by Ülgen Sarıkavak, 8 months ago

Cc: Ülgen Sarıkavak added
Note: See TracTickets for help on using tickets.
Back to Top