Opened 7 years ago
Last modified 8 months ago
#29023 new Cleanup/optimization
running tests in parallel doesn't show exception chain, even with tblib
Reported by: | Chris Jerdonek | Owned by: | nobody |
---|---|---|---|
Component: | Testing framework | Version: | dev |
Severity: | Normal | Keywords: | parallel, exception, chain, traceback, pickling, tblib |
Cc: | Chris Jerdonek, Sage Abdullah, Ülgen Sarıkavak | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Running tests in parallel doesn't show more than the first link of an exception chain. For example, with the following test (with tblib
installed):
def test(self): time.sleep(2) try: raise ValueError('foo') except Exception: raise KeyError('bar')
this is what the failure looks like when running tests in parallel:
Traceback (most recent call last): File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line 59, in testPartExecutor yield File "/Users/.../versions/3.6.4/lib/python3.6/unittest/case.py", line 605, in run testMethod() File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2 raise KeyError('bar') KeyError: 'bar'
In contrast, this is what it looks like when running them not in parallel:
Traceback (most recent call last): File "/Users/.../myproject/myproject/tests/test_a.py", line 11, in test2 raise ValueError('foo') ValueError: foo During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/.../myproject/myproject/tests/test_a.py", line 13, in test2 raise KeyError('bar') KeyError: 'bar'
Side question: why must pickling be used at all? Why can't we simply render the traceback to a string using Python's traceback
module, and then send the string back? It seems like that would result in simpler, more understandable code and sidestep many of the difficulties.
(This was reported using Python 3.6.4. and master
as of Jan. 15, 2018: 02365d3f38a64a5c2f3e932f23925a381d5bb151 .)
Change History (8)
comment:1 by , 7 years ago
Cc: | added |
---|
comment:2 by , 7 years ago
comment:3 by , 7 years ago
Side question: why must pickling be used at all? Why can't we simply render the traceback to a string using Python's traceback module, and then send the string back? It seems like that would result in simpler, more understandable code and sidestep many of the difficulties.
From what I know It's just the default behavior of the multiprocessing
module for inter process exception bubbling. I feel like it's a bit harder than that as you still have to construct a traceback
object from the string on the parent process side to raise the appropriate exception.
comment:4 by , 7 years ago
Summary: | running tests in parallel doesn't show exception chain (even with tblib) → running tests in parallel doesn't show exception chain, even with tblib |
---|---|
Triage Stage: | Unreviewed → Accepted |
comment:5 by , 7 years ago
For future reference, here is one possible work-around that I think would be better than the status quo. If an exception raised during a test either (1) fails the "pickleable test," or (2) has a chain of length bigger than one (e.g. non-None
__cause__
attribute), then the traceback could be formatted to a string in the child process and a pickleable exception of the form UnpickleableTestException(traceback)
could be sent back to the parent process. The end result won't look as clean as the pickled case, but at least the information of the full exception chain should display. Also, in the case of unpickleable exceptions, it should prevent the test run from aborting completely. So it would address two issues.
comment:6 by , 7 years ago
FYI, I implemented a proof-of-concept for the idea I described in my previous comment and posted it here: https://github.com/cjerdonek/django/tree/ticket_29023
In particular, it handles both chained exceptions and non-picklable exceptions, and without aborting the test run in the latter case.
I included example "before" and "after" outputs for both cases of chained exceptions and non-picklable exceptions in the commit message of this commit: https://github.com/cjerdonek/django/commit/0040040ca4fec08a9980a24f08bcad900e2b0f22
comment:7 by , 15 months ago
Cc: | added |
---|
comment:8 by , 8 months ago
Cc: | added |
---|
One more thing to add: in the case where the full chain isn't being shown, there's also no way for the user to know that anything is missing (without also running in non-parallel mode and comparing the test outputs).