#35010 closed Bug (invalid)
Parallel test runner hangs if one of the tests segfault
Reported by: | Michał Górny | Owned by: | nobody |
---|---|---|---|
Component: | Testing framework | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | Triage Stage: | Unreviewed | |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
When the tests are being run in parallel, and one of the tests cause a segfault, the test runner hangs:
$ PYTHONPATH=. python3.11 tests/runtests.py --parallel=2 queries.tests.{ProxyQueryCleanupTest,Queries1Tests} Testing against Django installed in '/tmp/django/django' with up to 2 processes Found 78 test(s). Creating test database for alias 'default'... Cloning test database for alias 'default'... Cloning test database for alias 'default'... System check identified no issues (1 silenced). .Fatal Python error: Segmentation fault Current thread 0x00007fe3c3d42740 (most recent call first): File "/usr/lib/python3.11/copy.py", line 265 in _reconstruct File "/usr/lib/python3.11/copy.py", line 102 in copy File "/tmp/django/django/db/models/expressions.py", line 426 in copy File "/tmp/django/django/db/models/expressions.py", line 284 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 277 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf ...
At this point, nothing else is happening, the processes don't utilize CPU:
$ ps -ft 111241 pts/2 Tl 0:02 \_ python3.11 tests/runtests.py --parallel=2 queries.tests.ProxyQueryCleanupTest queries.tests.Queries1Tests 111243 pts/2 T 0:00 | \_ python3.11 tests/runtests.py --parallel=2 queries.tests.ProxyQueryCleanupTest queries.tests.Queries1Tests 111248 pts/2 T 0:00 | \_ python3.11 tests/runtests.py --parallel=2 queries.tests.ProxyQueryCleanupTest queries.tests.Queries1Tests
If I cancel it, it seems to ignore the killed process:
^C ---------------------------------------------------------------------- Ran 1 test in 613.663s OK Destroying test database for alias 'default'... Destroying test database for alias 'default'... Destroying test database for alias 'default'...
(I'm still trying to figure out why that other test segfaults for me)
Change History (3)
comment:1 by , 11 months ago
comment:2 by , 11 months ago
I'm afraid this will be very hard to solve as this is bug multiprocessing.Pool
.
A solution alluded to in the Python bug report is to use concurrent.futures.ProcessPoolExecutor
instead as it doesn't seem to be affected by this problem.
Would you be interested in trying to adapt the current solution?
comment:3 by , 11 months ago
Resolution: | → invalid |
---|---|
Status: | new → closed |
Thanks for the report and investigation, however I don't think it's something we should workaround in Django itself.
Marking as "invalid" because this is an issue in Python.
As a note, the segfault is apparently caused by
jedi
setting a high recursion limit globally (sigh), and it gets imported viaipdb
.