Opened 5 months ago

Closed 5 months ago

Last modified 5 months ago

#35010 closed Bug (invalid)

Parallel test runner hangs if one of the tests segfault

Reported by: Michał Górny Owned by: nobody
Component: Testing framework Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When the tests are being run in parallel, and one of the tests cause a segfault, the test runner hangs:

$ PYTHONPATH=. python3.11 tests/runtests.py --parallel=2 queries.tests.{ProxyQueryCleanupTest,Queries1Tests}                                                                                                                              
Testing against Django installed in '/tmp/django/django' with up to 2 processes
Found 78 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (1 silenced).
.Fatal Python error: Segmentation fault

Current thread 0x00007fe3c3d42740 (most recent call first):
  File "/usr/lib/python3.11/copy.py", line 265 in _reconstruct
  File "/usr/lib/python3.11/copy.py", line 102 in copy
  File "/tmp/django/django/db/models/expressions.py", line 426 in copy
  File "/tmp/django/django/db/models/expressions.py", line 284 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 277 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  File "/tmp/django/django/db/models/sql/where.py", line 279 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 275 in _resolve_node
  File "/tmp/django/django/db/models/sql/where.py", line 283 in resolve_expression
  File "/tmp/django/django/db/models/sql/query.py", line 1203 in resolve_expression
  File "/tmp/django/django/db/models/sql/where.py", line 268 in _resolve_leaf
  ...

At this point, nothing else is happening, the processes don't utilize CPU:

$ ps -ft
 111241 pts/2    Tl     0:02          \_ python3.11 tests/runtests.py --parallel=2 queries.tests.ProxyQueryCleanupTest queries.tests.Queries1Tests
 111243 pts/2    T      0:00          |   \_ python3.11 tests/runtests.py --parallel=2 queries.tests.ProxyQueryCleanupTest queries.tests.Queries1Tests
 111248 pts/2    T      0:00          |   \_ python3.11 tests/runtests.py --parallel=2 queries.tests.ProxyQueryCleanupTest queries.tests.Queries1Tests

If I cancel it, it seems to ignore the killed process:

^C
----------------------------------------------------------------------
Ran 1 test in 613.663s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

(I'm still trying to figure out why that other test segfaults for me)

Change History (3)

comment:1 by Michał Górny, 5 months ago

As a note, the segfault is apparently caused by jedi setting a high recursion limit globally (sigh), and it gets imported via ipdb.

comment:2 by Simon Charette, 5 months ago

I'm afraid this will be very hard to solve ​as this is bug multiprocessing.Pool.

A solution alluded to in the Python bug report is to use concurrent.futures.ProcessPoolExecutor instead as it doesn't seem to be affected by this problem.

Would you be interested in trying to ​adapt the current solution?

comment:3 by Mariusz Felisiak, 5 months ago

Resolution: → invalid
Status: new → closed

Thanks for the report (Michał) and investigation (Simon), however I don't think it's something we should workaround in Django itself.

Marking as "invalid" because this is an issue in Python.

Last edited 5 months ago by Mariusz Felisiak (previous) (diff)
Note: See TracTickets for help on using tickets.
Back to Top