Opened 4 years ago

Last modified 7 months ago

#27734 new Cleanup/optimization

Add a helpful error message when a parallel test worker is assigned an unexpected index

Reported by: Adam Wróbel Owned by: nobody
Component: Testing framework Version: 1.10
Severity: Normal Keywords: parallel
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


I've had this problem since upgrading to Python 3.6 on macOS Sierra – when I run my tests in parallel many of them blow up trying to use database with index that is higher than the number of parallel test processes. On my MacBook the default process number is 4 and tests fail trying to connect to test_my_db_5. On my iMac the default process number is 8 and test fail connecting to test_my_db_9. The same thing happens if I set the number of processes explicitly, e.g. --parallel=2. Higher number of processes yields less failures. If I explicitly set the number of processes to a number that is greater than the number of my test files then the tests succeed.

Traceback (most recent call last):
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/test/", line 216, in __call__
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/test/", line 908, in _post_teardown
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/test/", line 943, in _fixture_teardown
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/core/management/", line 130, in call_command
    return command.execute(*args, **defaults)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/core/management/", line 345, in execute
    output = self.handle(*args, **options)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/core/management/commands/", line 54, in handle
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/core/management/", line 15, in sql_flush
    tables = connection.introspection.django_table_names(only_existing=True, include_views=False)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 88, in django_table_names
    existing_tables = self.table_names(include_views=include_views)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 55, in table_names
    with self.connection.cursor() as cursor:
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 233, in cursor
    cursor = self.make_cursor(self._cursor())
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 204, in _cursor
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 199, in ensure_connection
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/utils/", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 199, in ensure_connection
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/base/", line 171, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/django/db/backends/postgresql/", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/Users/amw/.virtualenvs/envname/lib/python3.6/site-packages/psycopg2/", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: FATAL:  database "test_my_db_5" does not exist

My requirements.txt file:


Change History (9)

comment:1 Changed 4 years ago by Tim Graham

I haven't see that issue in my Python 3.6 testing. Could you investigate and tell us why Django is at fault?

comment:2 Changed 4 years ago by Adam Wróbel

I don't know if it's Django's fault, but I do know that it's Django that is creating test databases and configuring connection parameters. If you take a look on the backtrace you'll see that the only other Python package on it is psycopg2.

Parallel nature of this problem makes it hard to investigate with my debugger. The exception occurs in the child process, but I don't know where the incorrect information is generated. I was hoping for a hint for where to start the search.

comment:3 Changed 4 years ago by Tim Graham

The parallel test runner is in django/test/ Can you provide a small sample project (without any third-party dependencies) that reproduces the issue?

comment:4 Changed 4 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

comment:5 Changed 3 years ago by David Breeden

I'm having this problem. It's not Django's fault, although there could be better error reporting.

--parallel creates N databases for the N workers to use. Each worker claims the database it will use by incrementing a shared counter. When the multiprocessing.Pool is initialized with N processes, the workers are assigned databases 1 through N.

The problem happens when a worker exits. Normally, the workers in a pool will stay alive for the whole life of the pool. In my case, workers are exiting because of seg faults. When this happens, the pool repopulates with new workers. When the first new worker is initialized, it increments the counter to N + 1 and tries to use that database, and this error results.

Only some of my tests cause the seg fault, so if I run --parallel with other tests, it works fine.

It would have saved some time diagnosing the issue if Django raised an error when a worker incremented the counter over N, which seems like a reasonable thing for the initializer to do.

comment:6 Changed 3 years ago by Aymeric Augustin

Resolution: needsinfo
Status: closednew
Triage Stage: UnreviewedAccepted

I agree it would be nice to have a better error message in this case.

comment:7 Changed 3 years ago by Tim Graham

Summary: Parallel tests try to use database with too high indicesAdd a helpful error message when a parallel test worker is assigned an unexpected index
Type: BugCleanup/optimization

comment:8 Changed 3 years ago by Adam Wróbel

I don't think my issue was with segfaulting processes, but it's certain that something caused the process to be restarted. Better error message might be nice to have, but won't be helpful unless it helps determining what caused the process segfault/restart.

Maybe there would be a way to reuse a connection of a dead process?

comment:9 Changed 7 months ago by Hameed Gifford

To anyone else who's still having this problem, it seems to be specific to OSX. Or at least this reason and fix is.

Putting this somewhere in your code before the test runner is initialized (settings worked good for me) fixes the issue:

import os
import platform
import sys

if 'test' in sys.argv and platform.system().lower() == 'darwin':
    os.environ.setdefault('OBJC_DISABLE_INITIALIZE_FORK_SAFETY', 'YES')
Note: See TracTickets for help on using tickets.
Back to Top