Opened 8 years ago

Closed 4 years ago

Last modified 4 years ago

#27086 closed Cleanup/optimization (fixed)

running servers.tests may hang in parallel mode on Mac OS X

Reported by: Chris Jerdonek Owned by: Carlton Gibson
Component: Core (Other) Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

From master (97513269d73520d10722bbd10404be6ac4d48d07), running the following command from the tests directory hangs for me after a couple tests:

$ ./runtests.py servers.tests

This is with Python 3.5.2 on Mac OSX.

There is no error message. It just hangs. It works okay when passing --parallel 1.

Change History (22)

comment:1 by Tim Graham, 8 years ago

I can't reproduce on Ubuntu 14.04, Python 3.5.2. Just in case, how many parallel process do you have? I have 4 here.

comment:2 by Chris Jerdonek, 8 years ago

8, but it also happens when I pass --parallel 2. To narrow it down, it happens when I run the following (I believe the test setup executes, but it never gets to running the tests):

$ ./runtests.py servers.tests.LiveServerDatabase servers.tests.LiveServerViews

comment:3 by Tim Graham, 8 years ago

You could check if the issue is present at the time that parallel test running was added in Django and if not, bisect to find the commit that introduced the problem.

comment:4 by Bas Peschier, 8 years ago

Just ran the tests at the mentioned commit on my macOS Sierra public beta with a fresh 3.5.2 python environment. No problems there.

comment:5 by Tim Graham, 8 years ago

Resolution: worksforme
Status: newclosed

Feel free to reopen if you find that Django is at fault.

comment:6 by Chris Jerdonek, 8 years ago

I narrowed this down further to the following:

The hang happens when calling urllib.request.urlopen() inside LiveServerBase. And digging further, inside CPython's request.py, the hang happens when calling _scproxy._get_proxies() (see here for the code).

There seem to be known issues around _scproxy in CPython for Mac OS X users. For example, see the following two issues in CPython's bug tracker:

(1) Apple-supplied libsqlite3 on OS X is not fork safe; can cause crashes:

http://bugs.python.org/issue27126

(See this comment, in particular: http://bugs.python.org/issue27126#msg266405 )

(2) exception error in _scproxy.so when called after fork

http://bugs.python.org/issue13829

There is a chance that there are easy workarounds on the Django side that can be made to accommodate Mac OS X users.

Also, for the record I'm using 10.11.6.

Last edited 8 years ago by Chris Jerdonek (previous) (diff)

comment:7 by Chris Jerdonek, 8 years ago

Resolution: worksforme
Status: closednew

I was able to get things working by causing Mac OS X's auto-detection of proxy information to be skipped (so that _scproxy._get_proxies() is not called). This possibility is mentioned here in CPython's urllib.request.ProxyHandler documentation.

More detailed instructions appear in urllib.request's module docstring (see here for that part of the docstring).

To disable the auto-detection, I followed the instructions in the above-mentioned module docstring and modified LiveServerBase.urlopen() to look like the following (this is just a proof of concept):

def urlopen(self, url):
    import urllib.request
    # Pass an empty dictionary to disable auto-detecting the proxy.
    proxy_support = urllib.request.ProxyHandler({})
    opener = urllib.request.build_opener(proxy_support)
    urllib.request.install_opener(opener)

    return urlopen(self.live_server_url + url)

After doing this, the following command worked fine:

$ ./runtests.py servers.tests

Since this seems like a fundamental bug / limitation on Mac OS X (in at least some versions), it seems worth considering including Mac-specific code to enable the parallel running of tests on this platform.

Last edited 8 years ago by Chris Jerdonek (previous) (diff)

comment:8 by Tim Graham, 8 years ago

Component: UncategorizedCore (Other)
Triage Stage: UnreviewedAccepted
Type: UncategorizedCleanup/optimization

I wonder what's special about your system compared to other Mac OS contributors who didn't encounter the problem.

comment:9 by Tim Graham, 8 years ago

Summary: running servers.tests hangs in parallel moderunning servers.tests may hang in parallel mode on Mac OS X

comment:11 by Chris Jerdonek, 8 years ago

I don't know yet why I'm the first person to notice / report this. In case it would help, I started working on a reduced way to reproduce this (independent of the Django test suite).

http://www.evanjones.ca/fork-is-dangerous.html

Last edited 8 years ago by Tim Graham (previous) (diff)

comment:12 by Chris Jerdonek, 8 years ago

Oh, the person that wrote the blog post already created a "reduced test case" that I started to look for: https://github.com/evanj/fork-thread-bugs/blob/master/osx_python_crash_v1.py

comment:13 by Kees Hink, 5 years ago

Has patch: set

Here is a documentation improvement showing a workaround: https://github.com/django/django/pull/11203

comment:14 by Tim Bell, 5 years ago

I have encountered the same problem (on MacOS 10.14.4 Mojave) and confirm that the solution works for me. This would be a useful addition to the documentation.

comment:15 by Tim Bell, 5 years ago

Patch needs improvement: set

comment:16 by Tim Bell, 5 years ago

Patch needs improvement: unset

comment:17 by Tim Bell, 5 years ago

Triage Stage: AcceptedReady for checkin

comment:18 by Carlton Gibson, 5 years ago

Resolution: wontfix
Status: newclosed

Closing as per discussion on PR: lack of a decent reproduce, plenty of Google results for the error, not our responsibility, and likely a problem that's been resolved in recent releases (but c.f. lack of reproduce...)

comment:19 by Carlton Gibson, 4 years ago

Resolution: wontfix
Status: closednew

I'm at the sprints at DjangoCon. Many new contributors have hit this, so I think the doesn't reproduce worry is passed. I'm going to re-open this accordingly. We should doc so new contributors are able to get going.

Related is #30806 (which is targeting running tox specifically.)

comment:20 by Carlton Gibson, 4 years ago

Owner: changed from nobody to Carlton Gibson
Status: newassigned
Triage Stage: Ready for checkinAccepted

I just want to adjust this, so moving to Accepted and assigning to myself.

comment:21 by Carlton Gibson <carlton@…>, 4 years ago

Resolution: fixed
Status: assignedclosed

In 5d9cf79b:

Fixed #27086 -- Doc'd fix for test suite hang on macOS.

comment:22 by Carlton Gibson <carlton.gibson@…>, 4 years ago

In fffce1a:

[3.0.x] Fixed #27086 -- Doc'd fix for test suite hang on macOS.

Backport of 5d9cf79baf07fc4aed7ad1b06990532a65378155 from master

comment:23 by Carlton Gibson <carlton.gibson@…>, 4 years ago

In 4f7ba25e:

[2.2.x] Fixed #27086 -- Doc'd fix for test suite hang on macOS.

Backport of 5d9cf79baf07fc4aed7ad1b06990532a65378155 from master

Note: See TracTickets for help on using tickets.
Back to Top