Opened 17 months ago

Last modified 17 months ago

#27086 new Cleanup/optimization

running servers.tests may hang in parallel mode on Mac OS X

Reported by: Chris Jerdonek Owned by: nobody
Component: Core (Other) Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

From master (97513269d73520d10722bbd10404be6ac4d48d07), running the following command from the tests directory hangs for me after a couple tests:

$ ./runtests.py servers.tests

This is with Python 3.5.2 on Mac OSX.

There is no error message. It just hangs. It works okay when passing --parallel 1.

Change History (11)

comment:1 Changed 17 months ago by Tim Graham

I can't reproduce on Ubuntu 14.04, Python 3.5.2. Just in case, how many parallel process do you have? I have 4 here.

comment:2 Changed 17 months ago by Chris Jerdonek

8, but it also happens when I pass --parallel 2. To narrow it down, it happens when I run the following (I believe the test setup executes, but it never gets to running the tests):

$ ./runtests.py servers.tests.LiveServerDatabase servers.tests.LiveServerViews

comment:3 Changed 17 months ago by Tim Graham

You could check if the issue is present at the time that parallel test running was added in Django and if not, bisect to find the commit that introduced the problem.

comment:4 Changed 17 months ago by Bas Peschier

Just ran the tests at the mentioned commit on my macOS Sierra public beta with a fresh 3.5.2 python environment. No problems there.

comment:5 Changed 17 months ago by Tim Graham

Resolution: worksforme
Status: newclosed

Feel free to reopen if you find that Django is at fault.

comment:6 Changed 17 months ago by Chris Jerdonek

I narrowed this down further to the following:

The hang happens when calling urllib.request.urlopen() inside LiveServerBase. And digging further, inside CPython's request.py, the hang happens when calling _scproxy._get_proxies() (see here for the code).

There seem to be known issues around _scproxy in CPython for Mac OS X users. For example, see the following two issues in CPython's bug tracker:

(1) Apple-supplied libsqlite3 on OS X is not fork safe; can cause crashes:

http://bugs.python.org/issue27126

(See this comment, in particular: http://bugs.python.org/issue27126#msg266405 )

(2) exception error in _scproxy.so when called after fork

http://bugs.python.org/issue13829

There is a chance that there are easy workarounds on the Django side that can be made to accommodate Mac OS X users.

Also, for the record I'm using 10.11.6.

Last edited 17 months ago by Chris Jerdonek (previous) (diff)

comment:7 Changed 17 months ago by Chris Jerdonek

Resolution: worksforme
Status: closednew

I was able to get things working by causing Mac OS X's auto-detection of proxy information to be skipped (so that _scproxy._get_proxies() is not called). This possibility is mentioned here in CPython's urllib.request.ProxyHandler documentation.

More detailed instructions appear in urllib.request's module docstring (see here for that part of the docstring).

To disable the auto-detection, I followed the instructions in the above-mentioned module docstring and modified LiveServerBase.urlopen() to look like the following (this is just a proof of concept):

def urlopen(self, url):
    import urllib.request
    # Pass an empty dictionary to disable auto-detecting the proxy.
    proxy_support = urllib.request.ProxyHandler({})
    opener = urllib.request.build_opener(proxy_support)
    urllib.request.install_opener(opener)

    return urlopen(self.live_server_url + url)

After doing this, the following command worked fine:

$ ./runtests.py servers.tests

Since this seems like a fundamental bug / limitation on Mac OS X (in at least some versions), it seems worth considering including Mac-specific code to enable the parallel running of tests on this platform.

Last edited 17 months ago by Chris Jerdonek (previous) (diff)

comment:8 Changed 17 months ago by Tim Graham

Component: UncategorizedCore (Other)
Triage Stage: UnreviewedAccepted
Type: UncategorizedCleanup/optimization

I wonder what's special about your system compared to other Mac OS contributors who didn't encounter the problem.

comment:9 Changed 17 months ago by Tim Graham

Summary: running servers.tests hangs in parallel moderunning servers.tests may hang in parallel mode on Mac OS X

comment:11 Changed 17 months ago by Chris Jerdonek

I don't know yet why I'm the first person to notice / report this. In case it would help, I started working on a reduced way to reproduce this (independent of the Django test suite).

http://www.evanjones.ca/fork-is-dangerous.html

Last edited 17 months ago by Tim Graham (previous) (diff)

comment:12 Changed 17 months ago by Chris Jerdonek

Oh, the person that wrote the blog post already created a "reduced test case" that I started to look for: https://github.com/evanj/fork-thread-bugs/blob/master/osx_python_crash_v1.py

Note: See TracTickets for help on using tickets.
Back to Top