Python 3.6 compatibility

[Tim Graham]

Python 3.6 final is scheduled for December 2016. This is a tracking ticket for compatibility fixes for Django submitted in the meantime.

[Tim Graham]

A couple failures bisected to ​the implementation of ​PEP 495 -- Local Time Disambiguation.

======================================================================
FAIL: test_localtime_naive (utils_tests.test_timezone.TimezoneTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/utils_tests/test_timezone.py", line 32, in test_localtime_naive
    timezone.localtime(datetime.datetime.now())
AssertionError: ValueError not raised

======================================================================
FAIL: test_make_naive (utils_tests.test_timezone.TimezoneTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/utils_tests/test_timezone.py", line 140, in test_make_naive
    timezone.make_naive(datetime.datetime(2011, 9, 1, 13, 20, 30), EAT)
AssertionError: ValueError not raised

[Aymeric Augustin]

Likely the time zone docs need some small additions to acknowledge the existence of PEP 495 -- even if the code can't rely on it until we drop support for Python < 3.6.

[GitHub <noreply@…>]

In 35225e2a:

Refs #27025 -- Fixed a servers test on Python 3.6.

After ​https://hg.python.org/cpython/rev/4ea79767ff75/,
test_strips_underscore_headers fails with:
'Stub' object has no attribute 'sendall'.

[GitHub <noreply@…>]

In a7a7ecd2:

Refs #27025 -- Fixed a couple timezone tests for Python 3.6.

Reflects behavior changes in PEP 495 (Local Time Disambiguation).

[Tim Graham]

Django's test are passing with master @ 2b759c94c562c9ee9b6ca970739be15014050fda and cpython master @ ​47f6c10084ed9aa8a08dbac20d5f6b814063c2df.

[Tim Graham <timograham@…>]

In 16202863:

Refs #27025 -- Fixed tests for the new ModuleNotFoundError in Python 3.6.

​http://bugs.python.org/issue15767

[Tim Graham <timograham@…>]

In 49412f55:

Refs #27025 -- Fixed a test for the new re.RegexFlag in Python 3.6.

​http://bugs.python.org/issue28082

[Tim Graham <timograham@…>]

In 8119b679:

Refs #27025 -- Fixed "invalid escape sequence" warnings in Python 3.6.

​http://bugs.python.org/issue27364

[Tim Graham <timograham@…>]

In e43ea36:

Refs #27025 -- Fixed a timezone test for Python 3.6.

Reflects behavior changes in PEP 495 (Local Time Disambiguation).

[Tim Graham <timograham@…>]

In b5aac66b:

Refs #27025 -- Fixed ArrayField querying on Python 3.6.

Python 3.6 parses strings like '0_1' as numeric literals.
​http://bugs.python.org/issue26331

[Tim Graham]

I setup a more complete test environment that includes all of the test suite dependencies and identified a few more failures fixed in the above commits.

2 failures remain as of Django b5aac66b28c615b2bda63548cbd695dbb5a0c381 and cpython ​d614d687d9db1100d0a4ec8c67e0cb3ce773342b due to a sqlparse incompatibility with Python 3.6. I submitted a ​PR to fix it and it's merged awaiting the next release.

[Tim Graham]

cpython fixed compatibility with sqlparse in ​https://hg.python.org/cpython/rev/1a2b8398f045/ so a new sqlparse release isn't needed and all tests are passing as of now.

[Andrii Soldatenko]

just tested with last Django master e044026dce063ec53c16d0f755ec75eb8c84b318 and Python 3.6.0b4 works for me.
BTW where can I find how to add or check test results for Python3.6 and latest version of Django?

[Tim Graham]

I have a local Jenkins running with the cpython 3.6 branch and Django master that I check every couple days. I don't think we'll add Python 3.6 compatibility to older versions of Django such as 1.10 but we'll discuss it on the DevelopersMailingList if anyone feel it's important.

[Anthony King]

py3.6 breaks calling super() (in the python3 style) inside Models with:
RuntimeError: super(): empty __class__ cell

This is also present on 1.8 and 1.10

class ModelToBreak(models.Model):
    def save(self, *args, **kwargs):
        return super().save(*args, **kwargs)

>>> m = ModelToBreak()
>>> m.save()
Traceback (most recent call last):
  File "/home/cybo/work/3.6-test/testing/d/tests.py", line 9, in test_empty_class_fail
    m.save()
  File "/home/cybo/work/3.6-test/testing/d/models.py", line 5, in save
    return super().save(*args, **kwargs)
RuntimeError: super(): empty __class__ cell

[Tim Graham]

Thanks, the relevant commit in cpython is ​https://hg.python.org/cpython/rev/feb1ae9d5381.

I'm not sure if Django or cpython is at fault.

[Tim Graham]

After some discussion on the ​cpython issue, I've created a ​PR to address the issue, but cpython may also restore backwards-compatibility and add a deprecation path.

[Anthony King]

Thanks. I'll check all other MetaClasses for similar behaviour today or tomorrow.
Will also add Py3 tests for them.

[Tim Graham]

I'm not particularly worried about tests since cpython added a deprecation path. Also, when we drop Python 2 compatibility in master in January, we'll probably change all super() calls to the new style.

[GitHub <noreply@…>]

In dd99e69f:

Refs #27025 -- Fixed Python 3.6 deprecation warning for empty model super() calls.

​https://bugs.python.org/issue23722

Thanks Nick Coghlan for advice and review.

[Tim Graham <timograham@…>]

In 4701abd5:

Refs #27025 -- Documented Python 3.6 compatibility for Django 1.11.

[Tim Graham <timograham@…>]

In 72be3152:

Refs #27025 -- Added Python 3.6 to tox.ini.

[Tim Graham <timograham@…>]

In d4d79d0f:

Refs #27025 -- Fixed "invalid escape sequence" warning in auth_tests on Python 3.6.

[Tim Graham <timograham@…>]

In e34cce79:

[1.11.x] Refs #27025 -- Fixed "invalid escape sequence" warning in auth_tests on Python 3.6.

Backport of d4d79d0f200357b28419203557cc61c8168316ab from master

[Tim Graham <timograham@…>]

In f0ffa3f4:

Refs #27025, #28593 -- Fixed "invalid escape sequence" warnings in urls/resolvers.py.

[Tim Graham <timograham@…>]

In 5091bb6:

[2.0.x] Refs #27025, #28593 -- Fixed "invalid escape sequence" warnings in urls/resolvers.py.

Backport of f0ffa3f4ea277f9814285085fde20baff60fc386 from master

[Jeroen Pulles]

Would it be possible to backport dd99e69fa8f89263d0396f23e0db9aa8fa667b01 (metaclass magic) to Django 1.8?

I'm seeing quite a few warnings for this, hiding the more useful warnings...

[Tim Graham]

No, Django 1.8 doesn't have official support for Python 3.6 and only receives data loss and security fixes its end-of-life in April 2018.

Perhaps you can use warnings.filterwarnings() to hide the warnings.

[Jeroen Pulles]

Okay. For the record: Adding this snippet to my development settings in my otherwise meta-class-free project, works for me:

import warnings
warnings.filterwarnings(
    action="ignore", 
    message=".*Was __classcell__ propagated to type[.]__new__.*")

[GitHub <noreply@…>]

In 717ee63e:

Refs #27025 -- Removed obsolete sqlite3 transaction management workaround for Python 3.6+.

Obsolete per ​https://bugs.python.org/issue10740#msg274816.