Cleanups for when we drop Python 2 compatibility

[Tim Graham]

This is a tracking ticket of things that we can remove when we drop Python 2 compatibility in Django 2.0. Please edit the description of the ticket as you come across new items.

• django.core.mail.message.make_msgid() #23905
• django.dispatch.weakref_backports
• django.http.cookie workarounds (remains for Python 3.4)
• django.utils.2to3_fixes
• django.utils.decorators.ContextDecorator
• django.utils.encoding
  • @python_2_unicode_compatible
  • force/smart _unicode
  • either force/smart _text or force/smart _str ​PR
• django.utils.html_parser.use_workaround
• django.utils.http functions like urlquote_plus -- I think the versions of these functions on Python 3 don't have the unicode characters bug we are working around ​PR
• Stop using django.utils.lru_cache
• Figure the deprecation plan for django.utils.lru_cache (staying for now)
• django.utils.six
• from __future__ import unicode_literals
• str() stuff for environment variables, e.g. 0bfb53866199f366ed140d49938fd185e5898156
• str() stuff for type(name) and __name__
• Inheriting from object in django.core.servers.basehttp (and perhaps other places) ala 4ee06ec3fc8e94d164afbd2f9c880c60c658a9ac
• git grep 'long int' (mostly docs)
• django.utils._os [npath,upath]
• In tests: contextlib.closing(self.urlopen (contextlib.closing no longer needed)
• Support for pysqlite (doesn't support Python 3)
• Replace super(ClassName, self) with super() ​PR
• Remove # -*- coding: utf-8 -*- source file encoding (that's the default on Python 3)
• Evaluate replacement of custom __del__ methods by ​weakref.finalize
• Remove __ne__ from objects already defining a __eq__
• Remove note about PYTHONHASHSEED (see #26243)
• Remove Field.creation_counter (replace by metaclass __prepare__ returning OrderedDict(), ​https://docs.python.org/3/reference/datamodel.html#preparing-the-class-namespace) (see comment:95)
• django.test.utils.reset_warning_registry()
• Replace errno checks by IOError subclasses defined by ​PEP 3151 ​PR
• django.utils.glob
• django.utils.cookies.SimpleCookie (remains for Python 3.4 compatibility)
• django.test.mock
• Replace tempfile.mkdtemp + remove with tempfile.TemporaryDirectory context manager
• Replace io.open() by a plain open()
• Evaluate need for assertRegex, assertRaisesRegex in tests. Some usage is merely to account for differences in messages between Python 2 and 3.
• re.U, re.UNICODE (default behavior of Python 3) - ​PR 7879

[Wojtek Ruszczewski]

Various nonlocal workarounds (​example).

[Anssi Kääriäinen]

I believe we can get rid of the Field.creation_counter hack. In Python 3 it is possible to store the attrs in a sorted dictionary. See ​http://stackoverflow.com/questions/4459531/how-to-read-class-attributes-in-the-same-order-as-declared.

[Curtis Maloney]

In far too many tests we rely on the fact six aliases builtins [like input] instead of using mock.

I submitted a patch recently to clean up a couple of these I found, but I am finding more.

[Claude Paroz]

Note that I have a local branch which does some boring removals (unicode_literals, utf-8 preambles, some six usage). So ping me as soon as the Django 2.0 deprecated stuff is removed and I'll offer a patch for that.

[Asif Saifuddin Auvi]

I am also working to remove python2 workarounds by following a module by module approach.

[Aymeric Augustin]

If Claude has already prepared these changes, perhaps you should wait for his patch instead of duplicating his work?

[Asif Saifuddin Auvi]

Replying to Aymeric Augustin:

If Claude has already prepared these changes, perhaps you should wait for his patch instead of duplicating his work?

Yes I talked with Claude on IRC to pause before his work is merged.

[Josh Smeaton]

Replying to Anssi Kääriäinen:

I believe we can get rid of the Field.creation_counter hack. In Python 3 it is possible to store the attrs in a sorted dictionary. See ​http://stackoverflow.com/questions/4459531/how-to-read-class-attributes-in-the-same-order-as-declared.

We could also remove this when we support python 3.6, as class attributes retain their order. I'm guessing this is a consequence (or the cause of) dictionaries retaining insertion order. ​https://docs.python.org/3.6/whatsnew/3.6.html#pep-520-preserving-class-attribute-definition-order

[Claude Paroz <claude@…>]

In d7b9aaa:

Refs #23919 -- Removed encoding preambles and future imports

[Claude Paroz <claude@…>]

In f3c43ad:

Refs #23919 -- Removed python_2_unicode_compatible decorator usage

[Tim Graham]

​PR for removing Python 2 references from docs.

[Claude Paroz <claude@…>]

In c716fe87:

Refs #23919 -- Removed six.PY2/PY3 usage

Thanks Tim Graham for the review.

[GitHub <noreply@…>]

In f6acd1d:

Refs #23919 -- Removed Python 2 notes in docs.

[Claude Paroz <claude@…>]

In 7b2f2e74:

Refs #23919 -- Removed six.<various>_types usage

Thanks Tim Graham and Simon Charette for the reviews.

[Claude Paroz <claude@…>]

In 2b281cc3:

Refs #23919 -- Removed most of remaining six usage

Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 3cc5f01d:

Refs #23919 -- Stopped using django.utils.lru_cache().

[Tim Graham <timograham@…>]

In eb422e47:

Refs #23919 -- Removed obsolete ne() methods.

ne() defaults to the opposite of eq() on Python 3
when it doesn't return NotImplemented.

[Tim Graham <timograham@…>]

In a5563963:

Refs #23919 -- Replaced io.open() with open().

io.open() is an alias for open() on Python 3.

[Claude Paroz <claude@…>]

In cecc079:

Refs #23919 -- Stopped inheriting from object to define new style classes.

[GitHub <noreply@…>]

In 5320fa7:

Refs #23919 -- Removed obsolete contextlib.closing() calls (for Python 2).

[GitHub <noreply@…>]

In e5c67f0:

Refs #23919 -- Removed reset_warning_registry() workaround for Python < 3.4.2.

[Simon Charette <charette.s@…>]

In 4c5ed3e6:

Refs #23919 -- Removed nonzero() methods (for Python 2).

Thanks Tim for the review.

[Tim Graham <timograham@…>]

In 41e0033c:

Refs #23919 -- Removed usage of django.utils.decorators.ContextDecorator.

[Simon Charette <charette.s@…>]

In 9695b14:

Refs #23919 -- Removed str() conversion of type and method name.

[GitHub <noreply@…>]

In d4bb3759:

Refs #23919 -- Removed obsolete compare_digest() and pbkdf2() implementations.

[GitHub <noreply@…>]

In 9e917cc2:

Fixed #23905, refs #23919 -- Used make_msgid() from stdlib.

[GitHub <noreply@…>]

In 9d27478:

Refs #23919 -- Removed docs references to long integers.

[Tim Graham <timograham@…>]

In 9ee47ce7:

Refs #23919 -- Removed enum ImportError handling for Python 2.

[Tim Graham <timograham@…>]

In eb0b921c:

Refs #23919 -- Removed SessionBase.iterkeys(), itervalues(), iteritems().

These methods only work on Python 2.

[Tim Graham <timograham@…>]

In fedda6d:

Refs #23919 -- Removed Python 2 version check in django.http.cookie.

[Tim Graham <timograham@…>]

In bf1c9570:

Refs #23919 -- Removed Python 2 workaround for hashing Oracle params (refs #27632).

[Claude Paroz <claude@…>]

In dc8834ca:

Refs #23919 -- Removed unneeded force_str calls

[Aymeric Augustin <aymeric.augustin@…>]

In 109b33f:

Refs #23919 -- Simplified assertRaisesRegex()'s that accounted for Python 2.

[GitHub <noreply@…>]

In 4e729fea:

Refs #23919 -- Removed django.utils._os.upath()/npath()/abspathu() usage.

These functions do nothing on Python 3.

[Claude Paroz <claude@…>]

In 042b735:

Refs #23919 -- Removed unneeded str() calls

[Tim Graham <timograham@…>]

In 7aba691:

Refs #23919 -- Removed django.test.mock Python 2 compatibility shim.

[Claude Paroz <claude@…>]

In 289fc1bf:

Refs #23919 -- Removed str_prefix usage

[Tim Graham <timograham@…>]

In 1b06d5e6:

Refs #23919 -- Removed pysqlite support (it's Python 2 only).

[GitHub <noreply@…>]

In 9e6e32bf:

Refs #23919 -- Removed django.utils.decorators.available_attrs() usage.

It's only needed to workaround a bug on Python 2.

[Tim Graham <timograham@…>]

In c222122:

Refs #23919 -- Removed re.U and re.UNICODE (default on Python 3).

[GitHub <noreply@…>]

In d170c63:

Refs #23919 -- Removed misc references to Python 2.

[Claude Paroz <claude@…>]

In 6e55e1d:

Refs #23919 -- Replaced six.reraise by raise

[GitHub <noreply@…>]

In 435e4bf3:

Refs #23919 -- Removed traceback setting needed for Python 2.

Partially reverted refs #25761 and refs #16245.

[Tim Graham <timograham@…>]

In 5b95d42:

Refs #23919 -- Removed a MySQLdb workaround (refs #6052) for Python 2.

[GitHub <noreply@…>]

In 632c4ffd:

Refs #23919 -- Replaced errno checking with PEP 3151 exceptions.

[GitHub <noreply@…>]

In 2d96c027:

Refs #23919 -- Removed obsolete MySQLdb references.

[Tim Graham <timograham@…>]

In dc165ec8:

Refs #23919 -- Replaced super(ClassName, self) with super() in docs.

[Tim Graham <timograham@…>]

In d6eaf7c:

Refs #23919 -- Replaced super(ClassName, self) with super().

[GitHub <noreply@…>]

In 1c466994:

Refs #23919 -- Removed misc Python 2/3 references.

[Tim Graham <timograham@…>]

In d1bab24:

Refs #23919, #27778 -- Removed obsolete mentions of unicode.

[Claude Paroz <claude@…>]

In fee42fd9:

Refs #23919 -- Replaced usage of django.utils.http utilities with Python equivalents

Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 6478e07:

Refs #23919 -- Replaced tempfile.mkdtemp() with TemporaryDirectory() context manager.

[GitHub <noreply@…>]

In e07e743e:

Refs #23919 -- Used DeclarativeFieldsMetaclass.prepare() for tracking form field order.

[GitHub <noreply@…>]

In 9e9e7373:

Refs #23919 -- Removed an obsolete test for a Python 2 code path (refs #15662).

Fixed #21628 by removing the last usage of the imp module.

[Josh Smeaton]

I had an attempt at dropping the creation_counter for model fields, but I *think* it might be more costly than the current method, considering so many things rely on the order being consistent. My attempt was here, with associated discussion I'll copy some of below: ​https://github.com/django/django/pull/7983

I'm not sure this is worth continuing with, at least not in the same way. Too many systems rely on the fields being ordered, which isn't going to be possible without maintaining another list of fields such as definition_order = []. This can be done for local fields, but then proxy/parent fields will need to be returned in the same definition order. This might be something like:

parent_fields + [field for field in self.definition_order if field in local_fields]

Then we'd need to account for private and hidden fields. But what's the point? We'd now be storing a separate list of all field references for each model, rather than a counter (int) on each field, and reordering fields in _get_fields() whether or not they need to be ordered. Of course I'd be happy to see a better solution that worked, I'm just giving up on this particular method for the moment.

[Tim Graham]

I also looked briefly at models field and manager usage of creation_counter but it wasn't obvious if the __prepare__() approach was feasible or if it would be simpler. I think we can defer that task for now.

[Claude Paroz <claude@…>]

In c688336e:

Refs #23919 -- Assumed request COOKIES and META are str

[Claude Paroz <claude@…>]

In 52138b1f:

Refs #23919 -- Removed usage of obsolete SafeBytes class

The class will be removed as part of #27753.
Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 84126f27:

Refs #23919 -- Removed unneeded code in force_text().

Unneeded since 7b2f2e74adb36a4334e83130f6abc2f79d395235.

[Tim Graham <timograham@…>]

In 8838d4d:

Refs #23919 -- Replaced kwargs.pop() with keyword-only arguments.

[Collin Anderson]

Can we remove empty init.py (and init.py-tpl) files?

[Aymeric Augustin]

IIRC there's a risk to making everything a namespace package: it means that if you have conflicting package / module names, they're somehow merged instead of having the first one shadow the other. (Perhaps someone more familiar with the topic can describe this more precisely.) I think it's safer to keep the __init__ files.

[Nick Pope]

Given that Django 2.0 is targeting Python 3.5+, are there other things that could be cleaned up that were added to support <3.5 or could be used now that 3.5 is the minimum version?

For example, there are many places with the following pattern:

try:
    ...
except ...:
    pass

These could be replaced with ​contextlib.suppress() which is available since 3.4.

Other potentially interesting things are:

• Use of ​unpacking generalisations
• Use of ​type hints and a type checker when building.
• Use of ​os.scandir() instead of os.listdir().
• Use of the ​enum and ​pathlib modules where appropriate.
• ​contextlib.redirect_stdout() and ​contextlib.redirect_stderr()

There may be more - this was just a quick look at the What's New pages for 3.4 and 3.5.

[Tim Graham]

Usage of Python 3 features could be proposed in new tickets (e.g. #27804). Master is still supporting Python 3.4 right now as per ​django-developers discussion.

[GitHub <noreply@…>]

In 2d899ce:

Refs #23919 -- Removed a Python 2 code path in force_text().

Reverted the obsolete fix and tests for refs #12302.

[GitHub <noreply@…>]

In 21f13ff5:

Refs #23919 -- Removed an used block in ExceptionReporter.get_traceback_data().

The test from refs #20368 only runs this block on Python 2.

[Tim Graham <timograham@…>]

In 500532c9:

Refs #23919 -- Removed default 'utf-8' argument for str.encode()/decode().

[Collin Anderson]

Should we deprecate cache.has_key() ?.... Nevermind, it takes the "version" argument....

[Collin Anderson]

The "Python 3 is recommended" text here could probably be rephrased:

​https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-should-i-use-with-django

[Tim Graham <timograham@…>]

In 4cffa9a1:

Fixed #27878, refs #23919 -- Used python3 shebangs.

[Tim Graham <timograham@…>]

In 3dcc351:

Refs #23919 -- Used yield from.

[Tim Graham <timograham@…>]

In 784a53b:

Reverted "Fixed #27878, refs #23919 -- Used python3 shebangs."

This reverts commit 4cffa9a1ffb37d4de7e99a851a9ed87b3c02d229.

[Tim Graham <timograham@…>]

In ed0cbc8:

Refs #23919 -- Removed some Python 2 compatibility code and comments.

[GitHub <noreply@…>]

In 84dcd162:

Refs #23919 -- Used "raise from" instead of cause in reraising backend-specific database exceptions.

Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 9b538ba:

Refs #23919 -- Removed File's Python 2 proxied methods.

[Claude Paroz <claude@…>]

In 59f8118c:

Refs #23919 -- Removed force_text() in Python deserializer needed only on Python 2

[Tim Graham <timograham@…>]

In 3eb3907b:

Refs #23919 -- Replaced stray super(ClassName, self) with super().

[GitHub <noreply@…>]

In 3d0a0ecd:

Refs #23919 -- Removed support for broken Model.str() in Model.repr().

Returning invalid bytestrings in str() is unlikely in Python 3.

[Tim Graham <timograham@…>]

In 081e787:

Refs #23919 -- Stopped inheriting from object to define new style classes.

Tests and docs complement to cecc079168e8669138728d31611ff3a1e7eb3a9f.

[Tim Graham <timograham@…>]

In 4af88cc:

Refs #23919 -- Removed Python 2 reference in django_bash_completion.

[Sergey Fedoseev <fedoseev.sergey@…>]

In 59a4b12a:

Refs #23919 -- Removed LazyObject.getstate() needed only on Python 2.

[Tom Forbes]

I was looking at potential cleanups I could help with and noticed a lot of os.path functions that could be replaced with pathlib - we have 219 occurrences of os.path (excluding usages in tests).

Is this something worth spending time on? It's pretty easy to convert, and it makes path related code a lot more readable IMO (no need for nested, multiple os.path.* calls in a single line for example). In 3.5 and above there is also a read_text method on paths which is quite useful in some cases.

[Tim Graham]

Sure, start small and we can see if the changes look like an improvement.

[Tim Graham <timograham@…>]

In 462f158:

Refs #23919 -- Updated contrib.admin's compress.py to use pathlib.

[Tim Graham <timograham@…>]

In 5b1c3896:

Refs #23919 -- Replaced usage of django.utils.functional.curry() with functools.partial()/partialmethod().

[Tim Graham <timograham@…>]

In d896809a:

Refs #23919 -- Removed unneeded float()/int() calls.

[Tim Graham <timograham@…>]

In e3a30f2d:

Refs #23919 -- Corrected django.utils.http.urlencode()'s documented signature.

Follow up to fee42fd99ee470528858c2ccb3621135c30ec262.

[Tim Graham <timograham@…>]

In 3eb447d8:

[2.0.x] Refs #23919 -- Corrected django.utils.http.urlencode()'s documented signature.

Follow up to fee42fd99ee470528858c2ccb3621135c30ec262.

Backport of e3a30f2d16d65e7f0b870e997e92425a2f795587 from master

[Tim Graham <timograham@…>]

In 23e551c:

Refs #23919 -- Updated references to urllib.quote() to Python 3 location.

[Tim Graham <timograham@…>]

In 542e6b02:

[2.0.x] Refs #23919 -- Updated references to urllib.quote() to Python 3 location.

Backport of 23e551ce6c7956e4c143357643dab2e89d30bfe0 from master

[GitHub <noreply@…>]

In 7945bc1d:

Refs #23919 -- Replaced doc references to django.utils.http.urlquote() and urlquote_plus() with standard library alternatives.

Follow up to fee42fd99ee470528858c2ccb3621135c30ec262.

[Tim Graham <timograham@…>]

In f1efedf0:

[2.0.x] Refs #23919 -- Replaced doc references to django.utils.http.urlquote() and urlquote_plus() with standard library alternatives.

Follow up to fee42fd99ee470528858c2ccb3621135c30ec262.

Backport of 7945bc1d08f7c8fd8f25eed0327e71dca81975e4 from master

[Tim Graham <timograham@…>]

In c68f66e0:

Refs #23919 -- Replaced super() calls for old-style classes.

[Tim Graham <timograham@…>]

In ccc25bfe:

Refs #23919 -- Removed obsolete init.py files in management command directories.

[GitHub <noreply@…>]

In c76d8742:

Refs #23919 -- Removed obsolete comments about u prefixes.

[Tim Graham <timograham@…>]

In 45808895:

Refs #23919 -- Removed obsolete u-prefix stripping in inspectdb.

[Tim Graham <timograham@…>]

In 1c0bf95f:

Refs #23919 -- Removed a stray object-inheritance.

Follow up to 081e78716085ff4af08604af5d084c8fd27c0730.

[Tim Graham <timograham@…>]

In ce7293b:

Refs #23919 -- Replaced codecs.open() with open().

On Python 3, open() handles encodings.

[Mariusz Felisiak <felisiak.mariusz@…>]

In d0861fcb:

Refs #23919 -- Replaced super(ASGIHandler, self) with super().

[Carlton Gibson <carlton.gibson@…>]

In 86a0231e:

Refs #23919 -- Replaced super(...) with super() in metaclasses.

[Mariusz Felisiak <felisiak.mariusz@…>]

In b7d05b2e:

Refs #23919 -- Used yield from in inspectdb.

[Claude Paroz <claude@…>]

In 6686238c:

Refs #23919 -- Removed obsolete comment in signing.py

[Mariusz Felisiak <felisiak.mariusz@…>]

In 578b3046:

Reverted "Refs #23919 -- Removed obsolete init.py files in management command directories."

This reverts commit ccc25bfe4f0964a00df3af6f91c2d9e20159a0c2.

​https://groups.google.com/d/topic/django-developers/GVHMH2ciAnk/discussion

[GitHub <noreply@…>]

In 712fe12:

Refs #23919 -- Removed unneeded AttributeError catching in collectstatic's link_file().

os.symlink() exists always on Python 3.2+.

[GitHub <noreply@…>]

In 7c884afe:

Refs #23919 -- Updated obsolete buffer() references to memoryview().

Obsolete since 8cdc84726e13da4c796db614765658835d4786a1.