Cleanups for when we drop Python 2 compatibility

[Tim Graham]

This is a tracking ticket of things that we can remove when we drop Python 2 compatibility in Django 2.0. Please edit the description of the ticket as you come across new items.

• django.core.mail.message.make_msgid() #23905
• django.dispatch.weakref_backports
• django.http.cookie workarounds (remains for Python 3.4)
• django.utils.2to3_fixes
• django.utils.decorators.ContextDecorator
• django.utils.encoding
  • @python_2_unicode_compatible
  • force/smart _unicode
  • either force/smart _text or force/smart _str ​PR
• django.utils.html_parser.use_workaround
• django.utils.http functions like urlquote_plus -- I think the versions of these functions on Python 3 don't have the unicode characters bug we are working around ​PR
• Stop using django.utils.lru_cache
• Figure the deprecation plan for django.utils.lru_cache (staying for now)
• django.utils.six
• from __future__ import unicode_literals
• str() stuff for environment variables, e.g. 0bfb53866199f366ed140d49938fd185e5898156
• str() stuff for type(name) and __name__
• Inheriting from object in django.core.servers.basehttp (and perhaps other places) ala 4ee06ec3fc8e94d164afbd2f9c880c60c658a9ac
• git grep 'long int' (mostly docs)
• django.utils._os [npath,upath]
• In tests: contextlib.closing(self.urlopen (contextlib.closing no longer needed)
• Support for pysqlite (doesn't support Python 3)
• Replace super(ClassName, self) with super() ​PR
• Remove # -*- coding: utf-8 -*- source file encoding (that's the default on Python 3)
• Evaluate replacement of custom __del__ methods by ​weakref.finalize
• Remove __ne__ from objects already defining a __eq__
• Remove note about PYTHONHASHSEED (see #26243)
• Remove Field.creation_counter (replace by metaclass __prepare__ returning OrderedDict(), ​https://docs.python.org/3/reference/datamodel.html#preparing-the-class-namespace) (see comment:95)
• django.test.utils.reset_warning_registry()
• Replace errno checks by IOError subclasses defined by ​PEP 3151 ​PR
• django.utils.glob
• django.utils.cookies.SimpleCookie (remains for Python 3.4 compatibility)
• django.test.mock
• Replace tempfile.mkdtemp + remove with tempfile.TemporaryDirectory context manager
• Replace io.open() by a plain open()
• Evaluate need for assertRegex, assertRaisesRegex in tests. Some usage is merely to account for differences in messages between Python 2 and 3.
• re.U, re.UNICODE (default behavior of Python 3) - ​PR 7879

[Wojtek Ruszczewski]

Various nonlocal workarounds (​example).

[Anssi Kääriäinen]

I believe we can get rid of the Field.creation_counter hack. In Python 3 it is possible to store the attrs in a sorted dictionary. See ​http://stackoverflow.com/questions/4459531/how-to-read-class-attributes-in-the-same-order-as-declared.

[Curtis Maloney]

In far too many tests we rely on the fact six aliases builtins [like input] instead of using mock.

I submitted a patch recently to clean up a couple of these I found, but I am finding more.

[Claude Paroz]

Note that I have a local branch which does some boring removals (unicode_literals, utf-8 preambles, some six usage). So ping me as soon as the Django 2.0 deprecated stuff is removed and I'll offer a patch for that.

[Asif Saifuddin Auvi]

I am also working to remove python2 workarounds by following a module by module approach.

[Aymeric Augustin]

If Claude has already prepared these changes, perhaps you should wait for his patch instead of duplicating his work?

[Asif Saifuddin Auvi]

Replying to Aymeric Augustin:

If Claude has already prepared these changes, perhaps you should wait for his patch instead of duplicating his work?

Yes I talked with Claude on IRC to pause before his work is merged.

[Josh Smeaton]

Replying to Anssi Kääriäinen:

I believe we can get rid of the Field.creation_counter hack. In Python 3 it is possible to store the attrs in a sorted dictionary. See ​http://stackoverflow.com/questions/4459531/how-to-read-class-attributes-in-the-same-order-as-declared.

We could also remove this when we support python 3.6, as class attributes retain their order. I'm guessing this is a consequence (or the cause of) dictionaries retaining insertion order. ​https://docs.python.org/3.6/whatsnew/3.6.html#pep-520-preserving-class-attribute-definition-order

[Claude Paroz <claude@…>]

In d7b9aaa:

Refs #23919 -- Removed encoding preambles and future imports

[Claude Paroz <claude@…>]

In f3c43ad:

Refs #23919 -- Removed python_2_unicode_compatible decorator usage

[Tim Graham]

​PR for removing Python 2 references from docs.

[Claude Paroz <claude@…>]

In c716fe87:

Refs #23919 -- Removed six.PY2/PY3 usage

Thanks Tim Graham for the review.

[GitHub <noreply@…>]

In f6acd1d:

Refs #23919 -- Removed Python 2 notes in docs.

[Claude Paroz <claude@…>]

In 7b2f2e74:

Refs #23919 -- Removed six.<various>_types usage

Thanks Tim Graham and Simon Charette for the reviews.

[Claude Paroz <claude@…>]

In 2b281cc3:

Refs #23919 -- Removed most of remaining six usage

Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 3cc5f01d:

Refs #23919 -- Stopped using django.utils.lru_cache().

[Tim Graham <timograham@…>]

In eb422e47:

Refs #23919 -- Removed obsolete ne() methods.

ne() defaults to the opposite of eq() on Python 3
when it doesn't return NotImplemented.

[Tim Graham <timograham@…>]

In a5563963:

Refs #23919 -- Replaced io.open() with open().

io.open() is an alias for open() on Python 3.

[Claude Paroz <claude@…>]

In cecc079:

Refs #23919 -- Stopped inheriting from object to define new style classes.

[GitHub <noreply@…>]

In 5320fa7:

Refs #23919 -- Removed obsolete contextlib.closing() calls (for Python 2).

[GitHub <noreply@…>]

In e5c67f0:

Refs #23919 -- Removed reset_warning_registry() workaround for Python < 3.4.2.

[Simon Charette <charette.s@…>]

In 4c5ed3e6:

Refs #23919 -- Removed nonzero() methods (for Python 2).

Thanks Tim for the review.

[Tim Graham <timograham@…>]

In 41e0033c:

Refs #23919 -- Removed usage of django.utils.decorators.ContextDecorator.

[Simon Charette <charette.s@…>]

In 9695b14:

Refs #23919 -- Removed str() conversion of type and method name.

[GitHub <noreply@…>]

In d4bb3759:

Refs #23919 -- Removed obsolete compare_digest() and pbkdf2() implementations.

[GitHub <noreply@…>]

In 9e917cc2:

Fixed #23905, refs #23919 -- Used make_msgid() from stdlib.

[GitHub <noreply@…>]

In 9d27478:

Refs #23919 -- Removed docs references to long integers.

[Tim Graham <timograham@…>]

In 9ee47ce7:

Refs #23919 -- Removed enum ImportError handling for Python 2.

[Tim Graham <timograham@…>]

In eb0b921c:

Refs #23919 -- Removed SessionBase.iterkeys(), itervalues(), iteritems().

These methods only work on Python 2.

[Tim Graham <timograham@…>]

In fedda6d:

Refs #23919 -- Removed Python 2 version check in django.http.cookie.

[Tim Graham <timograham@…>]

In bf1c9570:

Refs #23919 -- Removed Python 2 workaround for hashing Oracle params (refs #27632).

[Claude Paroz <claude@…>]

In dc8834ca:

Refs #23919 -- Removed unneeded force_str calls

[Aymeric Augustin <aymeric.augustin@…>]

In 109b33f:

Refs #23919 -- Simplified assertRaisesRegex()'s that accounted for Python 2.

[GitHub <noreply@…>]

In 4e729fea:

Refs #23919 -- Removed django.utils._os.upath()/npath()/abspathu() usage.

These functions do nothing on Python 3.

[Claude Paroz <claude@…>]

In 042b735:

Refs #23919 -- Removed unneeded str() calls

[Tim Graham <timograham@…>]

In 7aba691:

Refs #23919 -- Removed django.test.mock Python 2 compatibility shim.

[Claude Paroz <claude@…>]

In 289fc1bf:

Refs #23919 -- Removed str_prefix usage

[Tim Graham <timograham@…>]

In 1b06d5e6:

Refs #23919 -- Removed pysqlite support (it's Python 2 only).

[GitHub <noreply@…>]

In 9e6e32bf:

Refs #23919 -- Removed django.utils.decorators.available_attrs() usage.

It's only needed to workaround a bug on Python 2.

[Tim Graham <timograham@…>]

In c222122:

Refs #23919 -- Removed re.U and re.UNICODE (default on Python 3).

[GitHub <noreply@…>]

In d170c63:

Refs #23919 -- Removed misc references to Python 2.

[Claude Paroz <claude@…>]

In 6e55e1d:

Refs #23919 -- Replaced six.reraise by raise

[GitHub <noreply@…>]

In 435e4bf3:

Refs #23919 -- Removed traceback setting needed for Python 2.

Partially reverted refs #25761 and refs #16245.

[Tim Graham <timograham@…>]

In 5b95d42:

Refs #23919 -- Removed a MySQLdb workaround (refs #6052) for Python 2.

[GitHub <noreply@…>]

In 632c4ffd:

Refs #23919 -- Replaced errno checking with PEP 3151 exceptions.

[GitHub <noreply@…>]

In 2d96c027:

Refs #23919 -- Removed obsolete MySQLdb references.

[Tim Graham <timograham@…>]

In dc165ec8:

Refs #23919 -- Replaced super(ClassName, self) with super() in docs.

[Tim Graham <timograham@…>]

In d6eaf7c:

Refs #23919 -- Replaced super(ClassName, self) with super().

[GitHub <noreply@…>]

In 1c466994:

Refs #23919 -- Removed misc Python 2/3 references.

[Tim Graham <timograham@…>]

In d1bab24:

Refs #23919, #27778 -- Removed obsolete mentions of unicode.

[Claude Paroz <claude@…>]

In fee42fd9:

Refs #23919 -- Replaced usage of django.utils.http utilities with Python equivalents

Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 6478e07:

Refs #23919 -- Replaced tempfile.mkdtemp() with TemporaryDirectory() context manager.

[GitHub <noreply@…>]

In e07e743e:

Refs #23919 -- Used DeclarativeFieldsMetaclass.prepare() for tracking form field order.

[GitHub <noreply@…>]

In 9e9e7373:

Refs #23919 -- Removed an obsolete test for a Python 2 code path (refs #15662).

Fixed #21628 by removing the last usage of the imp module.

[Josh Smeaton]

I had an attempt at dropping the creation_counter for model fields, but I *think* it might be more costly than the current method, considering so many things rely on the order being consistent. My attempt was here, with associated discussion I'll copy some of below: ​https://github.com/django/django/pull/7983

I'm not sure this is worth continuing with, at least not in the same way. Too many systems rely on the fields being ordered, which isn't going to be possible without maintaining another list of fields such as definition_order = []. This can be done for local fields, but then proxy/parent fields will need to be returned in the same definition order. This might be something like:

parent_fields + [field for field in self.definition_order if field in local_fields]

Then we'd need to account for private and hidden fields. But what's the point? We'd now be storing a separate list of all field references for each model, rather than a counter (int) on each field, and reordering fields in _get_fields() whether or not they need to be ordered. Of course I'd be happy to see a better solution that worked, I'm just giving up on this particular method for the moment.

[Tim Graham]

I also looked briefly at models field and manager usage of creation_counter but it wasn't obvious if the __prepare__() approach was feasible or if it would be simpler. I think we can defer that task for now.

[Claude Paroz <claude@…>]

In c688336e:

Refs #23919 -- Assumed request COOKIES and META are str

[Claude Paroz <claude@…>]

In 52138b1f:

Refs #23919 -- Removed usage of obsolete SafeBytes class

The class will be removed as part of #27753.
Thanks Tim Graham for the review.

[Tim Graham <timograham@…>]

In 84126f27:

Refs #23919 -- Removed unneeded code in force_text().

Unneeded since 7b2f2e74adb36a4334e83130f6abc2f79d395235.

[Tim Graham <timograham@…>]

In 8838d4d:

Refs #23919 -- Replaced kwargs.pop() with keyword-only arguments.

[Collin Anderson]

Can we remove empty init.py (and init.py-tpl) files?

[Aymeric Augustin]

IIRC there's a risk to making everything a namespace package: it means that if you have conflicting package / module names, they're somehow merged instead of having the first one shadow the other. (Perhaps someone more familiar with the topic can describe this more precisely.) I think it's safer to keep the __init__ files.

[Nick Pope]

Given that Django 2.0 is targeting Python 3.5+, are there other things that could be cleaned up that were added to support <3.5 or could be used now that 3.5 is the minimum version?

For example, there are many places with the following pattern:

try:
    ...
except ...:
    pass

These could be replaced with ​contextlib.suppress() which is available since 3.4.

Other potentially interesting things are:

• Use of ​unpacking generalisations
• Use of ​type hints and a type checker when building.
• Use of ​os.scandir() instead of os.listdir().
• Use of the ​enum and ​pathlib modules where appropriate.
• ​contextlib.redirect_stdout() and ​contextlib.redirect_stderr()

There may be more - this was just a quick look at the What's New pages for 3.4 and 3.5.

[Tim Graham]

Usage of Python 3 features could be proposed in new tickets (e.g. #27804). Master is still supporting Python 3.4 right now as per ​django-developers discussion.

[GitHub <noreply@…>]

In 2d899ce:

Refs #23919 -- Removed a Python 2 code path in force_text().

Reverted the obsolete fix and tests for refs #12302.

[GitHub <noreply@…>]

In 21f13ff5:

Refs #23919 -- Removed an used block in ExceptionReporter.get_traceback_data().

The test from refs #20368 only runs this block on Python 2.

[Tim Graham <timograham@…>]

In 500532c9:

Refs #23919 -- Removed default 'utf-8' argument for str.encode()/decode().

[Collin Anderson]

Should we deprecate cache.has_key() ?.... Nevermind, it takes the "version" argument....