Opened 3 years ago

Last modified 2 weeks ago

#23919 new Cleanup/optimization

Cleanups for when we drop Python 2 compatibility

Reported by: Tim Graham Owned by: nobody
Component: Core (Other) Version:
Severity: Normal Keywords:
Cc: cmawebsite@…, Tom Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Collin Anderson)

This is a tracking ticket of things that we can remove when we drop Python 2 compatibility in Django 2.0. Please edit the description of the ticket as you come across new items.

  • django.core.mail.message.make_msgid() #23905
  • django.dispatch.weakref_backports
  • django.http.cookie workarounds (remains for Python 3.4)
  • django.utils.2to3_fixes
  • django.utils.decorators.ContextDecorator
  • django.utils.encoding
    • @python_2_unicode_compatible
    • force/smart _unicode
    • either force/smart _text or force/smart _str PR
  • django.utils.html_parser.use_workaround
  • django.utils.http functions like urlquote_plus -- I think the versions of these functions on Python 3 don't have the unicode characters bug we are working around PR
  • Stop using django.utils.lru_cache
  • Figure the deprecation plan for django.utils.lru_cache (staying for now)
  • django.utils.six
  • from __future__ import unicode_literals
  • str() stuff for environment variables, e.g. 0bfb53866199f366ed140d49938fd185e5898156
  • str() stuff for type(name) and __name__
  • Inheriting from object in django.core.servers.basehttp (and perhaps other places) ala 4ee06ec3fc8e94d164afbd2f9c880c60c658a9ac
  • git grep 'long int' (mostly docs)
  • django.utils._os [npath,upath]
  • In tests: contextlib.closing(self.urlopen (contextlib.closing no longer needed)
  • Support for pysqlite (doesn't support Python 3)
  • Replace super(ClassName, self) with super() PR
  • Remove # -*- coding: utf-8 -*- source file encoding (that's the default on Python 3)
  • Evaluate replacement of custom __del__ methods by weakref.finalize
  • Remove __ne__ from objects already defining a __eq__
  • Remove note about PYTHONHASHSEED (see #26243)
  • Remove Field.creation_counter (replace by metaclass __prepare__ returning OrderedDict(), https://docs.python.org/3/reference/datamodel.html#preparing-the-class-namespace) (see comment:95)
  • django.test.utils.reset_warning_registry()
  • Replace errno checks by IOError subclasses defined by PEP 3151 PR
  • django.utils.glob
  • django.utils.cookies.SimpleCookie (remains for Python 3.4 compatibility)
  • django.test.mock
  • Replace tempfile.mkdtemp + remove with tempfile.TemporaryDirectory context manager
  • Replace io.open() by a plain open()
  • Evaluate need for assertRegex, assertRaisesRegex in tests. Some usage is merely to account for differences in messages between Python 2 and 3.
  • re.U, re.UNICODE (default behavior of Python 3) - PR 7879

Change History (127)

comment:1 Changed 3 years ago by Wojtek Ruszczewski

Various nonlocal workarounds (example).

comment:2 Changed 3 years ago by Collin Anderson

Description: modified (diff)

comment:3 Changed 3 years ago by Tim Graham

Description: modified (diff)

comment:4 Changed 3 years ago by Anssi Kääriäinen

I believe we can get rid of the Field.creation_counter hack. In Python 3 it is possible to store the attrs in a sorted dictionary. See http://stackoverflow.com/questions/4459531/how-to-read-class-attributes-in-the-same-order-as-declared.

comment:5 Changed 3 years ago by Collin Anderson

Description: modified (diff)

comment:6 Changed 3 years ago by Tim Graham

Description: modified (diff)

comment:7 Changed 3 years ago by Tim Graham

Description: modified (diff)

comment:8 Changed 3 years ago by Collin Anderson

Cc: cmawebsite@… added
Description: modified (diff)

comment:9 Changed 3 years ago by Collin Anderson

Description: modified (diff)

comment:10 Changed 2 years ago by Curtis Maloney

In far too many tests we rely on the fact six aliases builtins [like input] instead of using mock.

I submitted a patch recently to clean up a couple of these I found, but I am finding more.

comment:11 Changed 2 years ago by Curtis Maloney

Description: modified (diff)

comment:12 Changed 2 years ago by Tim Graham

Description: modified (diff)

comment:13 Changed 2 years ago by Tim Graham

Description: modified (diff)

comment:14 Changed 2 years ago by Moritz Sichert

Description: modified (diff)

comment:15 Changed 2 years ago by Tim Graham

Description: modified (diff)

comment:16 Changed 23 months ago by Claude Paroz

Description: modified (diff)

comment:17 Changed 22 months ago by Simon Charette

Description: modified (diff)

comment:18 Changed 19 months ago by Collin Anderson

Description: modified (diff)

comment:19 Changed 19 months ago by Anssi Kääriäinen

Description: modified (diff)

comment:20 Changed 17 months ago by Tim Graham

Description: modified (diff)

comment:21 Changed 16 months ago by Jon Dufresne

Description: modified (diff)

comment:22 Changed 13 months ago by Curtis Maloney

Description: modified (diff)

comment:23 Changed 13 months ago by Curtis Maloney

Description: modified (diff)

comment:24 Changed 13 months ago by Jon Dufresne

Description: modified (diff)

comment:25 Changed 12 months ago by Claude Paroz

Description: modified (diff)

comment:26 Changed 10 months ago by Tim Graham

Description: modified (diff)

comment:27 Changed 9 months ago by Tim Graham

Description: modified (diff)

comment:28 Changed 9 months ago by Claude Paroz

Note that I have a local branch which does some boring removals (unicode_literals, utf-8 preambles, some six usage). So ping me as soon as the Django 2.0 deprecated stuff is removed and I'll offer a patch for that.

comment:29 Changed 8 months ago by Asif Saifuddin Auvi

I am also working to remove python2 workarounds by following a module by module approach.

comment:30 Changed 8 months ago by Aymeric Augustin

If Claude has already prepared these changes, perhaps you should wait for his patch instead of duplicating his work?

comment:31 in reply to:  30 Changed 8 months ago by Asif Saifuddin Auvi

Replying to Aymeric Augustin:

If Claude has already prepared these changes, perhaps you should wait for his patch instead of duplicating his work?

Yes I talked with Claude on IRC to pause before his work is merged.

comment:32 in reply to:  4 Changed 8 months ago by Josh Smeaton

Replying to Anssi Kääriäinen:

I believe we can get rid of the Field.creation_counter hack. In Python 3 it is possible to store the attrs in a sorted dictionary. See http://stackoverflow.com/questions/4459531/how-to-read-class-attributes-in-the-same-order-as-declared.

We could also remove this when we support python 3.6, as class attributes retain their order. I'm guessing this is a consequence (or the cause of) dictionaries retaining insertion order. https://docs.python.org/3.6/whatsnew/3.6.html#pep-520-preserving-class-attribute-definition-order

comment:33 Changed 8 months ago by Claude Paroz <claude@…>

In d7b9aaa:

Refs #23919 -- Removed encoding preambles and future imports

comment:34 Changed 8 months ago by Claude Paroz <claude@…>

In f3c43ad:

Refs #23919 -- Removed python_2_unicode_compatible decorator usage

comment:35 Changed 8 months ago by Tim Graham

Triage Stage: Someday/MaybeAccepted

PR for removing Python 2 references from docs.

comment:36 Changed 8 months ago by Claude Paroz <claude@…>

In c716fe87:

Refs #23919 -- Removed six.PY2/PY3 usage

Thanks Tim Graham for the review.

comment:37 Changed 8 months ago by GitHub <noreply@…>

In f6acd1d:

Refs #23919 -- Removed Python 2 notes in docs.

comment:38 Changed 8 months ago by Claude Paroz <claude@…>

In 7b2f2e74:

Refs #23919 -- Removed six.<various>_types usage

Thanks Tim Graham and Simon Charette for the reviews.

comment:39 Changed 8 months ago by Claude Paroz <claude@…>

In 2b281cc3:

Refs #23919 -- Removed most of remaining six usage

Thanks Tim Graham for the review.

comment:40 Changed 8 months ago by Aymeric Augustin

Description: modified (diff)

comment:41 Changed 8 months ago by Claude Paroz

Description: modified (diff)

comment:42 Changed 8 months ago by felixxm

Description: modified (diff)

comment:43 Changed 8 months ago by Tim Graham <timograham@…>

In 3cc5f01d:

Refs #23919 -- Stopped using django.utils.lru_cache().

comment:44 Changed 8 months ago by Tim Graham <timograham@…>

In eb422e47:

Refs #23919 -- Removed obsolete ne() methods.

ne() defaults to the opposite of eq() on Python 3
when it doesn't return NotImplemented.

comment:45 Changed 8 months ago by Tim Graham <timograham@…>

In a5563963:

Refs #23919 -- Replaced io.open() with open().

io.open() is an alias for open() on Python 3.

comment:46 Changed 8 months ago by Simon Charette

Description: modified (diff)

comment:47 Changed 8 months ago by Simon Charette

Description: modified (diff)

comment:48 Changed 8 months ago by Claude Paroz <claude@…>

In cecc079:

Refs #23919 -- Stopped inheriting from object to define new style classes.

comment:49 Changed 8 months ago by ChillarAnand

Description: modified (diff)

comment:50 Changed 8 months ago by GitHub <noreply@…>

In 5320fa7:

Refs #23919 -- Removed obsolete contextlib.closing() calls (for Python 2).

comment:51 Changed 8 months ago by GitHub <noreply@…>

In e5c67f0:

Refs #23919 -- Removed reset_warning_registry() workaround for Python < 3.4.2.

comment:52 Changed 8 months ago by Simon Charette <charette.s@…>

In 4c5ed3e6:

Refs #23919 -- Removed nonzero() methods (for Python 2).

Thanks Tim for the review.

comment:53 Changed 8 months ago by Tim Graham <timograham@…>

In 41e0033c:

Refs #23919 -- Removed usage of django.utils.decorators.ContextDecorator.

comment:54 Changed 8 months ago by Simon Charette <charette.s@…>

In 9695b14:

Refs #23919 -- Removed str() conversion of type and method name.

comment:55 Changed 8 months ago by Simon Charette

Description: modified (diff)

comment:56 Changed 8 months ago by GitHub <noreply@…>

In d4bb3759:

Refs #23919 -- Removed obsolete compare_digest() and pbkdf2() implementations.

comment:57 Changed 8 months ago by GitHub <noreply@…>

In 9e917cc2:

Fixed #23905, refs #23919 -- Used make_msgid() from stdlib.

comment:58 Changed 8 months ago by GitHub <noreply@…>

In 9d27478:

Refs #23919 -- Removed docs references to long integers.

comment:59 Changed 8 months ago by Tim Graham <timograham@…>

In 9ee47ce7:

Refs #23919 -- Removed enum ImportError handling for Python 2.

comment:60 Changed 8 months ago by Tim Graham <timograham@…>

In eb0b921c:

Refs #23919 -- Removed SessionBase.iterkeys(), itervalues(), iteritems().

These methods only work on Python 2.

comment:61 Changed 8 months ago by Tim Graham <timograham@…>

In fedda6d:

Refs #23919 -- Removed Python 2 version check in django.http.cookie.

comment:62 Changed 8 months ago by Tim Graham <timograham@…>

In bf1c9570:

Refs #23919 -- Removed Python 2 workaround for hashing Oracle params (refs #27632).

comment:63 Changed 8 months ago by Tim Graham

Description: modified (diff)

comment:64 Changed 8 months ago by ChillarAnand

Description: modified (diff)

comment:65 Changed 8 months ago by Claude Paroz <claude@…>

In dc8834ca:

Refs #23919 -- Removed unneeded force_str calls

comment:66 Changed 8 months ago by Aymeric Augustin <aymeric.augustin@…>

In 109b33f:

Refs #23919 -- Simplified assertRaisesRegex()'s that accounted for Python 2.

comment:67 Changed 8 months ago by GitHub <noreply@…>

In 4e729fea:

Refs #23919 -- Removed django.utils._os.upath()/npath()/abspathu() usage.

These functions do nothing on Python 3.

comment:68 Changed 8 months ago by Claude Paroz <claude@…>

In 042b735:

Refs #23919 -- Removed unneeded str() calls

comment:69 Changed 8 months ago by Claude Paroz

Description: modified (diff)

comment:70 Changed 8 months ago by Tim Graham <timograham@…>

In 7aba691:

Refs #23919 -- Removed django.test.mock Python 2 compatibility shim.

comment:71 Changed 8 months ago by Claude Paroz <claude@…>

In 289fc1bf:

Refs #23919 -- Removed str_prefix usage

comment:72 Changed 8 months ago by Tim Graham <timograham@…>

In 1b06d5e6:

Refs #23919 -- Removed pysqlite support (it's Python 2 only).

comment:73 Changed 8 months ago by GitHub <noreply@…>

In 9e6e32bf:

Refs #23919 -- Removed django.utils.decorators.available_attrs() usage.

It's only needed to workaround a bug on Python 2.

comment:74 Changed 8 months ago by Tim Graham <timograham@…>

In c222122:

Refs #23919 -- Removed re.U and re.UNICODE (default on Python 3).

comment:75 Changed 8 months ago by felixxm

Description: modified (diff)

comment:76 Changed 8 months ago by GitHub <noreply@…>

In d170c63:

Refs #23919 -- Removed misc references to Python 2.

comment:77 Changed 8 months ago by ChillarAnand

Description: modified (diff)

comment:78 Changed 8 months ago by Claude Paroz <claude@…>

In 6e55e1d:

Refs #23919 -- Replaced six.reraise by raise

comment:79 Changed 8 months ago by Claude Paroz

Description: modified (diff)

comment:80 Changed 8 months ago by GitHub <noreply@…>

In 435e4bf3:

Refs #23919 -- Removed traceback setting needed for Python 2.

Partially reverted refs #25761 and refs #16245.

comment:81 Changed 8 months ago by Tim Graham <timograham@…>

In 5b95d42:

Refs #23919 -- Removed a MySQLdb workaround (refs #6052) for Python 2.

comment:82 Changed 8 months ago by GitHub <noreply@…>

In 632c4ffd:

Refs #23919 -- Replaced errno checking with PEP 3151 exceptions.

comment:83 Changed 8 months ago by GitHub <noreply@…>

In 2d96c027:

Refs #23919 -- Removed obsolete MySQLdb references.

comment:84 Changed 8 months ago by Tim Graham <timograham@…>

In dc165ec8:

Refs #23919 -- Replaced super(ClassName, self) with super() in docs.

comment:85 Changed 8 months ago by Tim Graham <timograham@…>

In d6eaf7c:

Refs #23919 -- Replaced super(ClassName, self) with super().

comment:86 Changed 8 months ago by GitHub <noreply@…>

In 1c466994:

Refs #23919 -- Removed misc Python 2/3 references.

comment:87 Changed 8 months ago by Aymeric Augustin

Description: modified (diff)

comment:88 Changed 8 months ago by Tim Graham <timograham@…>

In d1bab24:

Refs #23919, #27778 -- Removed obsolete mentions of unicode.

comment:89 Changed 8 months ago by Claude Paroz <claude@…>

In fee42fd9:

Refs #23919 -- Replaced usage of django.utils.http utilities with Python equivalents

Thanks Tim Graham for the review.

comment:90 Changed 8 months ago by Tim Graham <timograham@…>

In 6478e07:

Refs #23919 -- Replaced tempfile.mkdtemp() with TemporaryDirectory() context manager.

comment:91 Changed 8 months ago by Claude Paroz

Description: modified (diff)

comment:92 Changed 8 months ago by ChillarAnand

Description: modified (diff)

comment:93 Changed 8 months ago by GitHub <noreply@…>

In e07e743e:

Refs #23919 -- Used DeclarativeFieldsMetaclass.prepare() for tracking form field order.

comment:94 Changed 8 months ago by GitHub <noreply@…>

In 9e9e7373:

Refs #23919 -- Removed an obsolete test for a Python 2 code path (refs #15662).

Fixed #21628 by removing the last usage of the imp module.

comment:95 Changed 8 months ago by Josh Smeaton

I had an attempt at dropping the creation_counter for model fields, but I *think* it might be more costly than the current method, considering so many things rely on the order being consistent. My attempt was here, with associated discussion I'll copy some of below: https://github.com/django/django/pull/7983

I'm not sure this is worth continuing with, at least not in the same way. Too many systems rely on the fields being ordered, which isn't going to be possible without maintaining another list of fields such as definition_order = []. This can be done for local fields, but then proxy/parent fields will need to be returned in the same definition order. This might be something like:

parent_fields + [field for field in self.definition_order if field in local_fields]

Then we'd need to account for private and hidden fields. But what's the point? We'd now be storing a separate list of all field references for each model, rather than a counter (int) on each field, and reordering fields in _get_fields() whether or not they need to be ordered. Of course I'd be happy to see a better solution that worked, I'm just giving up on this particular method for the moment.

comment:96 Changed 8 months ago by Tim Graham

Description: modified (diff)

I also looked briefly at models field and manager usage of creation_counter but it wasn't obvious if the __prepare__() approach was feasible or if it would be simpler. I think we can defer that task for now.

comment:97 Changed 8 months ago by Claude Paroz <claude@…>

In c688336e:

Refs #23919 -- Assumed request COOKIES and META are str

comment:98 Changed 8 months ago by Claude Paroz <claude@…>

In 52138b1f:

Refs #23919 -- Removed usage of obsolete SafeBytes class

The class will be removed as part of #27753.
Thanks Tim Graham for the review.

comment:99 Changed 8 months ago by Tim Graham <timograham@…>

In 84126f27:

Refs #23919 -- Removed unneeded code in force_text().

Unneeded since 7b2f2e74adb36a4334e83130f6abc2f79d395235.

comment:100 Changed 8 months ago by Tim Graham <timograham@…>

In 8838d4d:

Refs #23919 -- Replaced kwargs.pop() with keyword-only arguments.

comment:101 Changed 8 months ago by Collin Anderson

Description: modified (diff)

Can we remove empty init.py (and init.py-tpl) files?

comment:102 Changed 8 months ago by Aymeric Augustin

IIRC there's a risk to making everything a namespace package: it means that if you have conflicting package / module names, they're somehow merged instead of having the first one shadow the other. (Perhaps someone more familiar with the topic can describe this more precisely.) I think it's safer to keep the init files.

comment:103 Changed 8 months ago by Nick Pope

Given that Django 2.0 is targeting Python 3.5+, are there other things that could be cleaned up that were added to support <3.5 or could be used now that 3.5 is the minimum version?

For example, there are many places with the following pattern:

try:
    ...
except ...:
    pass

These could be replaced with contextlib.suppress() which is available since 3.4.

Other potentially interesting things are:

There may be more - this was just a quick look at the What's New pages for 3.4 and 3.5.

comment:104 Changed 8 months ago by Tim Graham

Usage of Python 3 features could be proposed in new tickets (e.g. #27804). Master is still supporting Python 3.4 right now as per django-developers discussion.

comment:105 Changed 8 months ago by GitHub <noreply@…>

In 2d899ce:

Refs #23919 -- Removed a Python 2 code path in force_text().

Reverted the obsolete fix and tests for refs #12302.

comment:106 Changed 8 months ago by GitHub <noreply@…>

In 21f13ff5:

Refs #23919 -- Removed an used block in ExceptionReporter.get_traceback_data().

The test from refs #20368 only runs this block on Python 2.

comment:107 Changed 8 months ago by Tim Graham <timograham@…>

In 500532c9:

Refs #23919 -- Removed default 'utf-8' argument for str.encode()/decode().

comment:108 Changed 7 months ago by Collin Anderson

Description: modified (diff)

Should we deprecate cache.has_key() ?.... Nevermind, it takes the "version" argument....

Last edited 7 months ago by Collin Anderson (previous) (diff)

comment:109 Changed 7 months ago by Collin Anderson

Description: modified (diff)

comment:110 Changed 7 months ago by Collin Anderson

The "Python 3 is recommended" text here could probably be rephrased:

https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-should-i-use-with-django

comment:111 Changed 7 months ago by Tim Graham <timograham@…>

In 4cffa9a1:

Fixed #27878, refs #23919 -- Used python3 shebangs.

comment:112 Changed 7 months ago by Tim Graham <timograham@…>

In 3dcc351:

Refs #23919 -- Used yield from.

comment:113 Changed 7 months ago by Tim Graham <timograham@…>

In 784a53b:

Reverted "Fixed #27878, refs #23919 -- Used python3 shebangs."

This reverts commit 4cffa9a1ffb37d4de7e99a851a9ed87b3c02d229.

comment:114 Changed 6 months ago by Tim Graham <timograham@…>

In ed0cbc8:

Refs #23919 -- Removed some Python 2 compatibility code and comments.

comment:115 Changed 5 months ago by GitHub <noreply@…>

In 84dcd162:

Refs #23919 -- Used "raise from" instead of cause in reraising backend-specific database exceptions.

Thanks Tim Graham for the review.

comment:116 Changed 5 months ago by Tim Graham <timograham@…>

In 9b538ba:

Refs #23919 -- Removed File's Python 2 proxied methods.

comment:117 Changed 5 months ago by Claude Paroz <claude@…>

In 59f8118c:

Refs #23919 -- Removed force_text() in Python deserializer needed only on Python 2

comment:118 Changed 3 months ago by Tim Graham <timograham@…>

In 3eb3907b:

Refs #23919 -- Replaced stray super(ClassName, self) with super().

comment:119 Changed 3 months ago by GitHub <noreply@…>

In 3d0a0ecd:

Refs #23919 -- Removed support for broken Model.str() in Model.repr().

Returning invalid bytestrings in str() is unlikely in Python 3.

comment:120 Changed 3 months ago by Tim Graham <timograham@…>

In 081e787:

Refs #23919 -- Stopped inheriting from object to define new style classes.

Tests and docs complement to cecc079168e8669138728d31611ff3a1e7eb3a9f.

comment:121 Changed 2 months ago by Tim Graham <timograham@…>

In 4af88cc:

Refs #23919 -- Removed Python 2 reference in django_bash_completion.

comment:122 Changed 2 months ago by Sergey Fedoseev <fedoseev.sergey@…>

In 59a4b12a:

Refs #23919 -- Removed LazyObject.getstate() needed only on Python 2.

comment:123 Changed 2 months ago by Tom

I was looking at potential cleanups I could help with and noticed a lot of os.path functions that could be replaced with pathlib - we have 219 occurrences of os.path (excluding usages in tests).

Is this something worth spending time on? It's pretty easy to convert, and it makes path related code a lot more readable IMO (no need for nested, multiple os.path.* calls in a single line for example). In 3.5 and above there is also a read_text method on paths which is quite useful in some cases.

comment:124 Changed 2 months ago by Tim Graham

Sure, start small and we can see if the changes look like an improvement.

comment:125 Changed 2 months ago by Tim Graham <timograham@…>

In 462f158:

Refs #23919 -- Updated contrib.admin's compress.py to use pathlib.

comment:126 Changed 8 weeks ago by Tom

Cc: Tom added

comment:127 Changed 2 weeks ago by Tim Graham <timograham@…>

In 5b1c3896:

Refs #23919 -- Replaced usage of django.utils.functional.curry() with functools.partial()/partialmethod().

Note: See TracTickets for help on using tickets.
Back to Top