Opened 8 years ago

Closed 6 years ago

Last modified 6 years ago

#27795 closed Cleanup/optimization (fixed)

Audit force_text invocations

Reported by: Claude Paroz Owned by: Claude Paroz
Component: Utilities Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no
Pull Requests:13801 unmerged, 11710 merged, 9654 merged, 10432 unmerged, 10487 merged, 10470 merged, 10444 merged, 10464 merged, 10469 unmerged, 10358 merged, 10429 merged, 10428 merged, 10489 merged, 10431 merged, 10433 merged, 10445 merged, 10348 merged, 10346 merged, 9653 merged, 9672 merged, 9652 merged, 9686 merged, 9684 merged, 9642 merged, 9681 merged, 9683 merged, 9648 merged, 9146 unmerged, 8603 merged, 8440 merged

Description

django.utils.encoding.force_text is called many times in Django code for several use cases:

  • force the evaluation of a lazy string
  • convert bytes or a bytestring (PY2) to str
  • convert an object to its string representation

A global audit is needed to clean that usage, also considering that Python 3 native strings are more wildly used.
I'm working on it currently.

Change History (40)

comment:1 by Claude Paroz <claude@…>, 8 years ago

In ccfd1295:

Refs #27795 -- Prevented SafeText from losing safe status on str()

This will allow to replace force_text() by str() in several places (as one of
the features of force_text is to keep the safe status).

comment:2 by Claude Paroz <claude@…>, 8 years ago

In 2757209c:

Refs #27795 -- Removed force_text from templatize function

comment:3 by Claude Paroz <claude@…>, 8 years ago

In 3a148f95:

Refs #27795 -- Removed force_text from the template layer

Thanks Tim Graham for the review.

comment:4 by Claude Paroz <claude@…>, 8 years ago

In 8346680e:

Refs #27795 -- Removed unneeded force_text calls

Thanks Tim Graham for the review.

comment:5 by GitHub <noreply@…>, 8 years ago

In e5dce7b0:

Refs #27795 -- Removed unneeded force_text calls from the Oracle backend.

comment:6 by Claude Paroz <claude@…>, 8 years ago

In 301de77:

Refs #27795 -- Replaced many force_text() with str()

Thanks Tim Graham for the review.

comment:7 by Claude Paroz <claude@…>, 8 years ago

In a8343fe7:

Refs #27795 -- Replaced force_text() with str() in feed generators

comment:8 by Tim Graham <timograham@…>, 8 years ago

In b39aabc:

Refs #27795 -- Reworked get_or_create test erroneously mixing bytes and str.

As CharField.to_python() now always calls str(), assigning bytes to a
CharField is no longer correct usage. Doing so results in a warning:

django/db/models/fields/init__.py:1061: BytesWarning: str() on a bytes instance

Use a unique constraint violation to trigger the database error instead.

Warning introduced in 301de774c21d055e9e5a7073e5bffdb52bc71079.

comment:9 by Claude Paroz <claude@…>, 8 years ago

In d842ada:

Refs #27795 -- Stopped converting integer format settings to str in JS/JSON i18n views

Thanks Tim Graham for the review.

comment:10 by Claude Paroz <claude@…>, 8 years ago

In f87256b1:

Refs #27795 -- Replaced force_text() with str() in contrib.messages

Thanks Tim Graham for the review.

comment:11 by Claude Paroz <claude@…>, 8 years ago

In 45bbb9aa:

Refs #27795 -- Removed force_text() usage in serialization framework

comment:12 by Tim Graham <timograham@…>, 7 years ago

In 6b2f8fb9:

Refs #27795 -- Replaced force_text() usage in django.core.management.

Use decode() since Popen.communicate() always returns bytes.

comment:13 by Tim Graham <timograham@…>, 7 years ago

In 16c5a334:

Refs #27795 -- Replaced force_text/bytes() with decode()/encode() in password hashers.

comment:14 by Carlton Gibson <carlton.gibson@…>, 7 years ago

In 6a039800:

Refs #27795 -- Replaced force_bytes() usage in django.core.cache.

comment:15 by Tim Graham <timograham@…>, 7 years ago

In b38532c:

Refs #27795 -- Replaced force_bytes() usage in django.core.signing.

comment:16 by GitHub <noreply@…>, 7 years ago

In fa75b2c:

Refs #27795 -- Removed force_bytes/text() usage in tests.

comment:17 by Tim Graham <timograham@…>, 7 years ago

In 7d96f0c4:

Refs #27795 -- Replaced force_bytes() usage in django.http.

comment:18 by Jon Dufresne <jon.dufresne@…>, 7 years ago

In 968329b:

Refs #27795 -- Replaced force_text() calls with str() in django.utils.html.

comment:19 by Tim Graham <timograham@…>, 7 years ago

In 1f3852b:

Refs #27795 -- Removed force_bytes() usage in django/template/loaders/cached.py.

comment:20 by Tim Graham <timograham@…>, 7 years ago

In 919d5981:

Refs #27795 -- Removed force_text() usage in django/test/testcases.py.

comment:21 by Tim Graham <timograham@…>, 7 years ago

In b8a41a28:

Refs #27795 -- Removed force_bytes() in django.test.client where possible.

comment:22 by Tim Graham <timograham@…>, 7 years ago

In 90f3ef3:

Refs #27795 -- Removed force_bytes()/smart_text() from model fields.

comment:23 by Tim Graham <timograham@…>, 7 years ago

In ee184e1:

Refs #27795 -- Removed force_bytes() usage in django/utils/cache.py.

comment:24 by Tim Graham <timograham@…>, 6 years ago

In cc79c7ee:

Refs #27795 -- Removed force_text() in Model._get_FIELD_display().

comment:25 by Tim Graham <timograham@…>, 6 years ago

In 1d65ddd9:

Refs #27795 -- Removed force_bytes() usage in db/backends/utils.py.

comment:26 by Tim Graham <timograham@…>, 6 years ago

In abeed587:

Refs #27795 -- Removed force_bytes() usage in db/backends/base/schema.py.

comment:27 by Tim Graham <timograham@…>, 6 years ago

In bb81c22:

Refs #27795 -- Removed force_bytes() usage in utils/_os.py.

comment:28 by Tim Graham <timograham@…>, 6 years ago

In 8d873503:

Refs #27795 -- Removed force_bytes() usage in contrib/auth/handlers/modwsgi.py.

comment:29 by Tim Graham <timograham@…>, 6 years ago

In d4fc111c:

Refs #27795 -- Removed force_bytes() usage in admindocs.

Refs #12892 is probably obsolete.

comment:30 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In efd8a82e:

Refs #27795 -- Removed force_bytes() usage in MySQL backend.

The mysqlclient cursor attribute _last_executed is always stored as
bytes. Decode it.

TextField values are already type str. No need to decode.

comment:31 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In a47dd99:

[2.1.x] Refs #27795 -- Removed force_bytes() usage in MySQL backend.

The mysqlclient cursor attribute _last_executed is always stored as
bytes. Decode it.

TextField values are already type str. No need to decode.

Backport of efd8a82e268a82b3ad0be77bd5b4548c30bcb4d7 from master

comment:32 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In bdae19cf:

Refs #27795 -- Removed force_bytes() usage in sessions.

SessionBase.decode() is the inverse operation to SessionBase.encode().
As SessionBase.encode() always returns a string, SessionBase.decode()
should always be passed a string argument. Fixed the file backend, which
was the only backend still passing a bytestring.

comment:33 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In f3f31b0:

[2.1.x] Refs #27795 -- Removed force_bytes() usage in sessions.

SessionBase.decode() is the inverse operation to SessionBase.encode().
As SessionBase.encode() always returns a string, SessionBase.decode()
should always be passed a string argument. Fixed the file backend, which
was the only backend still passing a bytestring.

Backport of bdae19cf6395d6bfee80864d9e87c4aec241eceb from master

comment:34 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In 54eedddb:

Revert "[2.1.x] Refs #27795 -- Removed force_bytes() usage in MySQL backend."

This reverts commit a47dd99cc6883fe3e35895e8c5d4bb53996bd184.

comment:35 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In 5045f8af:

Revert "[2.1.x] Refs #27795 -- Removed force_bytes() usage in sessions."

This reverts commit f3f31b0fc26f772b198c9683a427f2f7fe208c37.

comment:36 by Tim Graham <timograham@…>, 6 years ago

In 4ab071f4:

Refs #27795 -- Removed force_bytes() usage in contrib/staticfiles/storage.py.

comment:37 by Tim Graham <timograham@…>, 6 years ago

In 2ba588e7:

Refs #27795 -- Removed force_text() usage in db/models/sql/query.py.

comment:38 by Tim Graham <timograham@…>, 6 years ago

In c82893cb:

Refs #27795 -- Removed force_bytes() usage from django/utils/http.py.

django.utils.http.urlsafe_base64_encode() now returns a string, not a
bytestring. Since URLs are represented as strings,
urlsafe_base64_encode() should return a string. All uses immediately
decoded the bytestring to a string anyway.

As the inverse operation, urlsafe_base64_decode() accepts a string.

comment:39 by Claude Paroz, 6 years ago

Resolution: fixed
Status: assignedclosed

I think this is mostly done now.

comment:40 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In 1e6b9e2:

Refs #27795 -- Removed an unnecessary force_bytes() call in uri_to_iri().

The value returned from urllib.parse.quote() is always a string, so can
safely call .encode().

Note: See TracTickets for help on using tickets.
Back to Top