After migrating a Django project from Python 2 to 3, db migration of django.contrib.sites fails with "TypeError: attribute name must be string, not 'bytes'"

[Florian Mayer]

I'm migrating an existing Django project from Python 2.7 / Django 1.11 to Python 3.6 / Django 2.0.7 (latest stable at the time of writing).
Migrations are finicky as they (used to) be created without from __future__ import unicode_literals, therefore all strings (such as model manager names e.g. "'objects'") would be interpreted as bytestrings in Python 3.x.

Ticket 24701 is the closest issue I could find relating to this error, and it led to Django now inserting the unicode_literals import into newly created migrations. This however doesn't fix existing migrations, and while Django's migrations are doing now a decent job to write plain un-prefixed "strings" in field names, sometimes a model manager name or a more exotic migration command's argument slips through as b"explicitly prefixed bytestring".

Environment

• ​CircleCI v2 build using Python 3.6.1 (freshly upgraded from Python 2.7)
• ​requirements Django 2.0.7
• The Django project was written Django 1.9.7 to 1.11.x, and recently upgraded to Django 2.x.
• The migrations were created using Django 1.9.7 to 1.11.x
• Third party app migrations are potentially ancient

Stack trace

​Failing CircleCI build fails at ./manage.py migrate.
Can reproduce on Ubuntu 16.04 (Python 3.6.x Anaconda) and 14.04 (Python 3.4.3).

Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  [... working migrations ...]
  Applying sites.0001_initial...Traceback (most recent call last):
  File "manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 371, in execute_from_command_line
    utility.execute()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 365, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/commands/test.py", line 26, in run_from_argv
    super().run_from_argv(argv)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/commands/test.py", line 59, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/test/runner.py", line 601, in run_tests
    old_config = self.setup_databases()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/test/runner.py", line 548, in setup_databases
    self.parallel, **kwargs
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/test/utils.py", line 176, in setup_databases
    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/backends/base/creation.py", line 68, in create_test_db
    run_syncdb=True,
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 141, in call_command
    return command.execute(*args, **defaults)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
    output = self.handle(*args, **options)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 200, in handle
    fake_initial=fake_initial,
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/migration.py", line 112, in apply
    operation.state_forwards(self.app_label, project_state)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/operations/models.py", line 86, in state_forwards
    list(self.managers),
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py", line 97, in add_model
    self.reload_model(app_label, model_name)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py", line 158, in reload_model
    self._reload(related_models)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py", line 191, in _reload
    self.apps.render_multiple(states_to_be_rendered)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py", line 306, in render_multiple
    model.render(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py", line 575, in render
    return type(self.name, bases, body)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/models/base.py", line 152, in __new__
    new_class.add_to_class(obj_name, obj)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/models/base.py", line 315, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/models/manager.py", line 115, in contribute_to_class
    setattr(model, name, ManagerDescriptor(self))
TypeError: attribute name must be string, not 'bytes'

tl;dr: The name, a value coming from the migration sites.0001_initial, was interpreted by setattr(model, name, ManagerDescriptor(self)) as a bytestring.

Steps in detail

• django.contrib.sites ​migration 0001 is missing the unicode_literals statement. Under Python 3, e.g. its ​model manager name 'objects' will be interpreted as bytestring b'objects'.
• While running ./manage.py migrate, the ​django.db.models.manager L113 reads the migration's model manager name expecting a unicode string, and not defending against a bytestring.

I was able to fix my own existing migrations (created in Django 1.x) either by hand (inserting the unicode_literals import, or deleting the occasional explicit b"" prefix), or by squashing multiple migrations (where too numerous to manually edit) under Django 2.x, which inserted the unicode_literals statement into my newly created migrations.
However, this approach won't work with third party migrations without manually patching them (which won't work on e.g. CI and is brittle) or forking (creating a maintenance trail for me).

Suggested patches

• Primarily, add from __future__ import unicode_literals to django.contrib.sites' ​migration 0001.
• As a longer term fix, replace ​django.db.models.manager L113 setattr(model, name, ManagerDescriptor(self)) with setattr(model, force_text(name), ManagerDescriptor(self)) (and import force_text).
• If a bytestring is found, a Django warning should prompt the user to clean up the offending migration by adding the unicode_literals import. This would be more graceful and efficient than presenting the TypeError.

The benefit of patching db.models.manager would be that all Python 2.x-style migrations, especially of third party packages, would work in Python 3.x-based Django projects without the need to patch them.

Has anyone else encountered this bug? Happy to PR whichever fix is deemed most appropriate (bear with me, first Django bug report).