id summary reporter owner description type status component version severity resolution keywords cc stage has_patch needs_docs needs_tests needs_better_patch easy ui_ux 29632 "After migrating a Django project generated with cookiecutter-django from Python 2.x to 3.x, db migration of (cookiecutter-django generated, not Django module) django.contrib.sites fails with ""TypeError: attribute name must be string, not 'bytes'""" Florian Mayer nobody "I'm migrating an existing Django project from Python 2.7 / Django 1.11 to Python 3.6 / Django 2.0.7 (latest stable at the time of writing). Migrations are finicky as they (used to) be created without `from __future__ import unicode_literals`, therefore all strings (such as model manager names e.g. ""'objects'"") would be interpreted as bytestrings in Python 3.x. [https://code.djangoproject.com/ticket/24701 Ticket 24701] is the closest issue I could find relating to this error, and it led to Django now inserting the `unicode_literals` import into newly created migrations. This however doesn't fix existing migrations, and while Django's migrations are doing now a decent job to write plain un-prefixed `""strings""` in field names, sometimes a model manager name or a more exotic migration command's argument slips through as `b""explicitly prefixed bytestring""`. == Environment == * [https://github.com/dbca-wa/wastd/blob/master/.circleci/config.yml CircleCI v2 build] using Python 3.6.1 (freshly upgraded from Python 2.7) * [https://github.com/dbca-wa/wastd/blob/master/requirements/base.txt requirements] Django 2.0.7 * The Django project was written Django 1.9.7 to 1.11.x, and recently upgraded to Django 2.x. * The migrations were created using Django 1.9.7 to 1.11.x * Third party app migrations are potentially ancient * Project generated by cookiecutter-django, which generates a stand-alone contrib.sites app ([https://cookiecutter-django.readthedocs.io/en/latest/faq.html#why-is-there-a-django-contrib-sites-directory-in-cookiecutter-django rationale here]) * '''Solution: There was a stray bytestring in cookiecutter-django's contrib.sites migration.''' == Stack trace == [https://circleci.com/gh/dbca-wa/wastd/823 Failing CircleCI build] fails at `./manage.py migrate`. Can reproduce on Ubuntu 16.04 (Python 3.6.x Anaconda) and 14.04 (Python 3.4.3). {{{ Running migrations: Applying contenttypes.0001_initial... OK Applying contenttypes.0002_remove_content_type_name... OK [... working migrations ...] Applying sites.0001_initial...Traceback (most recent call last): File ""manage.py"", line 12, in execute_from_command_line(sys.argv) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/__init__.py"", line 371, in execute_from_command_line utility.execute() File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/__init__.py"", line 365, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/commands/test.py"", line 26, in run_from_argv super().run_from_argv(argv) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/base.py"", line 288, in run_from_argv self.execute(*args, **cmd_options) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/base.py"", line 335, in execute output = self.handle(*args, **options) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/commands/test.py"", line 59, in handle failures = test_runner.run_tests(test_labels) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/test/runner.py"", line 601, in run_tests old_config = self.setup_databases() File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/test/runner.py"", line 548, in setup_databases self.parallel, **kwargs File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/test/utils.py"", line 176, in setup_databases serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True), File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/backends/base/creation.py"", line 68, in create_test_db run_syncdb=True, File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/__init__.py"", line 141, in call_command return command.execute(*args, **defaults) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/base.py"", line 335, in execute output = self.handle(*args, **options) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py"", line 200, in handle fake_initial=fake_initial, File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/executor.py"", line 117, in migrate state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/executor.py"", line 147, in _migrate_all_forwards state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/executor.py"", line 244, in apply_migration state = migration.apply(state, schema_editor) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/migration.py"", line 112, in apply operation.state_forwards(self.app_label, project_state) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/operations/models.py"", line 86, in state_forwards list(self.managers), File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py"", line 97, in add_model self.reload_model(app_label, model_name) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py"", line 158, in reload_model self._reload(related_models) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py"", line 191, in _reload self.apps.render_multiple(states_to_be_rendered) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py"", line 306, in render_multiple model.render(self) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/migrations/state.py"", line 575, in render return type(self.name, bases, body) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/models/base.py"", line 152, in __new__ new_class.add_to_class(obj_name, obj) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/models/base.py"", line 315, in add_to_class value.contribute_to_class(cls, name) File ""/home/circleci/repo/venv/lib/python3.6/site-packages/django/db/models/manager.py"", line 115, in contribute_to_class setattr(model, name, ManagerDescriptor(self)) TypeError: attribute name must be string, not 'bytes' }}} tl;dr: The `name`, a value coming from the migration `sites.0001_initial`, was interpreted by `setattr(model, name, ManagerDescriptor(self))` as a bytestring. Confusingly, this migration was inside the project, not inside django.contrib.sites. == Steps in detail == * django.contrib.sites [https://github.com/django/django/blob/master/django/contrib/sites/migrations/0001_initial.py migration 0001] is missing the `unicode_literals` statement. Under Python 3, e.g. its [https://github.com/django/django/blob/master/django/contrib/sites/migrations/0001_initial.py#L28 model manager name] `'objects'` will be interpreted as bytestring `b'objects'`. * While running `./manage.py migrate`, the [https://github.com/django/django/blob/master/django/db/models/manager.py#L113 django.db.models.manager L113] reads the migration's model manager name expecting a unicode string, and not defending against a bytestring. I was able to fix my own existing migrations (created in Django 1.x) either by hand (inserting the `unicode_literals` import, or deleting the occasional explicit `b""""` prefix), or by squashing multiple migrations (where too numerous to manually edit) under Django 2.x, which inserted the `unicode_literals` statement into my newly created migrations. However, this approach won't work with third party migrations without manually patching them (which won't work on e.g. CI and is brittle) or forking (creating a maintenance trail for me). == Suggested patches == * ~~Primarily, add `from __future__ import unicode_literals` to django.contrib.sites' [https://github.com/django/django/blob/master/django/contrib/sites/migrations/0001_initial.py migration 0001].~~ * As a longer term fix, replace [https://github.com/django/django/blob/master/django/db/models/manager.py#L113 django.db.models.manager L113] `setattr(model, name, ManagerDescriptor(self))` with `setattr(model, force_text(name), ManagerDescriptor(self))` (and import `force_text`). * If a bytestring is found, a Django warning should prompt the user to clean up the offending migration ~~by adding the `unicode_literals` import~~. This would be more graceful and efficient than presenting the TypeError. The benefit of patching db.models.manager would be that all Python 2.x-style migrations, especially of third party packages, would work in Python 3.x-based Django projects without the need to patch them. ~~Has anyone else encountered this bug? Happy to PR whichever fix is deemed most appropriate (bear with me, first Django bug report).~~" Bug closed Migrations dev Normal invalid migration, unicode, bytestring, TypeError Unreviewed 0 0 0 0 1 0