Automatic migrations prevent creation of initial database table layout

[SimonSteinberger]

Using an AbstractUser based user model with Django 1.8 RC1 causes the management command to create the table layout on a *new and empty database* to fail: python manage.py migrate

Traceback:
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "...\site-packages\django\core\management\__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "...\site-packages\django\core\management\__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "...\site-packages\django\core\management\base.py", line 390, in run_from_argv
    self.execute(*args, *\*cmd_options)
  File "...\site-packages\django\core\management\base.py", line 441, in execute
    output = self.handle(*args, *\*options)
  File "...\site-packages\django\core\management\commands\migrate.py", line 179, in handle
    created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
  File "...\site-packages\django\core\management\commands\migrate.py", line 317, in sync_apps
    cursor.execute(statement)
  File "...\site-packages\django\db\backends\utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "...\site-packages\django\db\backends\utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "...\site-packages\django\db\utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "...\site-packages\django\db\backends\utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "auth_group" does not exist

What happens is that the django_migrations table gets created. Creation of all other tables are discarded/reverted due to this error.

If necessary, I may be able to provide a minimal code for reproducing this issue. It's happening in five independent projects for me. I hope I'm doing something wrong here, but it may just as well be a bug.

Possible solution is not to do any automatic migrations when creating a fresh database table layout.

[Tim Graham]

Yes, we will need some more details.

Do your apps have migrations? Are you mixing apps with migrations and apps without migrations?

[Markus Holtermann]

I digged into the issue. It occurs on Postgres when the app with the custom user model does not have migrations due to the dependency of your user model on auth.Group. In accordance with the documentation this is an unsupported behavior:

Be aware, however, that unmigrated apps cannot depend on migrated apps, by the very nature of not having migrations. This means that it is not generally possible to have an unmigrated app have a ForeignKey or ManyToManyField to a migrated app; some cases may work, but it will eventually fail. ​https://docs.djangoproject.com/en/1.8/topics/migrations/#dependencies

I thus close this ticket as expected behavior.

[SimonSteinberger]

You're right - those are PostgreSQL driven apps. Can you point me into the right direction how to avoid this issue, when using a very simple AbstractUser model without any additional fields?

I think that's a pretty normal use case. A solution should be pointed out in the documentation. And despite I'm pretty experienced with Django in general (and very happy with it), I can't see how to avoid this problem ...

[SimonSteinberger]

Well, I've digged into this myself now, and I must say: This is completely nonsensical:

When using the AbstractUser class to create a custom user model *exactly as outlined in Django's docs* at least the first "manage.py migrate" must work and must create an empty database table layout. The sole purpose of the AbstractUser class is to extend the basic user with more fields. But as you describe it, the whole class is just useless and doomed to fail. By the way: Permissions and auth group tables are automatically created with the AbstractUser model.

So, either I'm doing something plainly wrong here, or the docs are lacking some critical information/solutions, or the whole construct doesn't male sense as it is.

[Carl Meyer]

Did you run python manage.py makemigrations appname first, where "appname" is the name of the app containing your custom user model inheriting from AbstractUser?

[SimonSteinberger]

Okay, that worked. Thanks a lot!

I think this is an extremely important point that should appear in a warning box at the AbstractUser documentation.

[Juergen Brendel]

I have the same problem, but the suggested solution does not work for me, unfortunately.

Specifically, it's a 1.7.6 application (backed by Postgres), which I would like to update to 1.8.9. I'm currently happy to completely reset the database, so there are no changes, I just need the initial schema creation. There are NO migrations in any of my apps, so no 'migrations' folders exist anywhere.

I also have a custom user model (inheriting from 'AbstractUser'). When I do as suggested (python manage.py makemigrations <appname>), some migrations are created (even though there is no DB that might require migrations), but the subsequent 'migrate' still fails.

Needless to say, this worked well for me on 1.7.6, but I'm now completely stuck with 1.8.9.

One additional bit of information:

I can get the migrations to finally run through with the following hacky procedure:

1. python manage.py makemigrations <my-app-with-customer-user-model>.
2. python manage.py migrate <my-app-with-customer-user-model>. This ends with django.db.utils.ProgrammingError: relation "django_site" does not exist, but we'll continue anyway...
3. python manage.py migrate. Runs through without error. I need to make sure to have 'sites' listed before 'contentype' in my settings file.

During step (2) a few tables are created, even though it eventually fails. Specifically, the "contenttypes" and "auth" related tables. This is important, since without "contenttypes" I can't migrate sites and without auth I can't migrate contenttypes. The only way I found to create those tables is by running a failed migration attempt for my app. There doesn't seem to be a way to create those tables in a way that doesn't result in an error at some point, due to what appears to be some sort of circular dependency between sites, contentypes and auth.

Of course, such a hacky work-around is not really useful, since the test-database can't be created in that manner (its automated creation needs to work without hack).

It seems I at least need to get this to the stage where I can just run python manage.py migrate (or syncdb for that matter, which also fails since it needs my app with the custom user model). Any help or insight would be greatly appreciated.

Thank you very much!

[Simon Charette]

Hi jbrendel,

From your report I suspect your issue has little to do with migrations.

Is it possible to provide the full traceback of the ProgrammingError?

[Juergen Brendel]

Hello!

Here is the full traceback:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 226, in handle
    emit_post_migrate_signal(created_models, self.verbosity, self.interactive, connection.alias)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/core/management/sql.py", line 280, in emit_post_migrate_signal
    using=db)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 189, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/contrib/sites/management.py", line 20, in create_default_site
    if not Site.objects.using(using).exists():
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/models/query.py", line 586, in exists
    return self.query.has_results(using=self.db)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 484, in has_results
    return compiler.has_results()
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 811, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/jbrendel/projects/zzyyxx/repo/zzyyxx_test2/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "django_site" does not exist
LINE 1: SELECT (1) AS "a" FROM "django_site" LIMIT 1
                               ^

[Simon Charette]

I see, the code that makes sure to create a default site assumes that the Site's table exist after each migrate even if the migrations to apply were restricted to a subset that don't include the initial sites migration.

[Juergen Brendel]

Thank you for looking into this. So, is that a Django bug? Is this something I can work around via particular settings?

[Juergen Brendel]

@charettes saw something in the ProgrammingError that I had submitted.

But now I'm still stuck trying to figure out what to do about it. Is there any kind of work around I could use? Is this something that I did wrong? Or is this a bug in Django?

For now, I have placed a patch into "django/db/backends/base/creation.py" (inside of the create_test_db() function), in which I use "call_command()" twice (once with 'migrate', 'accounts', which causes an exception I catch and ignore, then once more with just 'migrate'). This allows me to run my unit tests.

Obviously, this is terrible, since I don't want to have to patch Django with an ugly workaround every time I install it. There must be a better way, but what is it?

Any help would be appreciated.

Thank you very much!

[Simon Charette]

Sorry for the delay jbrendel,

The issue you are hitting with contrib.sites is similar to what #22411 was for contrib.contenttypes and caused by the fact pre_migrate signals receivers have no way to determine the current state of applied migrations.

This is an issue that is also blocking #24067 where I initially thought that dispatching the migration plan (#24100) would expose enough details to fix it.

Could you provide us the traceback you get when trying to run your test suite with ./manage.py test -v2?

[Juergen Brendel]

Thank you for getting back to me.

I did as you asked and ran "./manage.py test -v2 <appname>" for one of my apps. Here is the output:

$ python manage.py test -v2 utils
Creating test database for alias 'default' ('test_db_utils')...
Operations to perform:
  Synchronize unmigrated apps: staticfiles, admindocs, utils, messages, social_auth, toplevel, django_extensions, check_js, crispy_forms
  Apply all migrations: sessions, admin, auth, sites, contenttypes, accounts, appcore
Synchronizing apps without migrations:
  Creating tables...
    Creating table social_auth_usersocialauth
    Creating table social_auth_nonce
    Creating table social_auth_association
    Running deferred SQL...
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/commands/test.py", line 74, in execute
    super(Command, self).execute(*args, **options)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/commands/test.py", line 90, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django_coverage/coverage_runner.py", line 76, in run_tests
    extra_tests, **kwargs)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/test/runner.py", line 210, in run_tests
    old_config = self.setup_databases()
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/test/runner.py", line 166, in setup_databases
    **kwargs
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/test/runner.py", line 370, in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/db/backends/base/creation.py", line 383, in create_test_db
    test_flush=not keepdb,
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 179, in handle
    created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 318, in sync_apps
    cursor.execute(statement)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/jbrendel/projects/zzyyxx/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "accounts_usermodel" does not exist

I hope this helps. Please let me know if there's any other info I can provide.

Thank you very much!

[Simon Charette]

Hi jbrendel,

From what I can see you have an application with no migrations (social_auth) depending on an application with migrations (accounts) which is not supported.

It looks like social_auth ships with migrations since at least v0.2.0, which version are you using? Let's move this discussion to the #django IRC channel instead, my nickname is charettes.

[Juergen Brendel]

Hello charettes,

I tried to find you on the Django IRC channel, but it looks like we keep missing each other there.

I use version 0.7.28 of social_auth. And, yes, it does come with migrations. I confirmed that they were installed by pip. However, those seems to be migrations of the wrong kind (south?), so they are not recognized.

Your comments, however, allows me to narrow my search, so I was finally able to find a solution that someone had posted on StackOverflow: ​http://stackoverflow.com/a/33562236

In short: Find the original migrations that django-social-auth came with. Delete them. Then run python manage.py makemigrations social_auth. This will replace the original migrations with 'the right kind' of migrations. When you then run python manage.py migrate everything works as advertised.

So, I guess the real problem is that django-social-auth does not yet come by default with the correct kind of migrations, which could be used by Django 1.8 and up.

A secondary problem, however, is that Django (when you run migrate) does not provide any illuminating warnings. I'm wondering whether it could include a check for old-style migrations, so that it can print a helpful error message?

Interestingly, you can do python manage.py makemigrations social_auth right after the pip install... and it will create a 0001_initial migration, effectively overwriting the old 0001_initial migration that django-social-auth came with. Unfortunately, there is a second migration (0002_...), which is also an old-style south migration. It remains in place and untouched by makemigrations. So, when I then do python manage.py migrate Django still complains with django.db.migrations.loader.BadMigrationError: Migrated app 'social_auth' contains South migrations. Make sure all numbered South migrations are deleted prior to creating Django migrations.

That's why the contents of the migration directory that comes with django-social-auth needs to be deleted at first.

I'm lucky, because I don't need to actually migrate old databases, I can start from scratch, but I could imagine that this could cause some issues for people that do need to migrate actual databases.