Opened 6 years ago

Last modified 4 months ago

#25313 new New feature

Document how to migrate from a built-in User model to a custom User model

Reported by: Carl Meyer Owned by: nobody
Component: Documentation Version: 1.8
Severity: Normal Keywords:
Cc: Carsten Fuchs Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Julien Palard)

So far our answer here has been "sorry, you can't do it, unless you're very familiar with the depths of the migrations system and willing to put a lot of time into it."

I don't believe that this is a tenable or defensible answer. It puts too many of our users, too frequently, into an impossible quandary. I think we need to clearly document how you can do it today, even if the process is nasty and ugly. Hopefully seeing that nasty and ugly documentation might clarify what could be improved to make the process less nasty and ugly.

Change History (25)

comment:1 Changed 6 years ago by Tim Graham

Component: MigrationsDocumentation
Description: modified (diff)
Triage Stage: UnreviewedAccepted

comment:2 Changed 6 years ago by Aymeric Augustin

I did it at least twice. Unfortunately I don't remember all the details.

I think a reasonable procedure is:

  1. Create a custom user model identical to auth.User, call it User (so many-to-many tables keep the same name) and set db_table='auth_user' (so it uses the same table)
  2. Throw away all your migrations
  3. Recreate a fresh set of migrations
  4. Sacrifice a chicken, perhaps two if you're anxious; also make a backup of your database
  5. Truncate the django_migrations table
  6. Fake-apply the new set of migrations
  7. Unset db_table, make other changes to the custom model, generate migrations, apply them

It is highly recommended to do this on a database that enforces foreign key constraints. Don't try this on SQLite on your laptop and expect it to work on Postgres on the servers!

comment:3 Changed 6 years ago by Carl Meyer

Hmm. I thought I recalled you mentioning (at DUTH last year?) that you achieved it using SeparateDatabaseAndState, without the need to wipe migrations and start over. Maybe that was someone else.

comment:4 Changed 6 years ago by Aymeric Augustin

At some point I thought it was doable with SeparateDatabaseAndState but eventually I realized that it isn't.

When you change settings.AUTH_USER_MODEL, suddenly Django has a different view of the migration history. Piling hacks cannot hide this fact. I don't think it's possible to salvage migration history when changing AUTH_USER_MODEL.

comment:5 Changed 6 years ago by Carl Meyer

Hmm. Clearly it must be partially possible, otherwise you could never do this at all if you use any third-party apps that link to User (since you won't be changing their migration files). The whole reason we special-case swappable models in migrations (instead of just treating them concretely) is to allow for migrations to not be dependent on the value of AUTH_USER_MODEL, so that reusable apps depending on User can still generate workable migrations.

It's true that changing AUTH_USER_MODEL changes the _meaning_ of historical migrations in some sense, but it still seems to me that if our approach for reusable apps actually works, the same migration files ought to be salvageable (presuming that when you switch AUTH_USER_MODEL you point it to a new model that is initially exactly the same as the previous one, and then only modify it in later, separate migrations).

But you've actually done this and I haven't, so I'm probably wrong...

comment:6 in reply to:  5 Changed 6 years ago by Aymeric Augustin

Replying to carljm:

Hmm. Clearly it must be partially possible, otherwise you could never do this at all if you use any third-party apps that link to User (since you won't be changing their migration files).

Indeed you don't change the migration files. But you drop django_migrations, change settings.AUTH_USER_MODEL and repopulate django_migrations. While some migrations have the same name in django_migrations, you don't have the same set of migrations and their semantic has changed. Specifically:

  • you add at least the migration that creates your custom user model
  • the auth.000_initial migration has a different semantic because it doesn't create a table for auth.User

The whole reason we special-case swappable models in migrations (instead of just treating them concretely) is to allow for migrations to not be dependent on the value of AUTH_USER_MODEL, so that reusable apps depending on User can still generate workable migrations.

That's the trick. A migration file that contains a migration that uses the swappable option isn't a self-contained definition. Only the combination of the migration file and the value of getattr(django.conf.settings, swappable) is. This isn't reflected in the structure of django_migrations because Django currently assumes AUTH_USER_MODEL to be immutable.

It's true that changing AUTH_USER_MODEL changes the _meaning_ of historical migrations in some sense, but it still seems to me that if our approach for reusable apps actually works, the same migration files ought to be salvageable (presuming that when you switch AUTH_USER_MODEL you point it to a new model that is initially exactly the same as the previous one, and then only modify it in later, separate migrations).

I remember feeling smart, then making a huge mess of a SQLite database, feeling dumb, editing the dump manually to fix broken FK constaints, feeling lucky.

But you've actually done this and I haven't, so I'm probably wrong...

Well, perhaps there's a way.

Even then I'd recommend the procedure I suggested above because:

  • it's reasonably convenient: people have a fair chance to execute it successfully
  • it makes it clear that you're voiding your warranty (not that Django comes with a warranty, but you get the point)
  • it's possible to reason about why it works

comment:7 Changed 6 years ago by Shai Berger

<brainstorming idea="halfbaked">

Perhaps it is possible to create a migration operation for changing a swappable model.

Something like:

ChangeSwappableModel(
    setting="AUTH_USER_MODEL",
    old="auth.User",
    new="my_auth.User"
)

This would be a migration in the my_auth app.

It would need to introspect the database, checking the constraints corresponding to my_auth.User's reverse relationships, and verifying that they indeed point to the correct table (if they point to auth_user, change them; if to my_auth_user, leave them be; otherwise, error out).

To change the swappable model, one would:
1) Create the set of migrations for creating the new user model, porting existing data to it, and the ChangeSwappableModel operation;
2) Create a "squashing" migration replacing them by just creating the new model, so new databases don't have to suffer

I think with careful migrations embargos in the right times this could be made to work. I don't have it in me to work out all the details now, and I'm probably missing something essential.

</brainstorming>

comment:8 Changed 5 years ago by James Addison

I think I've followed a very similar process in previous projects (older Django versions) to what Aymeric mentions above in https://code.djangoproject.com/ticket/25313#comment:2 - although I don't think you can ever 'unset' db_table without having to do low level SQL changes?

comment:9 in reply to:  2 Changed 4 years ago by Justin Smith

Replying to Aymeric Augustin:

I did it at least twice. Unfortunately I don't remember all the details.

I think a reasonable procedure is:

  1. Create a custom user model identical to auth.User, call it User (so many-to-many tables keep the same name) and set db_table='auth_user' (so it uses the same table)
  2. Throw away all your migrations
  3. Recreate a fresh set of migrations
  4. Sacrifice a chicken, perhaps two if you're anxious; also make a backup of your database
  5. Truncate the django_migrations table
  6. Fake-apply the new set of migrations
  7. Unset db_table, make other changes to the custom model, generate migrations, apply them

It is highly recommended to do this on a database that enforces foreign key constraints. Don't try this on SQLite on your laptop and expect it to work on Postgres on the servers!

Just recently had to go through this process using a Microsoft SQL Server backend and used the steps above as my guideline. Just thought I'd drop in and include some of my notes just in case they can help anyone in the future.

Notes (by step):

  1. Make sure the custom user model identical to auth.User is an AbstractUser model. I originally made this mistake because I did an inspectdb auth_user and just copy/pasted so I left it as models.Model at first. Since I copied and pasted from inspectdb I went ahead and removed managed = False
  2. Quick shortcut to delete migrations for all apps in a project I used was find . -path "*/migrations/*.py" -not -name "__init__.py" -delete.
  3. No additional notes
  4. Not kidding about the back up I had to start over a few times
  5. No additional notes
  6. I did --fake-initial first few times and not --fake
  7. No notes

Thank you very much for posting this in the first place I am not sure I would have figured this out on my own and saved us mid-project. I have learned my lesson about starting a django project and not setting up custom user model.

comment:10 Changed 4 years ago by Luke Plant

I created this project which was my attempt to automate the process:

https://bitbucket.org/spookylukey/django_custom_user_migration

However, I think Aymeric's solution looks better.

comment:11 Changed 4 years ago by Pi Delport

These are steps we took to switch our system to a custom AUTH_USER_MODEL, for the record:

  1. (Take full backups!)
  2. Dump the database with: django-admin dumpdata --natural-primary --natural-foreign --exclude contenttypes.contenttype
  3. Run the JSON dump through a script that rewrites references from the old user model to the new one. (See below.)
  4. Define our new custom user model as a AbstractUser subclass, with no other schema changes. Update AUTH_USER_MODEL to it, nuke all our app's old migrations, and make fresh initial migrations.
  5. Create and django-admin migrate a fresh new database, and load the rewritten dump.

After this point, we can customise our user model with normal Django migrations.

The script to rewrite the dump iterates through the list of objects, and rewrites:

  • The user's' model itself.
  • The user's user_permissions field's references.
  • The auth.group permissions field's references.
  • The auth.permission and admin.logentry content_type fields.
  • Any other references to the old auth.User type will need rewriting too.

comment:12 in reply to:  2 ; Changed 3 years ago by Dustin Torres

Replying to Aymeric Augustin:

I did it at least twice. Unfortunately I don't remember all the details.

I think a reasonable procedure is:

  1. Create a custom user model identical to auth.User, call it User (so many-to-many tables keep the same name) and set db_table='auth_user' (so it uses the same table)
  2. Throw away all your migrations
  3. Recreate a fresh set of migrations
  4. Sacrifice a chicken, perhaps two if you're anxious; also make a backup of your database
  5. Truncate the django_migrations table
  6. Fake-apply the new set of migrations
  7. Unset db_table, make other changes to the custom model, generate migrations, apply them

It is highly recommended to do this on a database that enforces foreign key constraints. Don't try this on SQLite on your laptop and expect it to work on Postgres on the servers!

Thanks for this, I followed your steps and Justin Smith's notes and want to add one more:

6b. Any model that has a link to ContentType may be linked to the now stale auth.User content type. I went through and updated all models that were pointing to auth.User content type to my new User model. You can find out what type of objects would be removed due to cascading delete of the stale auth.User content type by running manage.py remove_stale_contenttypes and make sure to answer NO when asked if you want to proceed.

If you don't unset db_table the above step may be optional since the content type lookup will use auth.User model which will still have a valid database table (although I don't recommend relying on this).

comment:13 in reply to:  12 Changed 3 years ago by Miko

Thank you Aymeric et. al for these steps, they worked for me. I made one silly mistake in 1. not creating the user properly. For other people who might also get stuck on this, here is what your User model should look like for step 1:

class User(AbstractUser):
    class Meta:
        db_table = 'auth_user'

Replying to Dustin Torres:

Replying to Aymeric Augustin:

I did it at least twice. Unfortunately I don't remember all the details.

I think a reasonable procedure is:

  1. Create a custom user model identical to auth.User, call it User (so many-to-many tables keep the same name) and set db_table='auth_user' (so it uses the same table)
  2. Throw away all your migrations
  3. Recreate a fresh set of migrations
  4. Sacrifice a chicken, perhaps two if you're anxious; also make a backup of your database
  5. Truncate the django_migrations table
  6. Fake-apply the new set of migrations
  7. Unset db_table, make other changes to the custom model, generate migrations, apply them

It is highly recommended to do this on a database that enforces foreign key constraints. Don't try this on SQLite on your laptop and expect it to work on Postgres on the servers!

Thanks for this, I followed your steps and Justin Smith's notes and want to add one more:

6b. Any model that has a link to ContentType may be linked to the now stale auth.User content type. I went through and updated all models that were pointing to auth.User content type to my new User model. You can find out what type of objects would be removed due to cascading delete of the stale auth.User content type by running manage.py remove_stale_contenttypes and make sure to answer NO when asked if you want to proceed.

If you don't unset db_table the above step may be optional since the content type lookup will use auth.User model which will still have a valid database table (although I don't recommend relying on this).

comment:14 Changed 3 years ago by David Sanders

On the point of "Throw away all your migrations", I'd like to add a note that some data migrations that setup fixtures may need to be kept. This may involve extracting these operations from their respective migrations and updating them to use the latest schema then appending them to the resulting initial migrations generated.

Version 0, edited 3 years ago by David Sanders (next)

comment:15 Changed 2 years ago by Adam Johnson

For anyone landing here, Tobias McNulty did half the work documenting it in this blog post: https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/ .

comment:16 Changed 21 months ago by Carsten Fuchs

Cc: Carsten Fuchs added

comment:17 Changed 20 months ago by Jakub Dorňák

I tried to address the mid-project migration in https://pypi.org/project/django-user-unique-email/
It seems to work with SQLite and MySQL. I'll appreciate any feedback.

comment:18 Changed 20 months ago by Carsten Fuchs

Based on

I tried to write a summary. These steps worked well for me, although I'm sure there is room for improvement:

Assumptions

  • Your project doesn't have a custom user model yet.
  • All existing users must be kept.
  • There are no pending migrations and all existing migrations are applied.
  • It is acceptable that all previous migrations are lost and can no longer be unapplied, even if you use version control to checkout old commits that still have the migration files. This is the relevant downside of this approach.

Preparations

  • Make sure that your own reusable apps (apps that are intended to be used by others) use the generic reference methods.
  • I suggest to not do the same with your project apps: The switch to a custom user model is only done once per project and never again. It is easier (and in my opinion also clearer) to change from django.contrib.auth.models import User to something else (as detailed below) than replacing it with generic references that are not needed in project code.
  • Make sure that you have a backup of your code and database!

Update the code

  • You can create the new user model in any existing app or a newly created one. My preference is to create a new app:
        ./manage.py startapp Accounts
    
    I chose the name "Accounts", but any other name works as well.
  • Aymeric: „Create a custom user model identical to auth.User, call it User (so many-to-many tables keep the same name) and set db_table='auth_user' (so it uses the same table).“ In Accounts/models.py:
        from django.contrib.auth.models import AbstractUser
        from django.db import models
    
    
        class User(AbstractUser):
            class Meta:
                db_table = 'auth_user'
    
  • In settings.py, add the app to INSTALLED_APPS and update the AUTH_USER_MODEL setting:
        INSTALLED_APPS = (
            # ...
            'Accounts',
        )
    
        AUTH_USER_MODEL = 'Accounts.User'
    
  • In your project code, replace all imports of the Django user model:
        from django.contrib.auth.models import User
    
    with the new, custom one:
        from Accounts.models import User
    
  • Delete all old migrations. (Beforehand, see if comment 14 is relevant to you!) For example, in the project root:
        find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
        find . -path "*/migrations/*.pyc" -delete
    
  • Create new migrations from scratch:
        ./manage.py makemigrations
    
  • Make any changes to your admin.py files as required. (I cannot give any solid information here, but this is not crucial for the result and so the details can still be reviewed later.)
  • Make sure that your testsuite completes successfully! (A fresh test database must be used, it cannot be kept from previous runs.)
  • At this point, the changes to the code are complete. This is a good time for a commit.

Note that we're done – except that the new migration files mismatch the contents of the django_migrations table.

(It may even be possible to serve your project at this point: It's easy to back off before the database is actually changed. Only do this if you understand that you cannot even touch the migrations system as long as the steps below are not completed!)

Update the database

  • Truncate the django_migrations table. MySQL 8 example:
        TRUNCATE TABLE django_migrations;
    
    This is possibly different for other databases or verions of MySQL < 8.
  • Fake-apply the new set of migrations
        ./manage.py migrate --fake
    
  • Check the ContentTypes as described at comment 12

Conclusion

  • The upgrade to the custom user model is now complete. You can make changes to this model and generate and apply migrations for it as with any other models.
  • As a first step, you may wish to unset db_table and generate and apply the resulting migrations.
  • In my opinion, the startproject management command should anticipate the introduction of a custom user model.

comment:19 Changed 20 months ago by Carsten Fuchs

After the change, rewriting the ContentType for the admin log's LogEntry instances was helpful:

./manage.py shell
>>>
>>> from django.contrib.admin.models import LogEntry
>>> from django.contrib.contenttypes.models import ContentType
>>>
>>> auth_user = ContentType.objects.get(app_label='auth', model='user')
>>> accouts_user = ContentType.objects.get(app_label='Accounts', model='user')
>>>
>>> for le in LogEntry.objects.filter(content_type=auth_user):
...     le.content_type = accouts_user
...     le.save()
...

comment:20 Changed 19 months ago by Eric Theise

Many thanks to Carsten Fuchs for integrating the various resources into a single procedure that worked well for me in my simple situation. One caveat: after running the find commands in the Delete all old migrations step I received a ModuleNotFoundError: No module named 'django.db.migrations.migration' error and had to uninstall/reinstall Django before proceeding.

comment:21 Changed 12 months ago by BrandonWoodward

I had posted a solution but it's still in progress - will update shortly.

Last edited 12 months ago by BrandonWoodward (previous) (diff)

comment:22 Changed 10 months ago by Julien Palard

Description: modified (diff)

I tried the procedure from comment 18 today and it worked perfectly (see my commit).

P.S.: About comment 20 I bet the find deleted things in the venv. Maybe add a -maxdepth 3?

comment:23 Changed 5 months ago by Johannes Wilm

Is anyone working on fixing this in Django itself? The workarounds may work in a lot of cases, but when you have an application set up to rely on the Django migration system for any database change and have it deployed on thousands of servers that are not under your direct control, it is somewhat useless.

Would it not be possible to create one special type of migration for this purpose? Maybe something as simple as let the migration that creates the user model check wither auth.User has been in use, and if so, copy over entries from there.

comment:24 Changed 5 months ago by Johannes Wilm

The following approach has not been tested with all database backends (only sqlite), but it seems to work by only using migrations, no raw SQL and not destroying any existing migrations. This is based of Tobias McNulty's walk-through [1], with some modifications to avoid doing anything "nasty".


  1. Assumptions:
  • You have an existing project without a custom user model.
  • You're using Django's migrations, and all migrations are up-to-date (and have been applied to the production database).
  • You have an existing set of users that you need to keep, and any number of models that point to Django's built-in User model.
  • You have an existing django app with existing migrations to which you want to add the custom user model.
  • You need the same code to run both for new installations of your code base and for existing ones.
  1. First, assess any third party apps to make sure they either don't have any references to the Django's User model, or if they do, that they use Django's generic methods for referencing the user model.
  1. Next, do the same thing for your own project. Go through the code looking for any references you might have to the User model, and replace them with the same generic references. In short, you can use the get_user_model() method to get the model directly, or if you need to create a ForeignKey or other database relationship to the user model, use settings.AUTH_USER_MODEL (which is simply a string corresponding to the appname.ModelName path to the user model).

Note that get_user_model() cannot be called at the module level in any models.py file (and by extension any file that a models.py imports), since you'll end up with a circular import. Generally, it's easier to keep calls to get_user_model() inside a method whenever possible (so it's called at run time rather than load time), and use settings.AUTH_USER_MODEL in all other cases. This isn't always possible (e.g., when creating a ModelForm), but the less you use it at the module level, the fewer circular imports you'll have to stumble your way through.

  1. In the existing app in which you want the custom User model to live (we will call it "user" in this example), add the following to the models.py file:
from django.db import models
from django.contrib.auth.models import AbstractUser


class User(AbstractUser):
    class Meta:
        db_table = 'auth_user'
  1. Set this new model as the default user model in the settings.py file by adding: AUTH_USER_MODEL = "user.User".
  1. Edit the migration with a name starting with "0001" of this app.

Under "operations" paste the following as the first item:

migrations.CreateModel(
            name='User',
            fields=[
                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('password', models.CharField(max_length=128, verbose_name='password')),
                ('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
                ('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
                ('username', models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150, unique=True, validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], verbose_name='username')),
                ('first_name', models.CharField(blank=True, max_length=150, verbose_name='first name')),
                ('last_name', models.CharField(blank=True, max_length=150, verbose_name='last name')),
                ('email', models.EmailField(blank=True, max_length=254, verbose_name='email address')),
                ('is_staff', models.BooleanField(default=False, help_text='Designates whether the user can log into this admin site.', verbose_name='staff status')),
                ('is_active', models.BooleanField(default=True, help_text='Designates whether this user should be treated as active. Unselect this instead of deleting accounts.', verbose_name='active')),
                ('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='date joined')),
                ('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.Group', verbose_name='groups')),
                ('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.Permission', verbose_name='user permissions')),
            ],
            options={
                'db_table': 'auth_user',
            },
            managers=[
                ('objects', django.contrib.auth.models.UserManager()),
            ],
        ),

Under "dependencies" add:

('auth', '0012_alter_user_first_name_max_length'),

and, if present, remove

migrations.swappable_dependency(settings.AUTH_USER_MODEL),

(The exact code to use here will likely change over time with newer versions of Django. You can find the current code by creating a new app temporarily, add the User model to it and then look at the migration file ./manage.py makemigrations produces.)

  1. Create a new data migration to the user app by typing: ./manage.py makemigrations --empty user
  1. Edit the newly created migration:


Under "operations" add migrations.RunPython(change_user_type),

Add this function to the top of the file:

def change_user_type(apps, schema_editor):
    ContentType = apps.get_model('contenttypes', 'ContentType')
    ct = ContentType.objects.filter(
        app_label='auth',
        model='user'
    ).first()
    if ct:
        ct.app_label = 'user'
        ct.save()
  1. You can now migrate to the custom user model by running ./manage.py migrate. This should always work, however, it will do so in two different ways depending on whether it is run on an existing or a new instance:
  • On a new instance, the new user model will be created in migration 0001. The last migration will have no effect.
  • On an existing instance, the migration 0001 will be ignored as the system has already previously applied migration 0001 and will therefore now ignore it. Instead the last migration will have the effect of change the app of the user model.
  1. Now, you should be able to make changes to your users.User model and run makemigrations / migrate as needed. For example, as a first step, you may wish to rename the auth_user table to something in your users app's namespace. You can do so by removing db_table from your User model, so it looks like this:
class User(AbstractUser):
    pass

You'll also need to create and run a new migration to make this change in the database:

./manage.py makemigrations --name rename_user_table
./manage.py migrate


Please let me know if this works for you - especially with other databases.

[1] https://www.caktusgroup.com/blog/2019/04/26/how-switch-custom-django-user-model-mid-project/

Last edited 5 months ago by Johannes Wilm (previous) (diff)

comment:25 Changed 4 months ago by Max F.

The procedure in comment:24 worked perfectly for me on MySQL (ubuntu 16.04) and on SQLite (macOS 11.2.3). I had made a couple mistakes on my first attempt and wanted to leave some feedback for others going through the process:

  1. In the change_user_type function I mistakenly didn't change ct.app_label = 'user' to my apps name.. :(
  2. I added elidable=True to the RunPython operation so that I could squash the migrations later on

Quick side note, when renaming the auth_user table I did run into an issue with my CI when using ubuntu 16.04. It was fixed when I upgraded to 20.04 LTS but for reference this was the message: Renaming the 'auth_user' table while in a transaction is not supported on SQLite < 3.26 because it would break referential integrity. Try adding atomic = False to the Migration class.

Thanks for the explanation Johannes!

Last edited 4 months ago by Max F. (previous) (diff)
Note: See TracTickets for help on using tickets.
Back to Top