Not all models available in custom data migration

[Daniel Kimmig]

I am building an EAV driven database. I need to write a complex data migration, which shall move the values an actual model field onto an attribute/value model instance. The code cannot be disclosed, but a simplified version looks like this:

app: actions

class Action(models.Models):
    name = models.CharField(...)
    values = generic.GenericRelation(Value, content_type_field="owner_ct", object_id_field="owner_id")
    items = models.ManyToManyField(Item, ...)

app: attributes

class Attribute(models.Model):
    name = models.CharField(...)
....



class Value(models.Model)
    value = models.CharField (...)
    attribute = models.ForeignKe(Attribute, ...)
    owner_id = models.PositiveIntegerField(blank=True, null=True)
    owner_ct = models.ForeignKey(ContentType, ....)
    owner = generic.GenericForeignKey(ct_field='owner_ct', fk_field='owner_id')

app: items

class Item(MPTTModel):
    user_label = models.CharField(...)

Basically the goal is to create an Attribute for the field "user_label" of class Item and migrate all user_label values onto a Value instance that are connected to the Item through the Action model.

When I try to create a data migration for this, only some of the models are loaded in the app registry, depending on where I put the data migration. When I try to load them using "get_model" only parts of all my models within my app are loaded..

def forwards(apps, schema_editor):
    Value = apps.get_model('attributes','Value')
    Attribute = apps.get_model('attributes','Attribute')
    Action = apps.get_model('actions','Action')
    Item = apps.get_model('items','Item')
    .....

class Migration(migrations.Migration):

    dependencies = [
        (....'),
    ]

    operations = [
        migrations.RunPython(code=forwards)
    ]

I cannot run the database access code, because the instances returned by get_model are missing

My current workaround:

Create a management command that runs the code necessary to fulfill the desired side-effects of the data migration. This works, because I have full access to all of my models.

The downside of this approach is that you have to remember when to run the management command. I cannot create a schema migration right now to remove the soon to be unused "user_label" charfield, because I need to run the management command in between the migrations manually.

Am I doing something wrong? How can I create a data migration that has access to all models just like management commands do?

[Marten Kenbeek]

It works fine for me. Make sure your custom migration has all the apps that you use in your RunPython function listed in its dependencies, or the app indeed won't be loaded.

[Daniel Kimmig]

Thank you for your input!

So I should depend on the first or latest migration of all the apps that I use?

In the case as described above, would as shown below be the correct way to go?

class Migration(migrations.Migration):

    dependencies = [
        ('items','0001_initial.py'),
        ('attributes','0001_initial.py'),
        ('actions','0001_initial.py'),
    ]

    operations = [
        migrations.RunPython(code=forwards)
    ]

[Daniel Kimmig]

When trying this within my app, I got:

File ".../site-packages/django/db/migrations/graph.py", line 46, in add_dependency

"Migration %s dependencies reference nonexistent parent node %r" % (migration, parent)

KeyError: u"Migration items.0006_auto_20150212_1138 dependencies reference nonexistent parent node (u'attributes', u'0001_initial.py')"

I changed it to:

class Migration(migrations.Migration):

    dependencies = [
        ('items', '0005_probe_is_active'),
        ('attributes', '__first__'),
        ('actions', '__first__'),
    ]

    operations = [
       migrations.RunPython(code=forwards)
    ]

Then it worked.

[Daniel Kimmig]

Oh god that was a bad idea. If anyone ever reads this: Use a real migration instead of "first" if your data migration requires fields that will be added after "first". It might sound nice to not depend on a specific version, but it can cause failure.