KeyError when upgrading to the new migrations system (related to git and .pyc)

[Henrik Heimbuerger]

There is a bit of an annoying gotcha connected to the upgrade path for the new migrations, git and the .pyc files. I would recommend to at least add this to the upgrade path docs at ​https://docs.djangoproject.com/en/dev/topics/migrations/#upgrading-from-south.

Here's what happened:

1. I followed the ​upgrade path instructions pretty much down to the letter: deleted my old South migration files, deleted the .pyc files, ran makemigrations and migrate. Everything worked fine and life was good.
2. I committed these changes to my git repository. (Of course, I have *.pyc in my .gitignore.)
3. My coworkers switched to my branch and upgraded to Django 1.7rc2.
4. They ran manage.py migrate and got this:

System check identified no issues (0 silenced).
Unhandled exception in thread started by <function wrapper at 0x03B75630>
Traceback (most recent call last):
  File "c:\Python27\lib\site-packages\django\utils\autoreload.py", line 222, in wrapper
    fn(*args, **kwargs)
  File "c:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 106, in inner_run
    self.check_migrations()
  File "c:\Python27\lib\site-packages\django\core\management\commands\runserver.py", line 158, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "c:\Python27\lib\site-packages\django\db\migrations\executor.py", line 17, in __init__
    self.loader = MigrationLoader(self.connection)
  File "c:\Python27\lib\site-packages\django\db\migrations\loader.py", line 48, in __init__
    self.build_graph()
  File "c:\Python27\lib\site-packages\django\db\migrations\loader.py", line 241, in build_graph
    self.graph.add_dependency(key, parent)
  File "c:\Python27\lib\site-packages\django\db\migrations\graph.py", line 42, in add_dependency
    raise KeyError("Dependency references nonexistent parent node %r" % (parent,))
KeyError: u"Dependency references nonexistent parent node (u'myapp', u'0001_initial')"

Obviously, the cause is that the myapp/migrations/0001_initial.pyc is still there, and while git replaced the myapp/migrations/0001_initial.py with the new migration one, the myapp/migrations/0001_initial.pyc is still the one from South. The error message is a bit weird, so it took us a few hours to figure out what's going on here.

We had the same issue on three different coworkers' machines, all running some modern Windows (7–8), all running Python 2.7.4+ (32-bit) and SourceTree as the git client.

---

I see three ways this could be tackled:

First of all, a warning about this should probably be added to the upgrade path instructions, to at least save others from spending time to debug this.

Second, the error message could probably be improved. Apparently, when trying to load the myapp/migrations/0001_initial.pyc, it detected that this wasn't a proper Django 1.7 migration file (correct) and just silently ignored it (not nice), later leading to that graph dependecy error. But why silently ignore this and not fail fast: "myapp/migrations/0001_initial.pyc is an invalid migration file!"

Third, there's probably a way for the management command to actively detect stale bytecode caches, but that sounds fairly difficult to implement.

[Henrik Heimbuerger]

I should add, in case anyone finds this ticket because you have the same issue and just want to fix it, the solution is simple:

Run del /s *.pyc on Windows or something like rm -R "*.pyc" (untested) on Linux in your Django project root. Now the migrations will work fine.

[anonymous]

The Linux command is find . -name '*.pyc' -ls -exec rm {} \;

[Collin Anderson]

a simplification: find -name '*.pyc' -delete, or on OS X: find . -name '*.pyc' -delete

[Andrew Godwin]

Python should be detecting that the bytecode cache is older than the Python file and renegerating it - if that's not working you have more serious issues here than what Django can provide, like machines with wrong timezones set (it's possible this is a Windows git issue).

There's no way Django can combat system-level issues like this - if your system tells Python that this bytecode cache is newer than this source file, then Python's going to use it, unless you've told python to turn off bytecode (which I suggest you do, given this error - you need to set the PYTHONDONTWRITEBYTECODE environment variable somehow), and this isn't a migration-specific bug, as you'll see other errors with this behaviour.

I'm not going to add something to the upgrade path docs, as this isn't an upgrade path bug; the ignoring is deliberate, to fix the real bug that was here before (stale .pyc files from South without a corresponding new-named migration); and the detection of stale bytecode caches is already done by Python.

Closing as INVALID, sorry, as it's outside the scope of Django.

[Andrew Godwin <andrew@…>]

In 51673c146e48ff6230eace39bac18fd555607bee:

Don't treat .pyc/pyo files as migrations. Refs #23237 among others.

[Andrew Godwin <andrew@…>]

In 267630ad50c69ebfe594de37a0636264aa5be7d6:

[1.7.x] Don't treat .pyc/pyo files as migrations. Refs #23237 among others.

[Henrik Heimbuerger]

Okay. But at any rate, your latest changes seem to fix the issue for me (at least in the most immediate case), so thanks!