Opened 21 months ago

Last modified 16 months ago

#23406 new Bug

Migrations not found when only .pyc files are available (e.g. in a frozen environment)

Reported by: danielmenzel Owned by: nobody
Component: Migrations Version: 1.7
Severity: Normal Keywords: migrations, .pyc, frozen, cx_Freeze
Cc: andrewgodwin Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no



When only .pyc files are available, for example in a frozen environment after running cx_Freeze or similar tools, migrations no longer work: the migrate command does not find any migrations to apply, and produces the following output:

Operations to perform:
  Synchronize unmigrated apps: <some apps that really have no migrations>
  Apply all migrations: (none)
Synchronizing apps without migrations:
  Creating tables...
    Creating table <some tables>
  Installing custom SQL...
  Installing indexes...
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run ' makemigrations' to make new migrations, and then re-run ' migrate' to apply them.

Subsequent operations then fail because the database is unmigrated.

The typical content of a frozen migrations folder looks like this:



Due to issue, there was a change in django.db.migrations.loader ( Previously, migrations were found if they had a .py, .pyc, or .pyo extension. Now only migrations with .py extensions are found.

Perhaps it would be possible to add a check if Django is running in a frozen environment:

if hasattr(sys, 'frozen'):
    extensions = ('.py', '.pyc')
    extensions = ('.py', )
if name.endswith(extensions):

or make the list of allowed extensions configurable by the user.

Attachments (1)

migrations_loader.patch (1.3 KB) - added by jdetaeye 16 months ago.
Patch to support loading from Python packages in any format

Download all attachments as: .zip

Change History (19)

comment:1 Changed 21 months ago by timgraham

  • Cc andrewgodwin added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

I forget the reasoning, but I believe the decision was not to support frozen environments. We should at least document this.

comment:2 Changed 21 months ago by carljm

The reason for the change was because treating an orphan .pyc as a real migration causes too many problems with things like switching branches in git.

I think not looking for .pyc files is the right default behavior, but I wouldn't have any problem with some form of configurability to support things like cx_Freeze.

comment:3 Changed 21 months ago by andrewgodwin

Well, we'd have to introduce a setting if we wanted to make this work (a la SOUTH_INCLUDE_PYC), and I'm somewhat averse to adding more settings than necessary. We definitely can't change the default behaviour back; people were seeing all sorts of bugs when switching branches due to .pyc files.

comment:4 Changed 21 months ago by bmispelon

Can't we get away with an option passed to the command (something like --pyc)?

comment:5 Changed 21 months ago by danielmenzel

Or perhaps add the list of searched extensions as a class attribute, so that monkey-patching is made easier?

comment:6 Changed 21 months ago by andrewgodwin

If it was to work correctly, it would need to be passed to makemigrations, migrate, sqlmigrate and squashmigrations, but I suspect we could get away with just adding it to migrate and sqlmigrate? I'm still not sure about that, but seems better than a setting.

comment:7 Changed 21 months ago by shaib

I think migrate and sqlmigrate are the only required ones for the use-case (you don't generate or manipulate migrations in a frozen environment).

But I don't like the flag approach -- it essentially passes the problem from the developer to their user.

I think the correct approach is an app that overrides ("enhances", if you like) the built-in migration commands, in much the same way that South enhanced syncdb. If it is hard to write as a 3rd-party app, we should make the changes required to make it easy.

comment:8 Changed 21 months ago by andrewgodwin

Well, there's definitely a call here to have the underlying implementation (i.e. MigrationLoader) take a list of extensions to allow, and then I think we may as well go the extra leg and add a --pyc flag; I'd hate to have people have to override the entire command just to get this behaviour.

comment:9 Changed 21 months ago by thedrow

+1 for --pyc. I think it's a good idea.
Having to extend anything in the code in order to migrate on frozen environments seems like a bit too much work.

comment:10 follow-up: Changed 20 months ago by shaib

Yeah, the idea to add an app was mostly a way to do this in without introducing a new setting.

So I take that back. But the main point remains: The determination of how migrations are found is a property of the deployment, not a property of a single invocation of migration commands. A setting is much better than a flag.

On a related note, the setting should probably be more general than just "use .pyc's". What about finding migrations in zip files? I can come up with a couple of less likely candidates as well.

So yes, scratch the app overriding the commands. We need a setting to specify the path to a migration loader, and we probably need to ship, along with the standard loader, one which picks up .pyc files (and maybe one which peeks in archives).

comment:11 in reply to: ↑ 10 Changed 17 months ago by andreydani

I agree a setting for the file extension would be better.

I would like to use *.pyc on production (frozen) environments, but not when developing or testing.

I have also made a small os.listdir replacement that searches for modules inside zip files.

def listdir(path):
    if not os.path.isdir(path):
        pos = path.lower().find('.zip')
        if pos >= 0:
            pos += 4
            zip = path[:pos]
            path_within = path[pos + 1:].replace('\\', '/')
            from zipfile import ZipFile
            zip = ZipFile(zip)
            return [a[len(path_within)+1:] for a in zip.namelist() if a.startswith(path_within)]
    return os.listdir(path)

comment:12 Changed 16 months ago by jdetaeye

The Django commands had the same issue: The code scanned for files in the folder with a certain extension, rather than using the proper Python API to peek inside the contents of the package.
Please have a look at ticket #8280 and its resolution.

6(!) years after reporting, this got fixed. And history repeats itself now with the migrations...

comment:13 Changed 16 months ago by carljm

I'm not sure the fix for #8280 is actually appropriate here - I don't believe that by default orphan .pyc files should be considered as present migrations. I think the practical hassles that causes to daily development with e.g. git branches outweigh the purity of using the proper Python API for module discovery.

The distinction here is that usually the presence or absence of a module that hasn't been explicitly requested doesn't fundamentally alter the behavior of the system. With management commands, it does a bit (they'll show up in help) but not in a way that's likely to break anything. Here, having extra migrations around that shouldn't be around in this branch seriously breaks things.

If we made migrations discover .pyc by default, we would basically have to document that PYTHONDONTWRITEBYTECODE=1 is required for effective local development in Django.

I'm not opposed to an option to enable .pyc discovery, but I am opposed to making it the default.

Changed 16 months ago by jdetaeye

Patch to support loading from Python packages in any format

comment:14 follow-up: Changed 16 months ago by jdetaeye

For what it's worth, a patch is attached for this issue. It allows migrations in any package format to be picked up, be it zip, egg, file, ...

I think the practical hassles that causes to daily development with e.g. git branches outweigh the purity of using the proper Python API for module discovery."

I don't agree with this comment.
IMHO, a migration is a python package just like any other, and should be treated as such.If some development practices have an issue with that it's their problem - not django's.

comment:15 in reply to: ↑ 14 Changed 16 months ago by carljm

Replying to jdetaeye:

IMHO, a migration is a python package just like any other, and should be treated as such.

It's not a package like any other, because normal Python packages don't have an effect on the system unless explicitly imported.

If some development practices have an issue with that it's their problem - not django's.

Your patch will cause serious, difficult to debug problems for any development practice which involves adding and removing Python modules (including migrations) from source code under local development, while ignoring pyc files. (I know this is the case, because for a period of time in the 1.7 development process, orphan pyc files were loaded as migrations, and it did cause these problems.) I think you would be hard pressed to find any current common Django development process that doesn't fit that description.

If you have a practical proposal for how to address that problem (whether with documentation changes or otherwise) in your patch, I'm interested in discussing it. If you think it's "not Django's problem" and can be ignored, I'm afraid we just disagree, and I have to remain -1 on merging your patch.

comment:16 Changed 16 months ago by aaugustin

I agree with Carl.

Arguments along the lines of "it's the user's problem" are a red flag when the problem is likely to affect the vast majority of users — in this case, everyone who uses a VCS and doesn't have PYTHONDONTWRITEBYTECODE set.

comment:17 Changed 16 months ago by jdetaeye

Fair enough, I understand the arguments.

My use case is somewhat different. I'm packaging my product with py2exe as an easy-to-install application (especially for Windows users). Out of the 7 python packages it needs, only Django needs patching to behave nicely in a frozen zip-file...

comment:18 Changed 16 months ago by carljm

I understand the use case. (I would guess that none of your other six dependencies have a situation with similar constraints to Django migration files). As I said before, I'm in favor of a patch that would change your case from "Django needs patching" to "Django needs a non-default flag/setting." That's why this ticket is in Accepted state. It's just waiting for a patch that implements it as a non-default option.

I also agree with Shai that a setting is more appropriate than a flag, because this is a property of a deployment, not a particular invocation.

(There remains an open question about zipfiles, I guess. I have no problem with supporting zip-import by default, but I'm not sure if it's possible to do that without also supporting .pyc by default, which is problematic.)

Note: See TracTickets for help on using tickets.
Back to Top