New find_commands(management_dir) to support .pyc and .pyo

[lgx@…]

In django/core/management/init.py:

def find_commands(management_dir):
    """
    Given a path to a management directory, returns a list of all the command
    names that are available.

    Returns an empty list if no commands are defined.
    """
    command_dir = os.path.join(management_dir, 'commands')
    try:
        return [f[:-3] for f in os.listdir(command_dir)
                if not f.startswith('_') and f.endswith('.py')]
    except OSError:
        return []

In our environment, We compile all .py to .pyc, and then remove all .py.
Then manage.py can't find our commands because find_commands only find .py.

Here is a modified version for find_command:

def find_commands(management_dir):
    """
    Given a path to a management directory, returns a list of all the command
    names that are available.

    Returns an empty list if no commands are defined.
    """
    command_dir = os.path.join(management_dir, 'commands')
    ret = {}
    try:
        filenames = os.listdir(command_dir)
        for filename in filenames:
            if filename.startswith('_'):
                continue

            if filename.endswith('.py'):
                modname = filename[:-3]
            elif filename.endswith('.pyc') or filename.endswith('.pyo'):
                modname = filename[:-4]
            else:
                modname = None

            if modname:
                ret[modname] = 1
    except OSError:
        pass

    return ret.keys()

[Alex Gaynor]

No, executing Python bytecode only files is gradually being removed from CPython, this is not a sane way to do deployments. Wontfixing.

[anonymous]

Do you have a source for "executing Python bytecode only files is gradually being removed from CPython"? There are no PEPs pertaining to removing the bytecode generation step for CPython.

[Carl Meyer]

Replying to anonymous:

Do you have a source for "executing Python bytecode only files is gradually being removed from CPython"? There are no PEPs pertaining to removing the bytecode generation step for CPython.

[PEP 3147 ​http://www.python.org/dev/peps/pep-3147/] doesn't remove bytecode generation, but it moves the pyc files into a __pycache__ subdirectory, and this is already implemented in Python 3. So adding any code to Django that explicitly look for pyc files next to the py file is adding code that we already know to be obsolete. This is just reinforcement of the point that bytecode files are an internal CPython implementation detail, and not something that other code should be looking at directly.

[Daniel Gonçalves]

It's a sort of functionality that cannot be ignored. The distribution of compiled sources is a necessary thing, unless you work *only* with open-source, there are some contracts and companies out there that requires (sometimes enforced by law) some sort of "compiled" or "executables".

My company is obliged to a law called "PAF-ECF" (a brazilian federal law which applies to all retail automation systems) and requires that we point and sign an "executable" file (that just cannot be "readable" or an editable source file). We use a complicated mix of Django for database modeling and the ERP part of the software which runs in a browser. The point-of-sales (POS) part is a GTK+ application that uses those Django models. We need to deploy only the compiled files. There is no other way.

By the way, PEP 3147 will do not ignore ".pyc" files, unless the ".py" file is there. ​http://www.python.org/dev/peps/pep-3147/#flow-chart.

It's so simple and so needed that I just can't figure out why this is marked as "won't fix".

(!) My english isn't so good (as I think it is :-). I have no intention to be rude or anything like that. Actually I love Django and all the fantastic things that you developers did. I learned a lot from you.

[Russell Keith-Magee]

I can appreciate the situation you're in here, but @carljm's point is entirely valid - PYC files aren't "compiled python" (or, at least, they shouldn't be treated like that). They're a cache of the runtime operation of a language interpreter -- an artefact that varies between language implementations and versions.

In this case, what you're faced with are legal requirements, rather than technical ones. If Brazilian law specifically uses words like "executable", then whoever drafted the law clearly didn't anticipate dynamic languages like Python. While this is unfortunate, it's unreasonable to expect us to modify the project to accommodate a non-recommended use of Python just to satisfy your interpretation of a Brazilian legal requirement.

In the short term, I'd suggest trying to work around the law. Get legal advice about *exactly* what must be signed in order to be compliant. Have you actually got legal advice that declares that in the case of Python, the .pyc file *is* the executable? I'm not a lawyer, and I have no familiarity with the PAF-ECF law, but if it says you need to sign the "executable", then is there any scope for you to sign the *Python* executable and say you've met your requirement? Could you sign the .py file, on the grounds that *it* is the executable? Could you sign an egg file that contains the source code? These are all questions that you'd need to get legal advice on to be certain, but from my past experience, the way that an engineer chooses to interpret the law doesn't always match the way that a lawyer or judge would consider compliant usage.

In the longer term, work to fix the law. Work with your representatives to get the wording altered in a way that is compatible with dynamic languages. This is the industry you work in, so you need to take an interest when governments start passing boneheaded laws that don't match industry practice.

[Daniel Gonçalves]

Thank you @russellm,
Since your response I read (and re-read) the PEP 3147 carefully and, although they will continue to support source-less distributions, we will be looking for a better way to distribute our code. You gave me a lot of good advices.

[andreydani]

I'm developing an application that can run either as a local webserver (frozen using cx_freeze) or as a regular webserver on aws.

Freezing has its advantages for non advanced users: makes the deploy simpler since they don't have to install python, django, rest-framework and all dependencies on a user. We also don't have to control multiple environments in the case user has already a python version.

Although some may consider Django isn't made for this purpose, I consider this an issue because python, in fact, supports importing from zipfiles (PEP 273) and also supports running pyc/pyo compiled files.
In fact, django find_commands tries to stick to import behaviour finding command modules that can be imported, but it fails because it only considers regular folders (and not folders inside zip) and only commands with .py extension for this purpose.

I've made a small patch that can solve both problems, although I believe this could be accomplished using some kind of setting in the case of the file extension:

def listdir(path):
    """
    Replacement for os.listdir that list dir inside zip files
    """
    if not os.path.isdir(path):
        pos = path.lower().find('.zip')
        if pos >= 0:
            pos += 4
            zip = path[:pos]
            path_within = path[pos + 1:].replace('\\', '/')
            from zipfile import ZipFile
            zip = ZipFile(zip)
            return [a[len(path_within)+1:] for a in zip.namelist() 
                  if a.startswith(path_within)]
    return os.listdir(path)

def find_commands(management_dir):
    """
    Given a path to a management directory, returns a list of all the command
    names that are available.

    Returns an empty list if no commands are defined.
    """
    command_dir = os.path.join(management_dir, 'commands')
    try:
        return [os.path.splitext(f)[0] for f in listdir(command_dir)
                if not f.startswith('_') and (f.endswith(('.py', '.pyc'))]
    except OSError:
        return []

[Nicola]

I would like to see included this feature too, I distribute pyo files and I have to leave .py for migrations and management commands

[Claude Paroz]

Support for management commands has been added in Django 1.8 (#8280). Migrations is another story (#23406).