Code

Opened 3 years ago

Closed 3 years ago

Last modified 18 months ago

#14952 closed Uncategorized (wontfix)

New find_commands(management_dir) to support .pyc and .pyo

Reported by: lgx@… Owned by: nobody
Component: Core (Other) Version: 1.2
Severity: Normal Keywords: find_commands
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

In django/core/management/init.py:

def find_commands(management_dir):
    """
    Given a path to a management directory, returns a list of all the command
    names that are available.

    Returns an empty list if no commands are defined.
    """
    command_dir = os.path.join(management_dir, 'commands')
    try:
        return [f[:-3] for f in os.listdir(command_dir)
                if not f.startswith('_') and f.endswith('.py')]
    except OSError:
        return []

In our environment, We compile all .py to .pyc, and then remove all .py.
Then manage.py can't find our commands because find_commands only find .py.

Here is a modified version for find_command:

def find_commands(management_dir):
    """
    Given a path to a management directory, returns a list of all the command
    names that are available.

    Returns an empty list if no commands are defined.
    """
    command_dir = os.path.join(management_dir, 'commands')
    ret = {}
    try:
        filenames = os.listdir(command_dir)
        for filename in filenames:
            if filename.startswith('_'):
                continue

            if filename.endswith('.py'):
                modname = filename[:-3]
            elif filename.endswith('.pyc') or filename.endswith('.pyo'):
                modname = filename[:-4]
            else:
                modname = None

            if modname:
                ret[modname] = 1
    except OSError:
        pass

    return ret.keys()

Attachments (0)

Change History (6)

comment:1 Changed 3 years ago by Alex

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to wontfix
  • Status changed from new to closed

No, executing Python bytecode only files is gradually being removed from CPython, this is not a sane way to do deployments. Wontfixing.

comment:2 follow-up: Changed 2 years ago by anonymous

  • Easy pickings unset
  • Severity set to Normal
  • Type set to Uncategorized
  • UI/UX unset

Do you have a source for "executing Python bytecode only files is gradually being removed from CPython"? There are no PEPs pertaining to removing the bytecode generation step for CPython.

comment:3 in reply to: ↑ 2 Changed 2 years ago by carljm

Replying to anonymous:

Do you have a source for "executing Python bytecode only files is gradually being removed from CPython"? There are no PEPs pertaining to removing the bytecode generation step for CPython.

PEP 3147 doesn't remove bytecode generation, but it moves the pyc files into a __pycache__ subdirectory, and this is already implemented in Python 3. So adding any code to Django that explicitly look for pyc files next to the py file is adding code that we already know to be obsolete. This is just reinforcement of the point that bytecode files are an internal CPython implementation detail, and not something that other code should be looking at directly.

Last edited 2 years ago by carljm (previous) (diff)

comment:4 Changed 18 months ago by danielgoncalves

It's a sort of functionality that cannot be ignored. The distribution of compiled sources is a necessary thing, unless you work *only* with open-source, there are some contracts and companies out there that requires (sometimes enforced by law) some sort of "compiled" or "executables".

My company is obliged to a law called "PAF-ECF" (a brazilian federal law which applies to all retail automation systems) and requires that we point and sign an "executable" file (that just cannot be "readable" or an editable source file). We use a complicated mix of Django for database modeling and the ERP part of the software which runs in a browser. The point-of-sales (POS) part is a GTK+ application that uses those Django models. We need to deploy only the compiled files. There is no other way.

By the way, PEP 3147 will do not ignore ".pyc" files, unless the ".py" file is there. http://www.python.org/dev/peps/pep-3147/#flow-chart.

It's so simple and so needed that I just can't figure out why this is marked as "won't fix".

(!) My english isn't so good (as I think it is :-). I have no intention to be rude or anything like that. Actually I love Django and all the fantastic things that you developers did. I learned a lot from you.

comment:5 Changed 18 months ago by russellm

I can appreciate the situation you're in here, but @carljm's point is entirely valid - PYC files aren't "compiled python" (or, at least, they shouldn't be treated like that). They're a cache of the runtime operation of a language interpreter -- an artefact that varies between language implementations and versions.

In this case, what you're faced with are legal requirements, rather than technical ones. If Brazilian law specifically uses words like "executable", then whoever drafted the law clearly didn't anticipate dynamic languages like Python. While this is unfortunate, it's unreasonable to expect us to modify the project to accommodate a non-recommended use of Python just to satisfy your interpretation of a Brazilian legal requirement.

In the short term, I'd suggest trying to work around the law. Get legal advice about *exactly* what must be signed in order to be compliant. Have you actually got legal advice that declares that in the case of Python, the .pyc file *is* the executable? I'm not a lawyer, and I have no familiarity with the PAF-ECF law, but if it says you need to sign the "executable", then is there any scope for you to sign the *Python* executable and say you've met your requirement? Could you sign the .py file, on the grounds that *it* is the executable? Could you sign an egg file that contains the source code? These are all questions that you'd need to get legal advice on to be certain, but from my past experience, the way that an engineer chooses to interpret the law doesn't always match the way that a lawyer or judge would consider compliant usage.

In the longer term, work to fix the law. Work with your representatives to get the wording altered in a way that is compatible with dynamic languages. This is the industry you work in, so you need to take an interest when governments start passing boneheaded laws that don't match industry practice.

comment:6 Changed 18 months ago by danielgoncalves

Thank you @russellm,
Since your response I read (and re-read) the PEP 3147 carefully and, although they will continue to support source-less distributions, we will be looking for a better way to distribute our code. You gave me a lot of good advices.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.