Opened 21 months ago

Closed 21 months ago

Last modified 21 months ago

#21075 closed Cleanup/optimization (fixed)

Doc improvement for using call_command with arguments

Reported by: nahumoz@… Owned by: timo
Component: Documentation Version: master
Severity: Normal Keywords:
Cc: bmispelon@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Hi Everyone,

I would like to created a backup of my models when new model elements are created, therefore I
defined in my models.py the following function:

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            management.call_command('dumpdata', 'TimePortal',
                                    indent=4, stdout=j)

    # later in my models file I register it with:
    post_save.connect(save_models_as_json, sender=Employee)

The above command is working perfectly fine when I use the admin GUI I find
inside the JSON_DATA_FILE all my relevand information from my application
TimePortal.
This function also works fine when running unit tests with:

    $ python mange test TimePortal

Things get ugly when I want the dumpdata command to include some more
arguments, e.g:

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            management.call_command('dumpdata', 'TimePortal', '--natural'
                                    indent=4, stdout=j)

or:

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            management.call_command('dumpdata', 'TimePortal', '-e', 'contenttypes',
                                    indent=4, stdout=j)

When I run my unit tests, django will crash, with either:

     $ python ../manage.py test TimePortal
    Creating test database for alias 'default'...
    ........Error: Unknown application: --natural
    E............
    ======================================================================
    ERROR: test_customer_has_employee (TimePortal.test_models.TestModels)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/ozn/software/zeitport/zeitportal/TimePortal/test_models.py", line 18, in setUp
        worksfor=self.customer)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 134, in get_or_create
        return self.get_query_set().get_or_create(**kwargs)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 452, in get_or_create
        obj.save(force_insert=True, using=self.db)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 463, in save
        self.save_base(using=using, force_insert=force_insert, force_update=force_update)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 565, in save_base
        created=(not record_exists), raw=raw, using=using)
      File "/usr/local/lib/python2.7/dist-packages/django/dispatch/dispatcher.py", line 172, in send
        response = receiver(signal=self, sender=sender, **named)
      File "/home/ozn/software/zeitport/zeitportal/TimePortal/models.py", line 61, in save_models_as_json
        'dumpdata', '--natural', 'TimePortal', indent=4,  stdout=j)
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 150, in call_command
        return klass.execute(*args, **defaults)
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 249, in execute
        sys.exit(1)
    SystemExit: 1

    ----------------------------------------------------------------------
    Ran 21 tests in 0.017s

    FAILED (errors=1)
    Destroying test database for alias 'default'...

or with

   $ python ../manage.py test TimePortal
    Creating test database for alias 'default'...
    ........Error: Unknown application: -e
    E............
    ======================================================================
      
      ... chopped for sanity ...
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 150, in call_command
        return klass.execute(*args, **defaults)
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 249, in execute
        sys.exit(1)
    SystemExit: 1

    ----------------------------------------------------------------------
    Ran 21 tests in 0.017s

    FAILED (errors=1)
    Destroying test database for alias 'default'...

Thanks for the excellent django documentation, I found out I could acchieve
the desired serialization without using management.call_command:

    from django.core import serializers
    JSONSerializer = serializers.get_serializer('json')
    json_serializer = JSONSerializer()

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            json_serializer.serialize(list(Employee.objects.all())
                                      + list(Customer.objects.all())
                                      +list(Rules.objects.all())
                                      + list(Account.objects.all()),
                                      indent=4, use_natural_keys=True)
            j.writelines(json_serializer.getvalue())

despite the above 'solution', I think this is worth the attention of the
developers, since it might give a hint for some problem with
management.call_command.

I am using django 1.5.2 and Python 2.7.5.
I also tested this with Python 2.6.8 and django 1.4.5

Thanks for reading so far. Hopefully, you can find a good explanation to
this behavior with multiple options passed to management.call_command.

Change History (14)

comment:1 Changed 21 months ago by bmispelon

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to invalid
  • Status changed from new to closed

Hi,

This looks like a duplicate of #18387, which was fixed in 1.5.

Can you double-check that you can reproduce the issue in 1.5 and provide a traceback (the one you gave is 1.4 if I'm not mistaken)?

I'll close the issue in the meantime. Please reopen it if you have some more information.

Thanks.

comment:2 Changed 21 months ago by nahumoz@…

  • Resolution invalid deleted
  • Status changed from closed to new

Hi,
Here is a full transcript of actions with Python 2.7.5 and Djang 1.5.2:

[11] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [master]  $ git stash
Saved working directory and index state WIP on master: 361cd0b fixes for formatting strings in python 2.6 vs. 2.7
HEAD is now at 361cd0b fixes for formatting strings in python 2.6 vs. 2.7
[12] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [master]  $ git checkout buggy
Switched to branch 'buggy'
[13] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ which python
/usr/bin/python
[14] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python -V
Python 2.7.5+
[15] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python -c 'import django; print(django.get_version())'
1.5.2
[16] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -e
[17] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ cat models.py
# -*- coding: UTF-8 -*-


from __future__ import absolute_import

from django.db import models

from django.contrib.auth.models import User

#
#
#


class File(models.Model):

    """
    the database model for the time portal application

    we store:
      - the uploaded file content
      - the date of the check
      - the uploader
      - the date of the processing
      - the processor
    """

    zeitFile = models.CharField(max_length=200000)
    comment = models.CharField(max_length=1000)
    checkDate = models.DateTimeField('date checked')
    uploadBy = models.CharField(max_length=100)
    processedDate = models.DateTimeField('date processed', null=True)
    processedMarked = models.CharField(max_length=10, null=True)
    processedBy = models.CharField(max_length=100, null=True)

#
#
#
#
from django.db.models.signals import post_save
from django.core import management
JSON_DATA_FILE = '/tmp/TimePortal.json'


def save_models_as_json(**kwargs):
    with open(JSON_DATA_FILE, 'w') as j:
        management.call_command('dumpdata', 'TimePortal', '-e', 'contenttypes',
                                indent=4, stdout=j)

comment:3 Changed 21 months ago by bmispelon

  • Resolution set to invalid
  • Status changed from new to closed

Hi,

This is not a bug, and the error message is quite clear:

CommandError: Unknown application: -e

You need to put the options before the appname, as indicated by manage.py help dumpdata:

Usage: manage.py dumpdata [options] [appname appname.ModelName ...]

Hope that helps.

comment:4 Changed 21 months ago by nahumoz@…

  • Resolution invalid deleted
  • Status changed from closed to new

I am sorry, but it crashes also if I change the order.

[24] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [master]  $ git checkout buggy
Switched to branch 'buggy'
[25] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ vim models.py
[26] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -e
[27] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ grep command models.py
        management.call_command('dumpdata', '-e', 'contenttypes', 'TimePortal',

The function is now:

   def save_models_as_json(**kwargs):                                              
        with open(JSON_DATA_FILE, 'w') as j:                                        
            management.call_command('dumpdata', '-e', 'contenttypes', 'TimePortal', 
                                    indent=4, stdout=j)                             

and even this is not strictly a bug the error message is not quite clear.
It did happen also with other parameters :

[33] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -n
[34] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ man cat
[35] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ grep command models.py
        management.call_command('dumpdata', '-n',  'TimePortal',
[36] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ 

comment:5 Changed 21 months ago by bmispelon

  • Cc bmispelon@… added
  • Resolution set to invalid
  • Status changed from new to closed

I had a look at the documentation for call_command [1] and figured out what was wrong.

You're not supposed to pass options like -e as positional arguments but as named arguments instead, like so:

call_command('dumpdata', 'TimePortal', exclude='contenttypes', indent=4, stdout=j)    

I agree that the error message is not completely clear (though it does tell you that the command is trying to interpret "-e" as an application name), but I think the documentation is helpful and the examples are pretty straightforward.

[1] https://docs.djangoproject.com/en/1.5/ref/django-admin/#django.core.management.call_command

comment:6 Changed 21 months ago by anonymous

  • Resolution invalid deleted
  • Status changed from closed to new

Sorry, this is still an issue ... it does not crash with sys.exit(1). But the documentation is just lacking and your example does not work:

$ grep contenttypes models.py
        management.call_command('dumpdata', 'TimePortal', exclude='django.contrib.contenttypes', indent=4, stdout=j)
[21] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown app in excludes: d

tried here with exclude='django.contrib.contenttypes' because exclude='contenttypes' gave the error:

CommandError: Unknown app in excludes: d

with exclude='natural' because natural gave the error:

CommandError: Unknown application: natural

Could you please check this?

comment:7 Changed 21 months ago by nahumoz@…

oops, posted wrong info.

about the 'natural' the line was simply:

management.call_command('dumpdata', 'TimePortal', exclude='django.contrib.contenttypes', indent=4, stdout=j)

Sorry about that.

comment:8 Changed 21 months ago by bmispelon

Looks like exclude is expecting a list of apps to exclude.

Try exclude=['contenttypes'] instead.

About the documentation, what do you find lacking? Maybe we can improve it with your feedback.

comment:9 Changed 21 months ago by nahumoz@…

phew, thanks for not giving up on me. I really appreciate it.

Well, for example that some keywords expect a list, and that that other are booleans, e.g.:

management.call_command('dumpdata', 'TimePortal', exclude=['contenttypes'], use_natural_keys=True)

To someone who does not know all the ins and outs of argparse and django.core.management. It is not that trivial that the above in code call is the transfomation of the following shell command*:

$ python manage.py dumpdata -e contenttypes -natural TimePortal 

I could probably figure out all
of that by reading the source code itself, but I guess documentation is there to save everyone that useful time.

  • btw. in the shell, argparse does not care about order, so this works too:
python ../manage.py dumpdata -n -e contenttypes TimePortal

The method call_command does care about order.
So maybe you could include all these insights under this section https://docs.djangoproject.com/en/1.5/ref/django-admin/#django.core.management.call_command.

Cheers, Oz

comment:10 Changed 21 months ago by nahumoz@…

Hi Baptiste,
You were very helpful and it encouraged me to contribute back. Thank you!
Please see this request:
https://github.com/django/django/pull/1609

comment:11 Changed 21 months ago by timo

  • Component changed from Core (Serialization) to Documentation
  • Has patch set
  • Owner changed from nobody to timo
  • Status changed from new to assigned
  • Summary changed from django crashes with sys.exit(1) when management.call_command is used with unittest to Doc improvement for using call_command with arguments
  • Triage Stage changed from Unreviewed to Accepted
  • Type changed from Bug to Cleanup/optimization
  • Version changed from 1.5 to master

comment:12 Changed 21 months ago by Tim Graham <timograham@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In fca4c4826e7b4cec84c3f8140bb929e38eea962c:

Fixed #21075 - Improved doc for calling call_command with arguments.

comment:13 Changed 21 months ago by Tim Graham <timograham@…>

In 960f5bc75901f76e9b4b356f98b22b494fb47611:

[1.6.x] Fixed #21075 - Improved doc for calling call_command with arguments.

Backport of fca4c4826e from master

comment:14 Changed 21 months ago by Tim Graham <timograham@…>

In 693ebff1a4f5eeda56279188ee473be6a1088e46:

[1.5.x] Fixed #21075 - Improved doc for calling call_command with arguments.

Backport of fca4c4826e from master

Note: See TracTickets for help on using tickets.
Back to Top