Doc improvement for using call_command with arguments

[nahumoz@…]

Hi Everyone,

I would like to created a backup of my models when new model elements are created, therefore I
defined in my models.py the following function:

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            management.call_command('dumpdata', 'TimePortal',
                                    indent=4, stdout=j)

    # later in my models file I register it with:
    post_save.connect(save_models_as_json, sender=Employee)

The above command is working perfectly fine when I use the admin GUI I find
inside the JSON_DATA_FILE all my relevand information from my application
TimePortal.
This function also works fine when running unit tests with:

    $ python mange test TimePortal

Things get ugly when I want the dumpdata command to include some more
arguments, e.g:

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            management.call_command('dumpdata', 'TimePortal', '--natural'
                                    indent=4, stdout=j)

or:

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            management.call_command('dumpdata', 'TimePortal', '-e', 'contenttypes',
                                    indent=4, stdout=j)

When I run my unit tests, django will crash, with either:

     $ python ../manage.py test TimePortal
    Creating test database for alias 'default'...
    ........Error: Unknown application: --natural
    E............
    ======================================================================
    ERROR: test_customer_has_employee (TimePortal.test_models.TestModels)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/ozn/software/zeitport/zeitportal/TimePortal/test_models.py", line 18, in setUp
        worksfor=self.customer)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 134, in get_or_create
        return self.get_query_set().get_or_create(**kwargs)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 452, in get_or_create
        obj.save(force_insert=True, using=self.db)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 463, in save
        self.save_base(using=using, force_insert=force_insert, force_update=force_update)
      File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 565, in save_base
        created=(not record_exists), raw=raw, using=using)
      File "/usr/local/lib/python2.7/dist-packages/django/dispatch/dispatcher.py", line 172, in send
        response = receiver(signal=self, sender=sender, **named)
      File "/home/ozn/software/zeitport/zeitportal/TimePortal/models.py", line 61, in save_models_as_json
        'dumpdata', '--natural', 'TimePortal', indent=4,  stdout=j)
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 150, in call_command
        return klass.execute(*args, **defaults)
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 249, in execute
        sys.exit(1)
    SystemExit: 1

    ----------------------------------------------------------------------
    Ran 21 tests in 0.017s

    FAILED (errors=1)
    Destroying test database for alias 'default'...

or with

   $ python ../manage.py test TimePortal
    Creating test database for alias 'default'...
    ........Error: Unknown application: -e
    E............
    ======================================================================
      
      ... chopped for sanity ...
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 150, in call_command
        return klass.execute(*args, **defaults)
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 249, in execute
        sys.exit(1)
    SystemExit: 1

    ----------------------------------------------------------------------
    Ran 21 tests in 0.017s

    FAILED (errors=1)
    Destroying test database for alias 'default'...

Thanks for the excellent django documentation, I found out I could acchieve
the desired serialization without using management.call_command:

    from django.core import serializers
    JSONSerializer = serializers.get_serializer('json')
    json_serializer = JSONSerializer()

    def save_models_as_json(**kwargs):
        with open(JSON_DATA_FILE, 'w') as j:
            json_serializer.serialize(list(Employee.objects.all())
                                      + list(Customer.objects.all())
                                      +list(Rules.objects.all())
                                      + list(Account.objects.all()),
                                      indent=4, use_natural_keys=True)
            j.writelines(json_serializer.getvalue())

despite the above 'solution', I think this is worth the attention of the
developers, since it might give a hint for some problem with
management.call_command.

I am using django 1.5.2 and Python 2.7.5.
I also tested this with Python 2.6.8 and django 1.4.5

Thanks for reading so far. Hopefully, you can find a good explanation to
this behavior with multiple options passed to management.call_command.

[Baptiste Mispelon]

Hi,

This looks like a duplicate of #18387, which was fixed in 1.5.

Can you double-check that you can reproduce the issue in 1.5 and provide a traceback (the one you gave is 1.4 if I'm not mistaken)?

I'll close the issue in the meantime. Please reopen it if you have some more information.

Thanks.

[nahumoz@…]

Hi,
Here is a full transcript of actions with Python 2.7.5 and Djang 1.5.2:

[11] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [master]  $ git stash
Saved working directory and index state WIP on master: 361cd0b fixes for formatting strings in python 2.6 vs. 2.7
HEAD is now at 361cd0b fixes for formatting strings in python 2.6 vs. 2.7
[12] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [master]  $ git checkout buggy
Switched to branch 'buggy'
[13] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ which python
/usr/bin/python
[14] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python -V
Python 2.7.5+
[15] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python -c 'import django; print(django.get_version())'
1.5.2
[16] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -e
[17] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ cat models.py
# -*- coding: UTF-8 -*-


from __future__ import absolute_import

from django.db import models

from django.contrib.auth.models import User

#
#
#


class File(models.Model):

    """
    the database model for the time portal application

    we store:
      - the uploaded file content
      - the date of the check
      - the uploader
      - the date of the processing
      - the processor
    """

    zeitFile = models.CharField(max_length=200000)
    comment = models.CharField(max_length=1000)
    checkDate = models.DateTimeField('date checked')
    uploadBy = models.CharField(max_length=100)
    processedDate = models.DateTimeField('date processed', null=True)
    processedMarked = models.CharField(max_length=10, null=True)
    processedBy = models.CharField(max_length=100, null=True)

#
#
#
#
from django.db.models.signals import post_save
from django.core import management
JSON_DATA_FILE = '/tmp/TimePortal.json'


def save_models_as_json(**kwargs):
    with open(JSON_DATA_FILE, 'w') as j:
        management.call_command('dumpdata', 'TimePortal', '-e', 'contenttypes',
                                indent=4, stdout=j)

[Baptiste Mispelon]

Hi,

This is not a bug, and the error message is quite clear:

CommandError: Unknown application: -e

You need to put the options before the appname, as indicated by manage.py help dumpdata:

Usage: manage.py dumpdata [options] [appname appname.ModelName ...]

Hope that helps.

[nahumoz@…]

I am sorry, but it crashes also if I change the order.

[24] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [master]  $ git checkout buggy
Switched to branch 'buggy'
[25] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ vim models.py
[26] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -e
[27] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ grep command models.py
        management.call_command('dumpdata', '-e', 'contenttypes', 'TimePortal',

The function is now:

   def save_models_as_json(**kwargs):                                              
        with open(JSON_DATA_FILE, 'w') as j:                                        
            management.call_command('dumpdata', '-e', 'contenttypes', 'TimePortal', 
                                    indent=4, stdout=j)                             

and even this is not strictly a bug the error message is not quite clear.
It did happen also with other parameters :

[33] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown application: -n
[34] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ man cat
[35] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ grep command models.py
        management.call_command('dumpdata', '-n',  'TimePortal',
[36] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ 

[Baptiste Mispelon]

I had a look at the documentation for call_command [1] and figured out what was wrong.

You're not supposed to pass options like -e as positional arguments but as named arguments instead, like so:

call_command('dumpdata', 'TimePortal', exclude='contenttypes', indent=4, stdout=j)    

I agree that the error message is not completely clear (though it does tell you that the command is trying to interpret "-e" as an application name), but I think the documentation is helpful and the examples are pretty straightforward.

[1] ​https://docs.djangoproject.com/en/1.5/ref/django-admin/#django.core.management.call_command

[anonymous]

Sorry, this is still an issue ... it does not crash with sys.exit(1). But the documentation is just lacking and your example does not work:

$ grep contenttypes models.py
        management.call_command('dumpdata', 'TimePortal', exclude='django.contrib.contenttypes', indent=4, stdout=j)
[21] ozn@deboz:~/software/zeitport/zeitportal/TimePortal  [buggy]  $ python ../manage.py test TimePortal
Creating test database for alias 'default'...
CommandError: Unknown app in excludes: d

tried here with exclude='django.contrib.contenttypes' because exclude='contenttypes' gave the error:

CommandError: Unknown app in excludes: d

with exclude='natural' because natural gave the error:

CommandError: Unknown application: natural

Could you please check this?

[nahumoz@…]

oops, posted wrong info.

about the 'natural' the line was simply:

management.call_command('dumpdata', 'TimePortal', exclude='django.contrib.contenttypes', indent=4, stdout=j)

Sorry about that.

[Baptiste Mispelon]

Looks like exclude is expecting a list of apps to exclude.

Try exclude=['contenttypes'] instead.

About the documentation, what do you find lacking? Maybe we can improve it with your feedback.

[nahumoz@…]

phew, thanks for not giving up on me. I really appreciate it.

Well, for example that some keywords expect a list, and that that other are booleans, e.g.:

management.call_command('dumpdata', 'TimePortal', exclude=['contenttypes'], use_natural_keys=True)

To someone who does not know all the ins and outs of argparse and django.core.management. It is not that trivial that the above in code call is the transfomation of the following shell command*:

$ python manage.py dumpdata -e contenttypes -natural TimePortal 

I could probably figure out all
of that by reading the source code itself, but I guess documentation is there to save everyone that useful time.

• btw. in the shell, argparse does not care about order, so this works too:

python ../manage.py dumpdata -n -e contenttypes TimePortal

The method call_command does care about order.
So maybe you could include all these insights under this section ​https://docs.djangoproject.com/en/1.5/ref/django-admin/#django.core.management.call_command.

Cheers, Oz

[nahumoz@…]

Hi Baptiste,
You were very helpful and it encouraged me to contribute back. Thank you!
Please see this request:
​https://github.com/django/django/pull/1609

[Tim Graham <timograham@…>]

In fca4c4826e7b4cec84c3f8140bb929e38eea962c:

Fixed #21075 - Improved doc for calling call_command with arguments.

[Tim Graham <timograham@…>]

In 960f5bc75901f76e9b4b356f98b22b494fb47611:

[1.6.x] Fixed #21075 - Improved doc for calling call_command with arguments.

Backport of fca4c4826e from master

[Tim Graham <timograham@…>]

In 693ebff1a4f5eeda56279188ee473be6a1088e46:

[1.5.x] Fixed #21075 - Improved doc for calling call_command with arguments.

Backport of fca4c4826e from master