Opened 16 years ago

Closed 11 years ago

Last modified 11 years ago

#6642 closed Bug (wontfix)

Fixtures (dumpdata/loaddata) do not check for illegal date types

Reported by: trbs Owned by: jcatalan
Component: Core (Serialization) Version: dev
Severity: Normal Keywords: fixtures, dumpdata, loaddata
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Working with a legacy database (in MySQL) and wanting to put it's data as a fixture for a django website i hit the following problem.

MySQL (unlike PostgreSQL) uses or at least allows completely invalid date's. Dates structured as '0000-00-00' seem to be a valid construction.
The table in this case, has a NOT NULL constraint on the date field and uses '0000-00-00' instead of null to indicate an empty of not recorded date.

Now Django (and Python) in general seem to restrict the use of invalid date's quite heavily. For example when using the datetime module in Python the following happens if one tries to use the invalid year '0'. See http://en.wikipedia.org/wiki/Year_zero for more information about the year 0.

>>> datetime.date(0,1,1)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: year is out of range

When creating the fixture with 'dumpdata' the records containing '0000-00-00' dates in the database are converted to 'null' in the fixture (which makes sense) but the model generated by inspectdb doesn't specify a null=True on the DateField because this is also NOT NULL in the database.

This creates a problem when inserting the data from the fixture again, since the database column is still NOT NULL but the fixture contains the valid date null instead of the invalid '0000-00-00' notation, this raises a:

Problem installing fixture 'testcase.json': Column 'Date' cannot be null.

In essence this is a database problem with back ends that allow invalid dates to be entered. But it raises some unexpected results in the behavior of fixtures.

It would help a lot of Django would guard against these kinds of conditions by doing some extra checking in for example 'inspectdb' and/or 'dumpdata' and 'loaddata'.

While testing this in different backends i found the following issues:

  • PostgreSQL: Checks dates strictly, the problem doesn't seem to accor here.
  • MySQL: Allow for invalid dates, problems with reloading data after dumping it. (because of the dumpdata converts invalid dates to 'null')
  • Sqlite3: Does not check dates, raises a Python exception while dumping data.
    # SQLITE 3 Backend
    ./manage.py dumpdata testcase
      File "/usr/lib/python2.4/site-packages/django/db/backends/util.py", line 49, in typecast_d
    ate
        return s and datetime.date(*map(int, s.split('-'))) or None # returns None if s is null
    ValueError: year is out of range
    

See the attached files for more information about the testcase i used.

  • testcase.txt: describes losely what i did to test invalid date input.
  • testcase*.sql: sql files for different backends to test dumpdata and loaddata on a legacy database.
  • testcase.tar.gz: complete tarball of the testcase django project.
  • django_db_backends_utils.patch: Patch for django/db/backends/utils.py to check for invalid 0 year/month/day and return None (possible raising an clear exception is better then silently dropping the mallformed date and returning None)

Attachments (6)

django_db_backends_utils.patch (677 bytes ) - added by trbs 16 years ago.
test_case.txt (1.4 KB ) - added by trbs 16 years ago.
test_case.sql (356 bytes ) - added by trbs 16 years ago.
test_case.psql.sql (384 bytes ) - added by trbs 16 years ago.
test_case.mysql.sql (383 bytes ) - added by trbs 16 years ago.
fixturestestcase.tar.gz (4.3 KB ) - added by trbs 16 years ago.

Download all attachments as: .zip

Change History (16)

by trbs, 16 years ago

by trbs, 16 years ago

Attachment: test_case.txt added

by trbs, 16 years ago

Attachment: test_case.sql added

by trbs, 16 years ago

Attachment: test_case.psql.sql added

by trbs, 16 years ago

Attachment: test_case.mysql.sql added

by trbs, 16 years ago

Attachment: fixturestestcase.tar.gz added

comment:1 by edgarsj, 16 years ago

Owner: changed from nobody to edgarsj
Status: newassigned
Triage Stage: UnreviewedDesign decision needed

comment:2 by edgarsj, 16 years ago

Owner: changed from edgarsj to nobody
Status: assignednew

comment:3 by Julien Phalip, 13 years ago

Type: Bug

comment:4 by Julien Phalip, 13 years ago

Severity: Normal

comment:5 by Carl Meyer, 13 years ago

Easy pickings: unset
Triage Stage: Design decision neededAccepted
UI/UX: unset

Dates that are all 0s should not be dumped as null - that's the bug, and should be fixed.

comment:6 by Aymeric Augustin, 11 years ago

Component: Core (Other)Core (Serialization)

comment:7 by jcatalan, 11 years ago

Owner: changed from nobody to jcatalan
Status: newassigned

comment:8 by jcatalan, 11 years ago

I've been looking at this issue and it tracked it down to MySQLdb. Apparently, MySQLdb's default conversion for date literals return None for date strings containing zeros. That's because it uses DateTime_or_None: https://github.com/jehiah/mysql-python/blob/master/MySQLdb/times.py#L44 which behaves as noted.

It might be possible to override default conversion to be able to keep the original value (including zeros) by using a custom conversion function, but I'm not 100% sure about this.

Talking to carljm he suggested we should just close this ticket as wontfix but he wanted to run it by another core dev.

comment:9 by Aymeric Augustin, 11 years ago

Resolution: wontfix
Status: assignedclosed

I was about to wontfix this ticket a few days ago, and refrained only because Carl had accepted it.

By using MySQL, you consent to having incoherent data in your database, and to being on your own to fix it.

comment:10 by Juan Catalano <jc@…>, 11 years ago

In 36b45611bcaee0ba55b40384f29e8b6546f109bb:

Added warn note to docs about MySQL issues with 0000-00-00 date strings

MySQL accepts 0000-00-00 as a valid date but MySQLdb converts those
values into None. So there will be problems for instance if trying to
transport the data using dumpdata/loaddata.

This patch refs #6642 that has been closed as wontfix since this is a
particular problem of MySQL.

Note: See TracTickets for help on using tickets.
Back to Top