Fixtures (dumpdata/loaddata) do not check for illegal date types
|Reported by:||trbs||Owned by:||jcatalan|
|Severity:||Normal||Keywords:||fixtures, dumpdata, loaddata|
|Has patch:||no||Needs documentation:||no|
|Needs tests:||no||Patch needs improvement:||no|
Working with a legacy database (in MySQL) and wanting to put it's data as a fixture for a django website i hit the following problem.
MySQL (unlike PostgreSQL) uses or at least allows completely invalid date's. Dates structured as '0000-00-00' seem to be a valid construction.
The table in this case, has a NOT NULL constraint on the date field and uses '0000-00-00' instead of null to indicate an empty of not recorded date.
Now Django (and Python) in general seem to restrict the use of invalid date's quite heavily. For example when using the datetime module in Python the following happens if one tries to use the invalid year '0'. See http://en.wikipedia.org/wiki/Year_zero for more information about the year 0.
>>> datetime.date(0,1,1) Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: year is out of range
When creating the fixture with 'dumpdata' the records containing '0000-00-00' dates in the database are converted to 'null' in the fixture (which makes sense) but the model generated by inspectdb doesn't specify a null=True on the DateField because this is also NOT NULL in the database.
This creates a problem when inserting the data from the fixture again, since the database column is still NOT NULL but the fixture contains the valid date null instead of the invalid '0000-00-00' notation, this raises a:
Problem installing fixture 'testcase.json': Column 'Date' cannot be null.
In essence this is a database problem with back ends that allow invalid dates to be entered. But it raises some unexpected results in the behavior of fixtures.
It would help a lot of Django would guard against these kinds of conditions by doing some extra checking in for example 'inspectdb' and/or 'dumpdata' and 'loaddata'.
While testing this in different backends i found the following issues:
- PostgreSQL: Checks dates strictly, the problem doesn't seem to accor here.
- MySQL: Allow for invalid dates, problems with reloading data after dumping it. (because of the dumpdata converts invalid dates to 'null')
- Sqlite3: Does not check dates, raises a Python exception while dumping data.
# SQLITE 3 Backend ./manage.py dumpdata testcase File "/usr/lib/python2.4/site-packages/django/db/backends/util.py", line 49, in typecast_d ate return s and datetime.date(*map(int, s.split('-'))) or None # returns None if s is null ValueError: year is out of range
See the attached files for more information about the testcase i used.
- testcase.txt: describes losely what i did to test invalid date input.
- testcase*.sql: sql files for different backends to test dumpdata and loaddata on a legacy database.
- testcase.tar.gz: complete tarball of the testcase django project.
- django_db_backends_utils.patch: Patch for django/db/backends/utils.py to check for invalid 0 year/month/day and return None (possible raising an clear exception is better then silently dropping the mallformed date and returning None)
Change History (16)
comment:1 Changed 7 years ago by edgarsj
- Needs documentation unset
- Needs tests unset
- Owner changed from nobody to edgarsj
- Patch needs improvement unset
- Status changed from new to assigned
- Triage Stage changed from Unreviewed to Design decision needed
comment:2 Changed 7 years ago by edgarsj
- Owner changed from edgarsj to nobody
- Status changed from assigned to new
comment:5 Changed 3 years ago by carljm
- Easy pickings unset
- Triage Stage changed from Design decision needed to Accepted
- UI/UX unset
comment:6 Changed 2 years ago by aaugustin
- Component changed from Core (Other) to Core (Serialization)
comment:7 Changed 2 years ago by jcatalan
- Owner changed from nobody to jcatalan
- Status changed from new to assigned
comment:9 Changed 2 years ago by aaugustin
- Resolution set to wontfix
- Status changed from assigned to closed