Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#31689 closed Cleanup/optimization (fixed)

Warn that bulk_create()'s ignore_conflicts option ignores not only duplicate keys on MySQL.

Reported by: Tobias Krönke Owned by: Tobias Krönke
Component: Documentation Version: 3.0
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

We were caught off guard that MySQL e.g. changes a not nullable date field into 0000-00-00 when trying to bulk insert with ignore_conflicts=True and None as the value for the column. You can read about that behavior here:

With IGNORE, invalid values are adjusted to the closest values and inserted;

Of course, you're encouraged to activate strict mode in MySQL (https://django-mysql.readthedocs.io/en/latest/checks.html), but this will not help in this specific case:

https://dev.mysql.com/doc/refman/8.0/en/sql-mode.html#sql-mode-strict

If strict mode is enabled, dates with zero parts are not permitted and inserts produce an error, unless IGNORE is given as well. For INSERT IGNORE and UPDATE IGNORE, dates with zero parts are inserted as '0000-00-00'

Change History (9)

comment:1 by Tobias Krönke, 4 years ago

Component: UncategorizedDocumentation
Type: UncategorizedCleanup/optimization

comment:2 by Mariusz Felisiak, 4 years ago

I'm not sure if the Django documentation is the right place for this warning. We cannot document all database caveats.

comment:3 by Tobias Krönke, 4 years ago

I was considering this, too. However the description is rather vague not really mentioning what "ignoring a failure" could mean and thereby easily implying that the row is simply not inserted. If you don't want to pollute this method's documentation, it could also be added to https://docs.djangoproject.com/en/3.0/ref/databases/#setting-sql-mode to make users aware of exceptions to strict mode.

comment:4 by Mariusz Felisiak, 4 years ago

Summary: Docs should warn about ignore_conflicts making MySQL tinker with your dataWarn that bulk_create()'s ignore_conflicts option ignores not only duplicate keys on MySQL.
Triage Stage: UnreviewedAccepted

Agreed. It is really unexpected that ignore_conflicts ignores also bunch of other errors. I don't think we should document this specific use case, I would add a short warning (to the bulk_create() docs) that MySQL ignores not only duplicate keys but also other errors, with a reference to the MySQL docs.

comment:5 by Tobias Krönke, 4 years ago

Has patch: set

Thank you for your feedback I used to craft a PR: https://github.com/django/django/pull/13051.

comment:6 by Mariusz Felisiak, 4 years ago

Owner: changed from nobody to Tobias Krönke
Status: newassigned
Triage Stage: AcceptedReady for checkin

comment:7 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

Resolution: fixed
Status: assignedclosed

In 69e0d9c:

Fixed #31689 -- Doc'd caveat about using bulk_create()'s ignore_conflicts on MariDB and MySQL.

comment:8 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 5ba2dfb:

[3.1.x] Fixed #31689 -- Doc'd caveat about using bulk_create()'s ignore_conflicts on MariDB and MySQL.

Backport of 69e0d9c553bb55dde8d7d1d479a78bfa7093f406 from master

comment:9 by Mariusz Felisiak <felisiak.mariusz@…>, 4 years ago

In 0afe8ba2:

[3.0.x] Fixed #31689 -- Doc'd caveat about using bulk_create()'s ignore_conflicts on MariDB and MySQL.

Backport of 69e0d9c553bb55dde8d7d1d479a78bfa7093f406 from master

Note: See TracTickets for help on using tickets.
Back to Top