Opened 16 years ago

Closed 5 years ago

#8467 closed New feature (fixed)

For ManyToMany manager, we should convert objects being added or removed to the pk type if they are not.

Reported by: Wonlay Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: Duplicate entry, add, remove, ManyToManyField
Cc: wonlay@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


For example:

class A(models.Model):
	name = models.CharField(blank=True)

class B(models.Model):
	a = models.ManyToManyField(A)

a = A(name='aaa')
b = B()

# This line works fine:

# But this one will raise 'Duplicate entry' error:

This is caused by '1' is not an integer, the duplication checking code in add() fails when checking the string type '1'.

The same problem is applied to the remove() method.

If we convert the objects to its pk type of the model, code will run correctly.

And my patch is attached.

Attachments (1)

smart_many2many_add_remove.patch (1.2 KB ) - added by Wonlay 16 years ago.

Download all attachments as: .zip

Change History (20)

by Wonlay, 16 years ago

comment:1 by James Bennett, 16 years ago

milestone: 1.0post-1.0

comment:2 by (none), 15 years ago

milestone: post-1.0

Milestone post-1.0 deleted

comment:3 by Jacob, 15 years ago

Triage Stage: UnreviewedDesign decision needed

comment:4 by Luke Plant, 13 years ago

Severity: Normal
Type: New feature

comment:5 by Aymeric Augustin, 12 years ago

UI/UX: unset

Change UI/UX from NULL to False.

comment:6 by Aymeric Augustin, 12 years ago

Easy pickings: unset

Change Easy pickings from NULL to False.

comment:7 by Aymeric Augustin, 11 years ago

Component: Core (Other)Database layer (models, ORM)

comment:8 by Anssi Kääriäinen, 11 years ago

Triage Stage: Design decision neededAccepted

I think this fix makes sense. The patch should be made aware of from_field/to_field if possible, even it is not possible to create such ManyToManys currently this feature will very likely appear some day.

comment:9 by Tim Graham, 11 years ago

Needs tests: set
Patch needs improvement: set

Patch needs improvement per @akaariai's comment as well as tests.

comment:10 by Tim Graham, 9 years ago

What's the use case for this functionality? Do we do similar type coercion elsewhere?

comment:11 by Anssi Kääriäinen, 9 years ago

I think we should fix this on the basis that using b.a.add('1') works partly, but the duplication check doesn't work. Either we should reject strings where ints are required, or handle type coercions fully.

To fix this, we should coerce the input values to right type right at the beginning of add() and remove(). The same applies likely also to reverse foreign key sets. Still, we need to convert to the primary key type of the target model, that is, the code should work also when A had a custom primary key (for example, was primary key).

Tests for m2m.add can be found from Similar tests should be added for .remove(), and for reverse foreign key .add() and .remove().

comment:12 by Tim Graham, 8 years ago

As noted in duplicate #27249, calling Field.to_python() on the input might work.

comment:13 by Baptiste Mispelon, 5 years ago

I was unable to reproduce this on a recent version (tag 3.0rc1 for example).

I'm using the following test case (same models as in the original ticket):

from django.db import IntegrityError
from django.test import TestCase

from .models import A, B

class Ticket8467TestCase(TestCase):
    def test_integrityerror(self):
        pk = A.objects.create().pk

        b = B.objects.create()
        with self.assertRaises(IntegrityError):

This test fails (meaning that an IntegrityError is not raised) on 3.0rc1 but passes for 2.2.
Using git bisect, I tracked it down to this commit: 28712d8acfffa9cdabb88cb610bae14913fa185d.

However it's not immediately clear to me why that commit would fix this ticket so I'm unsure whether I should close this ticket.

comment:14 by Simon Charette, 5 years ago

I'm pretty sure we can consider this fixed by 28712d8acfffa9cdabb88cb610bae14913fa185d.

Even if we don't perform the conversion to the pk type in the branch handling non-model instances of _get_target_ids we now ignore conflicts on bulk insertions so the collision on insertion is now ignored.

I guess we could adjust the branch as an optimization if we wanted to by calling target_field.to_python(obj) by re-purposing this ticket as an optimization.

comment:15 by Simon Charette, 5 years ago

A regression test for the above optimization could be to assertNumQueries(1) on the b.a.add(str(pk)) call when a m2m_changed signal is registered for the relationship. That is to disable the fast insertion mechanism that is enabled on all backends except for Oracle.

comment:16 by Simon Charette, 5 years ago

Actually, this is more than an optimization because it will still crash after 28712d8acfffa9cdabb88cb610bae14913fa185d on backends that don't have the supports_ignore_conflicts feature flag enabled (only Oracle).

I'll submit a PR for it.

comment:17 by Simon Charette, 5 years ago

Needs tests: unset
Patch needs improvement: unset

comment:18 by Mariusz Felisiak <felisiak.mariusz@…>, 5 years ago

In 8cc7119:

Refs #8467 -- Added test for RelatedManager.add()/remove() with an invalid type.

comment:19 by Mariusz Felisiak <felisiak.mariusz@…>, 5 years ago

Resolution: fixed
Status: newclosed

In 379bf1a2:

Fixed #8467 -- Prevented crash when adding existent m2m relation with an invalid type.

This was an issue anymore on backends that allows conflicts to be
ignored (Refs #19544) as long the provided values were coercible to the
expected type. However on the remaining backends that don't support
this feature, namely Oracle, this could still result in an

By attempting to coerce the provided values to the expected types in
Python beforehand we allow the existing value set intersection in
ManyRelatedManager._get_missing_target_ids to prevent the problematic
insertion attempts.

Thanks Baptiste Mispelon for triaging this old ticket against the
current state of the master branch.

Note: See TracTickets for help on using tickets.
Back to Top