fixture loading of M2M fields fails for child ids greater than 9

[adrian_nye]

Loading data in a JSON file that includes data for both models involved in a M2M relationship,
child ids up to 9 work fine, 10 or greater cause an error on MySQL or Postgres

On Postgres:

psycopg2.IntegrityError: insert or update on table "entity_person_address" violates foreign key constraint "entity_person_address_address_id_fkey"
DETAIL: Key (address_id)=(0) is not present in table "entity_address".

On MySQL:

Cannot add or update a child row: a foreign key constraint fails - tixsa.entity_person_telephone CONSTRAIN telephone_id_refs_id_92398a FOREIGN KEY telephone_id REFERENCES entity_telephone id

To see problem, use the attached model file and attached initial_data.json. You'll see the errors above. Then search for an id of "10" and change it to "1" and the problem goes away.

[adrian_nye]

json initial data with "10" value causing error

[Karen Tracey]

I believe you are specifying the ManyToMany fields incorrectly. As there are potentially many values for these, they need to be specified as a list. The PostgreSQL error here is more helpful than MySQL as PostgreSQL indicates specifically that it is objecting to an address ID of 0 associated with a person, which gives me the impression that for this bit of your initial data file:

       "pk": 10,
 	        "model": "entity.person",
 	        "fields": {
 	                "name_prefix": "MR",
 	                "first_name": "Valary",
 	                "middle_name": "",
 	                "last_name": "Howard",
 	                "name_suffix": "III",
 	                "address": "10",
 	                "telephone": "10",
 	                "email_address": "vhoward@interbaun.com",
 	                "email_accepted": "True",
 	                "website_url": "",
 	                "updated": "2009-04-20 15:31"
 	                }
 	        },

the address value is being interpreted as 2 values: first 1, then 0. The attempt to add a reference to an address with a pk value of 0 fails since there is no address with a pk value of 0.

I believe the correct format for specifying that (and all other ManyToManys you have) would be:

	                "address": ["10"],

[adrian_nye]

If you are right, then why does changing the "10" to "1" make the error message go away? The format should still be wrong and should still cause an error since the models have not changed.

[adrian_nye]

Furthermore, if Django does indeed take a list here and fills the intermediate table automatically from the list, then this feature needs to be documented with an example.

[Karen Tracey]

Replying to adrian_nye:

If you are right, then why does changing the "10" to "1" make the error message go away?

Because iterating over "1" produces "1" whereas iterating over "10" produces "1" and "0":

>>> for x in "10":
...    print x
...
1
0
>>> for x in "1":
...    print x
...
1

Which is also why I'd guess it would be fixed by changing the value to a list, as then iterating over it would yield the (single) correct value you are looking for:

>>> for x in ["10"]:
...    print x
...
10
>>>

The format should still be wrong and should still cause an error since the models have not changed.

I'd guess any iterable will do for a multi-valued field, so the format is not wrong as in completely invalid but just wrong because it isn't producing the results you want. "10" is iterable but iterating over it results in an attempt to create two address associations for this person (one to address with pk 1 and one to address with pk 0). The only reason you are seeing a database error is because address with pk 0 does not exist. I suspect you wouldn't see a database-level error with "12", since both 1 and 2 are valid address pks. But you'd get an object with 2 associated addresses (1 and 2), not the one address (12) I'm guessing you're expecting. Single-digit strings are also iterable but don't show cause a problem because iterating over them produces the same results as iterating over a list that contains them as a single item.

I'm not a serialization expert and I'm just basing the above on the error message and a brief look at a json initial data file for the aggregation_regress tests. The "authors" field specified there is for a ManyToMany field and it is specified using lists. Further the pk values specified there are integers, not strings, which seems more correct since in the DB these will be integer values. Your strings are working I'd guess because they can be cast to the right type without error. (If you had specified them all as ints instead of strings I suspect you'd have seen an error due ints not being iterable when it came time to iterate over the sequence of values for your ManyToMany fields.)

As for documenting this -- it may be that the doc on formats could use some enhancements, but I don't know enough about common usage to say for sure. Reading through the serialization doc I get the impression these files are most commonly creating by serializing existing objects, not writing the files from scratch. I don't see where we go into much detail on the actual formats used anywhere? If the usual way people create these files is either by serializing existing objects (or at least starting with what the serializer producers as an example) I don't know that it would be that useful to go into excruciating detail on the exact formats in the doc.