Long names are generated differently in Django 1.1 and Django 1.2, causing incompatability

[yngve@…]

Using long class names and member names in models can create composite table and sequence names that are longer than 63 characters, particularly for ManyToMany relations in Postgresql (v8.3)

When such a model that has been created in Django 1.1 is used in a Django 1.2 such names, particularly the id_seq, but probably others, are not found by v1.2 because it constructs a name different from what Django 1.1 originally created. I have also seen indications that this will break for models created entirely in Django 1.2, as well.

The attached testcase performs 4 different tests, the first three works in both v1.1 and v1.2, the fourth only works in v1.1.

It is possible that this problem is related to what ​http://code.djangoproject.com/changeset/13328 fixes, and probably related to the change to limit names to 63 characters.

I would recommend that Django at the very least should be able to use models that have been instantiated in an older version, and the names used there. Using the newer name convention should be done when 1.2 creates the tables.

For reference here are the emails I posted to django-users (At present I have downgraded the v1.2.1 systems back to v1.1.1) :

---

By accident I have ended up with a mixed Django 1.1.1 and Django 1.2.1 environment, on different machines.

I am currently considering whether to downgrade the 1.2 installations, or upgrade the older ones, or keep the current setup, but this depends on finding a solution to an apparent incompatibility between 1.1 and 1.2, and whether or not the upgrade can cause other problems.

The current applications, including the configuration of the Postgresql 8.3, have all been developed in v1.1.1

While this have not caused problems for one of my applications (AFAICT), I've discovered that v1.2 is apparently not compatible with the ManyToMany relations used in a second application, causing an exception to be thrown when adding to the many to many field:

  File "/usr/lib/pymodules/python2.5/django/db/models/fields/related.py", line 490, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/usr/lib/pymodules/python2.5/django/db/models/fields/related.py", line 574, in _add_items
    '%s_id' % target_field_name: obj_id,
  File "/usr/lib/pymodules/python2.5/django/db/models/query.py", line 352, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/lib/pymodules/python2.5/django/db/models/base.py", line 435, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/usr/lib/pymodules/python2.5/django/db/models/base.py", line 528, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/usr/lib/pymodules/python2.5/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/usr/lib/pymodules/python2.5/django/db/models/query.py", line 1479, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/lib/pymodules/python2.5/django/db/models/sql/compiler.py", line 789, in execute_sql
    self.query.model._meta.db_table, self.query.model._meta.pk.column)
  File "/usr/lib/pymodules/python2.5/django/db/backends/postgresql/operations.py", line 57, in last_insert_id
    cursor.execute("SELECT CURRVAL('\"%s_%s_seq\"')" % (table_name, pk_name))
  File "/usr/lib/pymodules/python2.5/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
django.db.utils.DatabaseError: relation "model_bar_foo_id_s" does not exist

A condensed example of my code (not tested independently) is the following:

  class foo(models.Model):
	#declarations
 
  class bar(models.Model):
	foo_list = models.ManyToManyField(foo, null=True)
	
	def update_foo(x):
		self.foo_list.add(x)

The call to "self.foo_list.add()" is what triggers the above exception.

AFAICT the SQL code generated by sqlall by v1.1.1 is equivalent to the code generated by 1.2.1 (in v1.2.1 there is a use of ALTER in the bar-reference of the many-to-many table). There is however no mention in the sqlall output of an identifier with the name reported by the exception, though it is possible that it may be an identifier generated server-side, or that it would be generated in a resetdb situation.

I have not been able to discover any mention of such a problem in the v1.2 release notes.

While there will be no significant problem replacing this particular application's set of tables after an upgrade, since the application is currently not in production and I had just wiped the tables anyway, I am concerned that there may be issues in the currently working application that I have not yet discovered.

Is there a setting in v1.2 that I can use to tell the database handler that the tables were generated by v1.1? Or is there another reason for this problem?

For reference:

• the psycopg2 version on the 1.1 installation is 2.0.13 (windows and Linux), the one on the 1.2 installation is 2.0.14 (linux)
• the names of the classes are much longer than used here (63 characters for the id_s name).
• There are many more ManyToMany uses in the model used by this application, including the class implicated in the exception.

---

This seems to have to do with long names on ManyToMany members, when model_bar_foo_id_seq exceeds 63 characters, there is also a similar problem if the referenced class name is too long.

I have now created a small testcase.

When running the foobar_test.py script after doing syncdb from Django 1.1.1 all four tests works in Django 1.1.1, while the fourth fails in 1.2.1 as mentioned below.

Then, when deleting the tables and syncdb-ing from Django 1.2.1 the fourth test still does not work, while the tests still works in 1.1.1.

Therefore, this seems to be a bug in Django 1.2.1.

There is some complaint when doing syncdb, but they seem to be for some other database operation, not configuring the tables.

It is possible that ​http://code.djangoproject.com/changeset/13328 may have fixed part of the problem, but when testing the last_insert_id() part of that fix, it did not work with names that contain upper case characters. I have not tested the full fix.

---

[yngve@…]

Small testcase demonstrating problem

[Ramiro Morales]

Replying to yngve@opera.com:

It is possible that ​http://code.djangoproject.com/changeset/13328 may have fixed part of the problem, but when testing the last_insert_id() part of that fix, it did not work with names that contain upper case characters. I have not tested the full fix.

That has been already reported in #13821 that contains a patch that should be ready for being checked in soon.

[Russell Keith-Magee]

You sure didn't make this easy.

• The line endings in all the files from your test case are all messed up. I had to manually convert everything so that the scripts would run.
• The project doesn't synchronize out of the box. The permissions required by the models you've provided exceed length limits.
• The example is really hard to read, because it's completely bass-ackwards with regards to PEP8. Classes have capital names. Methods don't.
• You've written a custom test script instead of a unit test, so it's difficult to know what exactly is "success" or "failure".

That said, the problem no longer exists in trunk; it was fixed by the fix for #8901 in [13328], reapplied in [13363].