error flushing database after app_loading tests (DB-specific)

[Karen Tracey]

If you try to run the app_loading tests along with any doctests that flush the database after r12982 on a database that creates sequences for AutoFields (e.g. postgres), an error occurs during flush:

Installed 23 object(s) from 1 fixture(s)
Models module can be loaded from an app in an egg ... ok
Loading an app from an egg that has no models returns no models (and no error) ... ok
Models module can be loaded from an app located under an egg's top-level package ... ok
Loading an app with no models from under the top-level egg package generates no error ... ok
Loading an app from an egg that has an import error in its models module raises that error ... ok
Doctest: regressiontests.app_loading.tests.__test__.API_TESTS ... ok
Doctest: modeltests.aggregation.models.__test__.API_TESTS ... Error: Database test_postgres couldn't be flushed. Possible reasons:
  * The database isn't running or isn't configured correctly.
  * At least one of the expected database tables doesn't exist.
  * The SQL was invalid.
Hint: Look at the output of 'django-admin.py sqlflush'. That's the SQL this command wasn't able to run.
The full error: relation "app_with_models_ham_id_seq" does not exist
LINE 1: SELECT setval('"app_with_models_ham_id_seq"', 1, false);
                      ^

FAIL

======================================================================
FAIL: Doctest: modeltests.aggregation.models.__test__.API_TESTS
----------------------------------------------------------------------
Traceback (most recent call last):
..etc...

The app_loading tests manually load models modules from eggs; this seems to be having some side-effect in the app cache/models datastructures that is making a subsequent flush attempt think that some sequence table must have been created for the loaded model, but it has not (the models module is just loaded, no DB sync is done for it). For the life of me I cannot figure out how to save/restore the appropriate datastructures in the EggLoadingTest setup/teardown to avoid this side-effect. I suppose I could change the models modules in the eggs to not have any AutoFields, so that there would be no need for any sequences, but it would seem to be cleaner to properly clean up in the test. Can someone with more of a clue with the app cache code suggest how to clean up after manually loading a models module?

[Ramiro Morales]

More data points:

• This also happens with MySQL 5 for me, the error message doesn't talk about a sequence but the table name instead:

Error: Database test_main couldn't be flushed. Possible reasons:
  * The database isn't running or isn't configured correctly.
  * At least one of the expected database tables doesn't exist.
  * The SQL was invalid.
Hint: Look at the output of 'django-admin.py sqlflush'. That's the SQL this command wasn't able to run.
The full error: (1146, "Table 'test_main.app_with_models_ham' doesn't exist")

• This can be reproduced by running the tests for the app_loading and aggregation in that order.
• Or, with more granularity, running python runtests.py app_loading.EggLoadingTest.test_egg1 or python runtests.py app_loading.EggLoadingTest.test_egg3 because test_egg1 and test_egg3 are the tests showing the problem.

[Ramiro Morales]

The patch I've attached fixes the symptoms in the app_loading tests. I don't know if that's the correct way to fix the issue because I don't know if the condition is a result of the particular environment we are setting up for these tests or is something than can also appear in more real-world scenarios.

It seems particularly strange to me that the call stack that are triggered with execution of the sqlflush management command and that ends in a call to connection.introspection.django_table_names(only_existing=True) is picking up the app_with_models_ham_id_seq sequence name and/or the app_with_models_ham to flush them even when they don't exist. It's like if the ​filtering against the list of introspected table names it performs wasn't working.

[Ramiro Morales]

Replying to ramiro:

(reposting, unifying and fixing my horrible English, ignore my babbling above)

More data points:

• This also happens with MySQL 5 for me, the error message isn't about a sequence but the table name instead:

  Error: Database test_main couldn't be flushed. Possible reasons:
    * The database isn't running or isn't configured correctly.
    * At least one of the expected database tables doesn't exist.
    * The SQL was invalid.
  Hint: Look at the output of 'django-admin.py sqlflush'. That's the SQL this command wasn't able to run.
  The full error: (1146, "Table 'test_main.app_with_models_ham' doesn't exist")
  

• This can be reproduced by running the 'app_loading' and 'aggregation' tests, in that order.
• Or, with more granularity, running 'python runtests.py app_loading.EggLoadingTest.test_egg1 aggregation' or 'python runtests.py app_loading.EggLoadingTest.test_egg3 aggregation' because 'test_egg1' and 'test_egg3' are the problematic test methods.

The patch I've attached fixes the symptoms we are seeing. I'm not sure that's the correct way to fix the issue because I don't know if the condition is a result of the particular environment we are setting up for these tests or is something than can also appear in more real-world scenarios.

It seems particularly strange to me that the sequence of calls that is triggered with execution of 'sqlflush' and that ends in a call to connection.introspection.django_table_names(only_existing=True) is picking up the 'app_with_models_ham_id_seq' sequence name and/or the 'app_with_models_ham' table and trying to flush them even when they don't exist. It's like if the ​filtering against the list of introspected table names it performs wasn't working.

[Russell Keith-Magee]

(In [13011]) Fixed #13389 -- Ensured that the app_loading test clears out the app cache at the end of each test, so that it doesn't interact badly with flush and other introspected database commands. Thanks to Karen for the report, and Ramiro Morales for the debugging hints.

[Russell Keith-Magee]

(In [13012]) [1.1.X] Fixed #13389 -- Ensured that the app_loading test clears out the app cache at the end of each test, so that it doesn't interact badly with flush and other introspected database commands. Thanks to Karen for the report, and Ramiro Morales for the debugging hints.

Backport of r13011 from trunk.

[Jacob]

Milestone 1.2 deleted