#22487 closed Bug (fixed)
Moving from initial_data to data migrations stops test data persisting
Reported by: | Andrew Godwin | Owned by: | Andrew Godwin |
---|---|---|---|
Component: | Testing framework | Version: | 1.7-beta-1 |
Severity: | Release blocker | Keywords: | |
Cc: | jakerothenbuhler@…, merb, cmawebsite@… | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Before migrations came along, tests would happily flush the database before each test so they could restore the data that was previously in there - initial data could be easily loaded from the initial_data fixture.
Now we're encouraging data migrations to load initial data in - which are superior in almost every way - there's no easy way to replay that data back at the moment and so tests just don't see this data. Something needs to be done to allow initial data to persist across tests.
Change History (34)
comment:1 by , 11 years ago
comment:2 by , 11 years ago
Status: | new → assigned |
---|---|
Type: | Uncategorized → Bug |
comment:3 by , 11 years ago
Triage Stage: | Unreviewed → Accepted |
---|
comment:4 by , 11 years ago
Cc: | added |
---|
comment:5 by , 11 years ago
Cc: | added |
---|
comment:6 by , 11 years ago
I have done some exploratory work on this, and implemented the dump-into-memory-and-restore-after-each-test solution here: https://github.com/django/django/pull/2643
It seems to not break - I haven't tested if it fully works yet - but it does slow the runtime of TransactionTestCases down by a factor of 3 (the schema/migrations test suite goes from 6 seconds to 18 seconds under PostgreSQL).
Tests can opt out of the new behaviour by setting serialized_rollback = False
on the TestCase, which speeds things up again, so I think overall it's not _too_ bad. Would appreciate some comments, though. There's the possibility we could default that to False
(nothing in Django's main tests needs it turned on), and then just document that users who are loading data in migrations probably want it on for their TransactionTestCase
s and/or their main tests if they're on MyISAM.
comment:7 by , 10 years ago
Currently this won't work on my side.
I always get something like that:
Sqlite3 http://pastebin.com/mb89mVqf
Postgresql http://pastebin.com/WZu9JGF9
This is the application I tested: http://gccc.de/unittest_persistance.tar.gz
Just a really simple application, so that I could run it with:
PYTHONPATH=../django python manage.py test
comment:8 by , 10 years ago
..nvm this should work, but i think we should add some tests to the pull request
comment:9 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
comment:10 by , 10 years ago
Resolution: | fixed |
---|---|
Status: | closed → new |
This seems to be causing some (non-deterministic?) failures in proxy_model_inheritance.tests.ProxyModelInheritanceTests.test_table_exists
, when running the full test suite:
Traceback (most recent call last): File "/mnt/jenkinsdata/workspace/django-1.7/database/sqlite3/python/python3.4/django/db/backends/utils.py", line 65, in execute return self.cursor.execute(sql, params) File "/mnt/jenkinsdata/workspace/django-1.7/database/sqlite3/python/python3.4/django/db/backends/sqlite3/base.py", line 485, in execute return Database.Cursor.execute(self, query, params) sqlite3.OperationalError: table "migration_test_data_persistence_book" already exists
See:
I can reproduce it locally; not sure why ci.djangoproject.com doesn't appear to be affected.
comment:13 by , 10 years ago
What looks to be happening in those failures is that somehow the 0001 migration for migration_test_data_persistence
is being applied but not being marked as such, and so the call to migrate in proxy_model_inheritance
tests is trying to re-apply it. Not sure why.
comment:14 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
OK, I've pushed a fix for the test failures in f146e70cb1cfe34a9993a9a1d95faf2bb8b15ab1; re-closing optimistically.
comment:17 by , 10 years ago
Resolution: | fixed |
---|---|
Status: | closed → new |
As discussed on IRC, this change broke the tests on Oracle:
Applying migration_test_data_persistence.0001_initial... Traceback (most recent call last): ... File "/var/lib/jenkins/workspace/DjangOra-1.8/database/oracle11/python/python2.7/django/db/backends/oracle/base.py", line 894, in execute return self.cursor.execute(query, self._param_generator(params)) django.db.utils.IntegrityError: ORA-01400: cannot insert NULL into ("DJANGODJANGORA18_27"."MIGRATION_TEST_DATA_PERSISFB95"."ID")
comment:18 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
comment:19 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
comment:20 by , 10 years ago
Running commit db9cb83d2fec781f6479f19b63d50b44a35ae632, I can't say this is fixed:
======================================================================
ERROR: test_root_category (misago.forums.tests.test_forum_model.ForumManagerTests)
root_category returns forums tree root
Traceback (most recent call last):
File "/Users/rafapiton/Documents/misago/misago/forums/tests/test_forum_model.py", line 15, in test_root_category
forum = Forum.objects.root_category()
File "/Users/rafapiton/Documents/misago/misago/forums/models.py", line 19, in root_category
return self.get(special_role='root_category')
File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, kwargs)
File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/models/query.py", line 355, in get
self.model._meta.object_name)
DoesNotExist: Forum matching query does not exist.
There is DB migration in place that creates forums tree. When I run "manage.py migrate", DB is created correctly. When I put debug code in my migration to raise Exception() with .count() for apps.get_model('misago_forums', 'Forum"), it returns valid number (5 forums), however as soon as test tries to reach to this table, its empty. I've included similiar code at beginning of test, this time exception returns 0.
I am running tests on Postgres 9.3:
comment:21 by , 10 years ago
Can you confirm you have set serialized_rollback = True
on your TransactionTestCase?
comment:22 by , 10 years ago
Hello Andrew,
I've removed all test cases from my test suite, leaving only one. It's TransactionTestCase
with serialized_rollback = True
, and now I have errors on table repopulation:
====================================================================== ERROR: test_invalid_name (misago.users.tests.test_validators.ValidateUsernameLengthTests) validate_username_length disallows invalid names ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/test/testcases.py", line 182, in __call__ self._pre_setup() File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/test/testcases.py", line 754, in _pre_setup self._fixture_setup() File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/test/testcases.py", line 797, in _fixture_setup connections[db_name]._test_serialized_contents File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/backends/creation.py", line 435, in deserialize_db_from_string obj.save() File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/core/serializers/base.py", line 176, in save setattr(self.object, accessor_name, object_list) File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/models/fields/related.py", line 1177, in __set__ manager.add(*value) File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/models/fields/related.py", line 917, in add self._add_items(self.source_field_name, self.target_field_name, *objs) File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/models/fields/related.py", line 1026, in _add_items for obj_id in new_ids File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/models/query.py", line 407, in bulk_create self._batched_insert(objs_without_pk, fields, batch_size) File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/transaction.py", line 339, in __exit__ connection.commit() File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/backends/__init__.py", line 176, in commit self._commit() File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/backends/__init__.py", line 145, in _commit return self.connection.commit() File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/Users/rafapiton/venvs/misago06/lib/python2.7/site-packages/django/db/backends/__init__.py", line 145, in _commit return self.connection.commit() IntegrityError: insert or update on table "misago_users_rank_roles" violates foreign key constraint "misago_users_ran_role_id_5cb6d82bf6031b71_fk_misago_acl_role_id" DETAIL: Key (role_id)=(3) is not present in table "misago_acl_role".
Also please correct me if I am wrong, but from reading documentation I get picture that if my app contains data migrations, and this data is important for my tests, I should make those tests inherit from TestCase
and use database with transactions support (such as Postgres)?
Thanks for your time,
Ralph
comment:23 by , 10 years ago
Yes, you should not be using TransactionTestCase unless you're very sure you want to use it (i.e. you're testing transaction handling).
As for your traceback, it looks like you have a custom user with some extra ForeignKeys that aren't being populated right. Does normal use of loaddata/dumpdata work with your app and this model?
comment:24 by , 10 years ago
Andrew,
Indeed I have custom User model with relations to other models.
I've did dumpdata
to file, then ran flush
and finally ran loaddata
for file I've created ealier, and Django raised no errors.
To sum this up, if my test case is TransactionTestCase
with serialized_rollback = True
, test case fails on IntegrityError
. This test case is dependant on data created in data migration, so when I make it inherit from TestCase
instead, it fails because database tables are emptied before tests are ran. I am running my tests against PostgreSQL 9.3 database and I have no idea why this happens, because I was under impression that flushes should happen only on DB's that can't use transactions to reverse db changes made by tests?
Again, thanks for your time and energy!
comment:25 by , 10 years ago
It happens on DBs without transactions _and_ with TransactionTestCase
, because the latter is not enclosed in a transaction and so we can't roll it back, unfortunately. This is why you should avoid TransactionTestCase
unless you really need it :)
Is it possible that you could reduce the problem to a couple of models and a migration so I can replicate it? Failing that, if you're comfortable, a larger slice/the whole project would work (I'm happy to directly negotiate access so it's not in public if the project isn't open source).
comment:26 by , 10 years ago
Its open source, and branch that I am trying to make work with Dj 1.7 is available here:
https://github.com/rafalp/Misago/tree/future
I may have messed up the test runner when I've started experimenting just-in-case its entirely my idiocy that tests that passed suddenly started to fail badly, so if setup.py test
blurps, give line 49 in runtests.py a look and adapt it accordingly.
comment:27 by , 10 years ago
Andrew,
I have created tiny project just to see if its not me doing something wrong. It contains one app. This app contains three things: model, migrations, and tests. First migration creates table in database and second migration populates it with some data. At end of migration there is count() to make sure bulk_create succeeded. Then tests in test suite see if Model.objects.count() == 6.
There are two tests cases, two tests in each. My intention was to see if data disappears after individual test, or tests case. Migration succeds meaning bulk_create did its job, but all tests fail on PostgreSQL 9.3:
https://github.com/rafalp/django-data-migrations-bug
So its either me being real idiot with using data migrations in tests, or bug 22487 is not fixed.
comment:28 by , 10 years ago
Resolution: | fixed |
---|---|
Status: | closed → new |
comment:29 by , 10 years ago
Using django-data-migrations-bug above on postgres 9.1 (ubuntu 12.04) and latest 1.7.x, I can confirm that the migration does fill the database with initial data, and that the data is gone by the time the TestCase
s are run.
comment:30 by , 10 years ago
Cc: | added |
---|
comment:31 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
comment:34 by , 10 years ago
I can confirm fix. All tests in demo project pass, my project test suite dropped form 76 fails to 6.
Great work, thanks a lot!
This is being discussed at https://groups.google.com/forum/#!topic/django-developers/wR6hTy9Nu2k
My initial proposal, and what I'll look at first, is dumping the data in the database into a temporary (possibly in-memory) fixture at the start of the test run, and then having flush restore from that.