test database setup can truncate non-test database if two database aliases point to the same database

[Muse]

Reappear

When your database config look like following code, reader is the same as default.

"default" is the data source database.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'test',
        'USER': 'root',
        'PASSWORD': 'xxxx.',
        'HOST': "127.0.0.1",
        'PORT': '',

        "TEST": {
            "COLLATION": "utf8_general_ci",
            "CHARSET": "utf8",
        },
    },

    'reader': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'test',
        'USER': 'root',
        'PASSWORD': 'xxxx.',
        'HOST': "127.0.0.1",
        'PORT': '',

        "TEST": {
            "COLLATION": "utf8_general_ci",
            "CHARSET": "utf8",
        },
    }
}

When I run ./manage.py test, this's a chance that db test will be truncated, all data will be removed.
My project run under 1.11.2, but I tested on 2.0.1, problem remained.

Trace

I following the code, when test database created, there's a method at django/test/testcases.py was called by unittest.

    def _fixture_teardown(self):
        # Allow TRUNCATE ... CASCADE and don't emit the post_migrate signal
        # when flushing only a subset of the apps
        for db_name in self._databases_names(include_mirrors=False):
            # Flush the database
            inhibit_post_migrate = (
                self.available_apps is not None or
                (   # Inhibit the post_migrate signal when using serialized
                    # rollback to avoid trying to recreate the serialized data.
                    self.serialized_rollback and
                    hasattr(connections[db_name], '_test_serialized_contents')
                )
            )
            call_command('flush', verbosity=0, interactive=False,
                         database=db_name, reset_sequences=False,
                         allow_cascade=self.available_apps is not None,
                         inhibit_post_migrate=inhibit_post_migrate)

Calling command flush, there's my database murderer. When calling flush, they using db test not using test_test.

I kept tracing, when testing framework start to initialize environment, there's a function "setup_databases"(django/test/utils.py) called, it used to create test database.

def setup_databases(verbosity, interactive, keepdb=False, debug_sql=False, parallel=0, **kwargs):
    """
    Create the test databases.
    """
    test_databases, mirrored_aliases = get_unique_databases_and_mirrors()

    old_names = []

    for signature, (db_name, aliases) in test_databases.items():
        first_alias = None   
        for alias in aliases:
            connection = connections[alias]
            old_names.append((connection, db_name, first_alias is None))

            # Actually create the database for the first connection
            if first_alias is None:
                first_alias = alias
                connection.creation.create_test_db(
                    verbosity=verbosity,
                    autoclobber=not interactive,
                    keepdb=keepdb,
                    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),
                )
                if parallel > 1:
                    for index in range(parallel):
                        connection.creation.clone_test_db(
                            number=index + 1,
                            verbosity=verbosity,
                            keepdb=keepdb,
                        )
            # Configure all other connections as mirrors of the first one
            else:
                connections[alias].creation.set_as_test_mirror(connections[first_alias].settings_dict)

At my case, aliases looks like {"reader", "default"} or {"default", "reader"}.

When default was the first one, create_test_db made connections.get("default") point to test_test.

On the contrary, connections.get("reader") point to test_test. But , connections.get("default") still point to test.

When choosing which database to flush, if just one database. Just using default.

    @classmethod
    def _databases_names(cls, include_mirrors=True):
        # If the test case has a multi_db=True flag, act on all databases,
        # including mirrors or not. Otherwise, just on the default DB.
        if getattr(cls, 'multi_db', False):
            return [
                alias for alias in connections
                if include_mirrors or not connections[alias].settings_dict['TEST']['MIRROR']
            ]
        else:            
            return [DEFAULT_DB_ALIAS]

But default still point to test, not test_test, so, all data gone.

[Tim Graham]

I haven't reproduced this, but the report makes sense.

[Urth]

I've created a pull request which fixes this ticket by fully creating the test databases and patching the settings of all aliases.
​https://github.com/django/django/pull/13507

[Mariusz Felisiak <felisiak.mariusz@…>]

In 06e5f7ae:

Fixed #29052 -- Made test database creation preserve alias order and prefer the "default" database.

This fixes flushing test databases when two aliases point to the same
database.

Use a list() to store the test database aliases so the order remains
stable by following the order of the connections. Also, always use the
"default" database alias as the first alias to accommodate migrate.

Previously migrate could be executed on a secondary alias which
caused truncating the "default" database.

[Mariusz Felisiak <felisiak.mariusz@…>]

In b89ce413:

[3.2.x] Fixed #29052 -- Made test database creation preserve alias order and prefer the "default" database.

This fixes flushing test databases when two aliases point to the same
database.

Use a list() to store the test database aliases so the order remains
stable by following the order of the connections. Also, always use the
"default" database alias as the first alias to accommodate migrate.

Previously migrate could be executed on a secondary alias which
caused truncating the "default" database.

Backport of 06e5f7ae1639f1e275e7cc1076dc70ca3ebaa946 from master