Opened 4 years ago

Last modified 9 months ago

#14296 new Bug

'manage.py test' failing for apps that access read-only databases

Reported by: kthhrv Owned by: nobody
Component: Testing framework Version: 1.2
Severity: Normal Keywords:
Cc: tomek@…, gregchapple1@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

http://groups.google.com/group/django-users/browse_thread/thread/466ff5d2ab57c5a1/8131fd4a7d7a7cb6?lnk=gst&q=testing#8131fd4a7d7a7cb6

searching 'Testing framework' component i can't find a ticket for this issue so here it is.

I have an app that accesses two databases one of which is read-only, when i run 'manage.py test <app>' i get an insufficient privileges error while its trying to create/destroy the tables on the read-only database. This causes the test run to fail then and there.

all the models in the read-only app are marked managed=False.

we can't always have RW access to legacy databases.

is there a workaround for this issue? i'd love to start running tests on this app.

thx
Keith

Change History (12)

comment:1 Changed 4 years ago by lukeplant

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

What kind of workaround do you envisage?

If you are looking for read-write tests, the normal work-around for a read-only database is to create a separate, writeable, blank database and use that for your tests. You will need a different settings file, or a settings file that has some way of knowing to use a different DB when you are in test mode.

If you cannot create a new database, and cannot create any tables in your existing one, you would have to run on the live data. Assuming this database is being used for something, you are presumably not going to want to do any writes to the DB at all, which limits you to read-only tests on your actual live data. Is that what you are wanting? It is already possible - just use the TestCase and test runners provided by unittest rather than using the Django subclasses and manage.py, and your tests will connect to the normal database. You will miss out on some goodness from Django's TestCase, but not much, because most of it is to do with writing to databases. You can also instantiate your own Client for testing views, so you don't miss out on that.

comment:2 Changed 4 years ago by jonathan.ca@…

A simple hack that seems to work for this is to modify the action of setup_databases in django/test/simple.py

def setup_databases(self, kwargs):

from django.db import connections
old_names = []
mirrors = []
for alias in connections:

connection = connections[alias]

+ # If the alias name is contained within this setting,
+ # just skip round the loop.
+ # This causes all tests to occur against the aliased
+ # connection, and not create a test db.
+ try:
+ if alias in settings.RUN_TESTS_ON_LIVE_DB:
+ continue
+ except:
+ pass

# If the database is a test mirror, redirect it's connection
# instead of creating a test database.
if connection.settings_dictTEST_MIRROR?:

mirrors.append((alias, connection))
mirror_alias = connection.settings_dictTEST_MIRROR?
connections._connections[alias] = connections[mirror_alias]

else:

old_names.append((connection, connection.settings_dictNAME?))
connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)

return old_names, mirrors

and then place the names of the connections inside the variable in your settings.py:

RUN_TESTS_ON_LIVE_DB = ['dbname',]

This seems to force the tests to run against the live database, write operations will simply fail due to permissions and one no longer needs complex alternative db setups.

Hack..

comment:3 Changed 4 years ago by jonathan.ca@…

brilliant formatting on my above statement... sorry about that

Code change above should read:

      def setup_databases(self, **kwargs):
        from django.db import connections
        old_names = []
        mirrors = []
        for alias in connections:
            connection = connections[alias]
+           # If the alias name is contained within this setting, 
+           # just skip round the loop.
+           # This causes all tests to occur against the aliased
+           # connection, and not create a test db.
+           try:
+               if alias in settings.RUN_TESTS_ON_LIVE_DB:
+                   continue
+           except:
+               pass

            # If the database is a test mirror, redirect it's connection
            # instead of creating a test database.
            if connection.settings_dict['TEST_MIRROR']:
                mirrors.append((alias, connection))
                mirror_alias = connection.settings_dict['TEST_MIRROR']
                connections._connections[alias] = connections[mirror_alias]
            else:

                old_names.append((connection, connection.settings_dict['NAME']))
                connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)
        return old_names, mirrors

comment:4 Changed 4 years ago by russellm

  • Triage Stage changed from Unreviewed to Accepted

This sort of thing makes my teeth itch, because it's *really* bad testing practice -- tests should be repeatable, and anything with live data, by definition, isn't guaranteed repeatable.

A more interesting, but related use case is when you don't have permission to create or destroy databases, but you do have access to a database that you can use for testing purposes. Since solving this use case essentially implies solving the use case for this ticket, I'll mark this accepted.

comment:5 Changed 4 years ago by russellm

#14339 was a ticket requesting the alternate use case I described.

comment:6 Changed 4 years ago by julien

  • Severity set to Normal
  • Type set to Bug

comment:7 Changed 3 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:8 Changed 3 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:9 Changed 21 months ago by oinopion

  • Cc tomek@… added

comment:10 Changed 13 months ago by RK

What about if it were possible to specify in the database parameters that the test runner should create the test databases using a different database server config entirely? Eg:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'myproject',
        'HOST': 'maindb',
         # ... plus some other settings
    },
    'readonly': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'myproject',
        'HOST': 'readonlydb',
        'TEST_DATABASE': 'default'
        # ... plus some other settings
    }
}

This would trigger the test runner to create another test database on the "default" server, for use with anything during the test which tries to access the "readonly" database.

comment:11 Changed 12 months ago by shai

The Oracle backend supports a test parameter CREATE_DB that tells it not to create the (tablespace, in this context equivalent to database) for testing but rather use an existing one. It still tries to create tables, but I guess the managed=False setting would take care of that in this context.

I would like to make that parameter apply to all database backends.

comment:12 Changed 9 months ago by gchp

  • Cc gregchapple1@… added

As of b7aa7c4ab4372d2b7994d252c8bc87f77dd217ae you can use the --keepdb parameter to use an existing database rather than creating a new one for testing. This sounds more or less like what shai is describing above. It will run any outstanding migrations, and flush the data, but ultimately the database will be preserved at the end of the run.

Note: See TracTickets for help on using tickets.
Back to Top