Nested @override_settings on class and method do not work correctly

[Paolo D'Apice]

I am using the `@override_settings~ decorator to delete some settings during tests (accordint to the ​documentation).

I reproduced the problem with the following code:

from django.test import TestCase, override_settings
from django.conf import settings

TEST_SETTINGS = {
    'MY_DATA': {
        'foo': 1,
        'bar': 2
    }
}


class Dummy:
    def foo(self):
        return settings.MY_DATA.get('foo')

    def bar(self):
        return settings.MY_DATA.get('bar')


@override_settings(**TEST_SETTINGS)
class SimpleTest(TestCase):
    def setUp(self):
        self.dummy = Dummy()

    def test_foobar(self):
        self.assertEqual(self.dummy.foo(), 1)
        self.assertEqual(self.dummy.bar(), 2)

    @override_settings()
    def test_delete_my_data(self):
        del settings.MY_DATA
        with self.assertRaises(AttributeError):
            self.dummy.foo()
        with self.assertRaises(AttributeError):
            self.dummy.bar()

    @override_settings()
    def test_delete_foo(self):
        del settings.MY_DATA['foo']
        self.assertIsNone(self.dummy.foo())
        self.assertEqual(self.dummy.bar(), 2)

    def test_foobar_again(self):
        self.assertEqual(self.dummy.foo(), 1)
        self.assertEqual(self.dummy.bar(), 2)

I expect all the test to pass, but instead I have some failures:

..FF
======================================================================
FAIL: test_foobar (api.tests.test_bug.SimpleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/paolo/testproject/api/tests/test_bug.py", line 26, in test_foobar
    self.assertEqual(self.dummy.foo(), 1)
AssertionError: None != 1

======================================================================
FAIL: test_foobar_again (api.tests.test_bug.SimpleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/paolo/testproject/api/tests/test_bug.py", line 44, in test_foobar_again
    self.assertEqual(self.dummy.foo(), 1)
AssertionError: None != 1

----------------------------------------------------------------------
Ran 4 tests in 0.045s

FAILED (failures=2)

It seems to me that the overridden settings at method level are not cleared up after the test method runs, leaving the settings in a dirty state.

[Paolo D'Apice]

Sample test case

[Stanislav Filin]

Good evening, it's problem not only for Django 1.10.

[Piotr Domański]

Is it really bad behaviour? In test_delete_foo you don't change value of MY_DATA setting but you modify item in previous value of MY_DATA setting.
Rollback will be performed if you change

        del settings.MY_DATA['foo']

into:

        settings.MY_DATA = {'bar': 2}

We don't freeze settings anywhere so presence of @override_settings doesn't matter for this test case. Should we perform deep copy of settings for each test case before run of it?
If yes than these tests are correct, otherwise they are wrong.

[Simon Charette]

I agree with Piotr's reasoning. We aren't deep copying mutable settings on override and I don't think we should be.

If you want to alter mutable settings I'd suggest you override the setting itself in the context of your alteration. In your case that would be using decorating your test_delete_foo method with override_settings(**deepcopy(TEST_SETTINGS)).

[Paolo D'Apice]

Thanks for the explanation Simon. I believe it is reasonable to update the documentation as well, especially where it says:

You can also simulate the absence of a setting by deleting it after settings have been overridden, like this:

@override_settings()
def test_something(self):
    del settings.LOGIN_URL
    ...

because that does not work as expected.

[Simon Charette]

Paolo, from my understanding top level setting deletion works just fine as documented. Try commenting out your test_delete_foo test and the currently failing one should just pass.

The issue you are experiencing here is that you are mutating a dict by deleting a key from your top level MY_DATA setting value which isn't documented to be working.