Django4 does not pick up tests according to tags.

[Thorben Luepkes]

Hey!
We use Django for an online webshop, and want to upgrade to Django 4. However, since upgrading on a test branch, we have encountered a problem, that did not appear before upgrading:

We serve two different markets, lets call them Market A and B. Since these two markets can have different functionalities, our INSTALLED_APPS gets populated like so:
Here is more information, and the error traces:

INSTALLED_APPS = [
   ...
   'payments', # THIS INCLUDES `BasePayment` Model
   ....
]

MARKET_SPECIFIC_APPS = {
    MARKET_B: [
        'market_b.apps.MarketBConfig',
        'market_b_payments.apps.MarketBPaymentsConfig'
    ],
    MARKET_A: [
        'market_a.apps.MarketAConfig',
        'market_a_payments.apps.MarketAPaymentsConfig'
    ],
}

if MARKET in MARKET_SPECIFIC_APPS:
    # If there is a market-specific app, add it to INSTALLED_APPS
    INSTALLED_APPS += MARKET_SPECIFIC_APPS[MARKET]

======================================================================
ERROR [0.004s]: market_a.test_redirects (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: market_a.test_redirects
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/e-commerce/market_a/test_redirects.py", line 5, in <module>
    from market_a.models import AdvertisementIdMapping
  File "/e-commerce/market_a/models.py", line 14, in <module>
    class MigratedMissingData(models.Model):
  File "/e-commerce/venv/lib/python3.9/site-packages/django/db/models/base.py", line 113, in __new__
    raise RuntimeError(
RuntimeError: Model class market_a.models.MigratedMissingData doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.


======================================================================
ERROR [0.000s]: market_a_payments.models (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: market_a_payments.models
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/e-commerce/market_a_payments/models/__init__.py", line 1, in <module>
    from .payment import Payment
  File "/e-commerce/market_a_payments/models/payment.py", line 12, in <module>
    class Payment(BasePayment):
  File "/e-commerce/venv/lib/python3.9/site-packages/django/db/models/base.py", line 113, in __new__
    raise RuntimeError(
RuntimeError: Model class market_a_payments.models.payment.Payments doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.


======================================================================
ERROR [0.000s]: market_a_payments.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: market_a_payments.tests
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/e-commerce/market_a_payments/tests.py", line 20, in <module>
    from market_a_payments.models import Payment
  File "/e-commerce/market_a_payments/models/__init__.py", line 1, in <module>
    from .payment import Payment
  File "/e-commerce/market_a_payments/models/payment.py", line 12, in <module>
    class Payment(BasePayment):
  File "/e-commerce/venv/lib/python3.9/site-packages/django/db/models/base.py", line 113, in __new__
    raise RuntimeError(
RuntimeError: Model class market_a_payments.models.payment.Payments doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

all these test sit in an app called market_a_payments
This app should not be discovered when running tests for market b:

MARKET=MARKET_B python3 manage.py test --tag market_b --timing

I can try to share a bit of the test, and where it fails according to the stack:

from urllib.parse import quote
from django.test import TestCase, Client
from core.mocks import Mocks
... other imports

MARKET_A = "market_a"
MARKET_B = "market_b"


def valid_market(market: str):
    return market in [MARKET_A, MARKET_B]

def override_market(market):
    if not valid_market(market):
        raise Exception(f"{market} is not a valid market.")
    return tag(market)

class TestRedirects(TestCase):

    @override_market(MARKET_A)
    def test_landing_page_redirects(self):
        client = Client()
        cases = {
       ....
        }

        for input, dst in cases.items():
            with self.subTest(input):
                response = client.get(input, secure=False, follow=True)
                self.assertRedirects(response, dst, 301, 200)

Line 12 is actually class TestRedirects(TestCase):

Django 3.2.8 worked super fine, so i assume sth in the test runner must have changed from that version to 4, as our code remained unchanged in this test.
Annotating the class with the @override_market(MARKET_A) didnt do anything either.

Test are run on circleci using circleci/python:3.9.7-node-browsers image.
The overall goal here is that this test only gets run/discovered when the specified market gets hit.
so MARKET=MARKET_B python3 manage.py test --tag market_b --timing should not discover or run this test. taking off --parallel as suggested didnt do anything either.

Minimal reproduction here:
​https://github.com/Thorbenl/django4-testrunner.git

[Tim Graham]

You need to debug the issue and explain where Django is at fault. See if you can ​bisect to find the commit where the behavior changed.

[Mariusz Felisiak]

Thanks for the report. This behavior was changed intentionally in 038940cf5525c41464a1b9e9ba3801042320b0cc. Previously tests that failed to load did not match tags. I was able to fix the attached project (django4-testrunner) by:

• removing the ​__init__.py file from django4_testrunner,
• changing imports from django4_testrunner.default.market_settings to default.market_settings.

It looks like an issue in your code.

[Thorben Luepkes]

Replying to Mariusz Felisiak:

Thanks for the report. This behavior was changed intentionally in 038940cf5525c41464a1b9e9ba3801042320b0cc. Previously tests that failed to load did not match tags. I was able to fix the attached project (django4-testrunner) by:

• removing the ​__init__.py file from django4_testrunner,
• changing imports from django4_testrunner.default.market_settings to default.market_settings.

It looks like an issue in your code.

Thanks for the quick reply.

I would really like to be able to share the whole codebase, but as you might know, thats not really possible. We serve two different markets with the same codebase, and have market specific tests: Main market runs 1000 tests, and market b just runs 27 tests that are specific to that market.
But due to the change to tags, the test-suit tries to include apps into the test that arent in INSTALLED_APPS. I have dumped out the variable during the test, and it does not have the apps where the error is thrown even included.

We run test like so:

python3.9 manage.py test --exclude-tag=functional --timing # Main Market, no functional test
MARKET=market_b python3.9 manage.py test --tag=market_b --exclude-tag=functional --timing # The other market, also no functional tests

Could it maybe have to do with:

from xmlrunner.extra.djangotestrunner import XMLTestRunner

from.market_settings.markets import MARKET_B


class CustomTestSuiteRunner(XMLTestRunner):
    """Custom test runner which excludes market-specific tests by default if no tests are specified."""
    def build_suite(self, test_labels=None, extra_tests=None, **kwargs):
        if MARKET_B not in self.tags and not test_labels:
            self.exclude_tags.add(MARKET_B)
        return super().build_suite(test_labels, extra_tests, **kwargs)

    def get_resultclass(self):
        if strtobool(os.environ.get("PROFILE_TESTS", "0")):
            return ProfilingTestResult
        else:
            return super().get_resultclass()

Printing these tags give me:

Including test tag(s): MARKET_B.
Excluding test tag(s): functional.

Still, tests from the other market are included:

main_market.models (unittest.loader._FailedTest) ... ERROR
main_market.tests (unittest.loader._FailedTest) ... ERROR
main_market_receipt.models (unittest.loader._FailedTest) ... ERROR
main_market_receipt.tests (unittest.loader._FailedTest) ... ERROR
test_iformer_redirects (market_b.test_redirects.TestRedirects) ... ok
test_redirects (market_b.test_redirects.TestRedirects) ... ok

And main_market_receipt is an app NOT included during the test run, when I dump out the INSTALLED_APPS

[Mariusz Felisiak]

Replying to Thorben Luepkes:

And main_market_receipt is an app NOT included during the test run, when I dump out the INSTALLED_APPS

Have you tried the solution proposed in my previous comment? You can try to use one of support channels if you have further questions.

[Thorben Luepkes]

Replying to Mariusz Felisiak:

Replying to Thorben Luepkes:

And main_market_receipt is an app NOT included during the test run, when I dump out the INSTALLED_APPS

Have you tried the solution proposed in my previous comment? You can try to use one of support channels if you have further questions.

Yes, I did. To no avail :)
Also, why should this behavior change from 3.2.11 to 4.0.1? :) Is there a different way apps get included now?

[Mariusz Felisiak]

Is there a different way apps get included now?

It's not about apps but about tags. In your case main_market and main_market_receipt failed to load which should not happened, even when they do not match tags. If you want to debug your issue with the help of other Django folks you should use support channels, Trac is not one of them.

[Thorben Luepkes]

Replying to Mariusz Felisiak:

Is there a different way apps get included now?

It's not about apps but about tags. In your case main_market and main_market_receipt failed to load which should not happened, even when they do not match tags. If you want to debug your issue with the help of other Django folks you should use support channels, Trac is not one of them.

I know this is closed now. And I am very sorry for the late reply. I have now managed to create a little reproducible repo. I have extracted everything I could into it and managed to reproduce this bug.

Could you please take a look at ​https://github.com/Thorbenl/django4-testrunner again?
This is exactly our market setup, and now it might be easier for you guys to assists on this. I am very sorry for the confusion again :)

[Mariusz Felisiak]

Replying to Thorben Luepkes:

Could you please take a look at ​https://github.com/Thorbenl/django4-testrunner again?

As ​documented: "Test discovery is based on the unittest module’s built-in test discovery. By default, this will discover tests in any file named “test*.py” under the current working directory." You can pass a test label to ignore a specific market, e.g.

./manage.py test market_a_app --tag=market_a --timing --keepdb --verbosity 2
./manage.py test market_b_app --tag=market_b --timing --keepdb --verbosity 2

or you can use conditional imports:

if settings.MARKET == MARKET_A:
    from .models import AdvertisementIdMapping