Allow being able to run a specific PW test or class for multiple browsers

[Varun Kasyap Pentamaraju]

you can run the full Playwright suite or a module against several browsers at once. But, if you ask for one test class or one test, only the first browser actually runs. The other browsers are ignored.

For example:

python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1 admin_changelist.tests.PlaywrightTests
Found 10 test(s).

But 10 * 3 = 30 tests should run.

One more example:

python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1 admin_changelist.tests.PlaywrightTests.test_add_row_selection
Found 1 test(s).

But three tests should run.

This issue is not seen when running the full module:

python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1 admin_changelist.tests
Found 30 test(s).

After the fix, the same command you already use to target a class or a single test should honor every browser you pass.

[AJ Collins]

Seems like this was confirmed as an existing issue before PlayWright in ​PR #21596.

[AJ Collins]

Our debugging process (skip to summary if you want a shorter answer)

We have narrowed down the bug to test/runner.py in DiscoverRunner.load_tests_for_label method:

• When running a single test, tests is assigned to self.test_loader.loadTestsFromName(label) and returned. This is where it doesn't find additional browsers.
• When running a full test module, tests is assigned to self.test_loader.discover(start_dir=label, **kwargs) and returned. This correctly adds the additional browser tests.

Single Test (bug case)

We commented out all but one PlaywrightTests to make debugging easier. When tests = self.test_loader.loadTestsFromName(label) runs, we get this debugger output:

(Pdb) pp tests
<unittest.suite.TestSuite tests=[<admin_changelist.tests.PlaywrightTests testMethod=test_add_row_selection>]>

admin_changelist.tests.PlaywrightTests is the first browser in the list to run the tests with

Full Test Module (happy case)

tests = self.test_loader.loadTestsFromName(label) returns no tests, so load_tests_for_label continues, ultimately finding all tests properly with self.test_loader.discover

(Pdb) pp tests
...<unittest.suite.TestSuite tests=[<admin_changelist.tests.FirefoxPlaywrightTests testMethod=test_add_row_selection>]> .. .<unittest.suite.TestSuite tests=[<admin_changelist.tests.PlaywrightTests testMethod=test_add_row_selection>]

Summary

When you pass two browsers, a second test class with its own methods is generated, in our case it's called admin_changelist.tests.FirefoxPlaywrightTests. So when you specify a single test (e.g. admin_changelist.tests.PlaywrightTests.test_add_row_selection) it's only going to run that one. It will not run the generated one: admin_changelist.tests.FirefoxPlaywrightTests.test_add_row_selection...

A fun discovery is that you technically can run the other browser tests, you just need to specify them as well

python tests/runtests.py --playwright=chromium,firefox --parallel=1 admin_changelist.tests.PlaywrightTests.test_add_row_selection admin_changelist.tests.FirefoxPlaywrightTests.test_add_row_selection --verbosity 2

[Varun Kasyap Pentamaraju]

Thank you for the investigation Collins 👍

[AJ Collins]

This is a tough problem to solve technically, since the test script is *technically* doing what the user asked. At the very least, I'd like to see this documented in ​https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#running-the-playwright-tests .

[AJ Collins]

You're very welcome, I paired with Shubham Akhilesh Singh who deserves credit for the investigation. Shoutout djangocon sprints! Replying to Varun Kasyap Pentamaraju:

Thank you for the investigation Collins 👍

[Varun Kasyap Pentamaraju]

Just in case you'd like to brainstorm more around Playwright, you can also look into ​https://github.com/django/django/pull/21596#issuecomment-5166852846

[Zubair Hassan]

​PR

[Zubair Hassan]

python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1 admin_changelist.tests.PlaywrightTests
Found 30 test(s).
python tests/runtests.py --playwright=chromium,firefox,webkit --parallel=1 admin_changelist.tests.PlaywrightTests.test_add_row_selection
Found 3 test(s).

Now single test and class tests with multiple breowers is solved in the above PR.
AJ Collins Thanks for the investigation.