Provide a more generic way to test different browsers in selenium tests

[Tim Graham]

There is some work in Julien Phalip's ​sauce branch which allows running the tests against different browsers without having to creating separate FooBar<Browser>Tests classes. It would be nice to bring this patch current and merge it (we can defer the work in progress in that branch that allows running the tests on Sauce Labs).

[Akshesh Doshi]

PR: ​https://github.com/django/django/pull/6099

[Tim Graham]

Left comments for improvement on the pull request.

[Tim Graham]

Multiple browsers aren't working on Python 3. Zero tests are run for --selenium=firefox,chrome.

[Moritz Sichert]

I may be a bit late for this, but I see a problem with this approach. Right now the selenium browser is started in setUpClass() and closed in tearDownClass(). This means that the browser has to be started once for every test class.

This new approach does a bit of Meta class and method copying hacking to provide a new method test_foo__browser for all test functions and browsers. But then a single test class has tests for different browsers. This means that the selenium browser started can't be started in setUpClass() so the logic for starting it was moved to setUp(). That however means that the browser is started and closed once for every test case. If you consider that starting the browser can take up to a few seconds (especially on the CI system), I'd say that solution is not acceptable.

I can think of a solution that works by creating a new test class for each browser in the meta class' __new__() method and then return a proxy class containing those newly created classes that knows how to run the tests on all of them.

[Tim Graham]

Yes, I agree that would be better.

[Moritz Sichert]

I ran all selenium tests with firefox and chrome on my 4-core machine and got the following numbers:

On master branch:

Ran 67 tests in 124.703s

On the PR:

Ran 68 tests in 173.816s

That's an increase of about 40% even on a powerful desktop machine.

[Simon Charette <charette.s@…>]

In 44c0ecdd:

Fixed #25364 -- Added generic way to test on all browsers supported by selenium.

Browser names should be passed as a comma separated list to the --selenium flag.

Thanks Tim Graham, Simon Charette and Moritz Sichert for review and discussion.