Run Playwright tests against non-chromium browsers as well in CI

[Varun Kasyap Pentamaraju]

After selenium to playwright migration, the tests are less flaky in non chromium browsers, and it might be a good time to raise the bar.

Playwright tests already support multiple browsers (--playwright=chromium,firefox,webkit). but, CI runs chromium only.

We could discuss and agree upon how we can add other browsers to CI.

[David Smith]

Thanks for all your efforts in this space.

Given your knowledge in this area could you give a proposal of what you think that CI should look like in terms of browser support, frequency and so on.

[Varun Kasyap Pentamaraju]

The ​admin FAQ says:

The admin provides a fully-functional experience to the recent versions of
modern, web standards compliant browsers. On desktop this means Chrome, Edge,
Firefox, Opera, Safari, and others.

Chrome, Edge, and Opera share the Chromium engine, so they are covered. That leaves Firefox and Safari. Playwright's webkit is the usual CI stand for Safari.

Current CI

Workflow	Trigger	SQLite	PostgreSQL	Screenshots
playwright.yml	PR with the playwright label	Chromium	Chromium	—
schedule_tests.yml	Daily	Chromium	Chromium	—
screenshots.yml	PR with the screenshots label	—	—	Chromium

Proposed CI

Workflow	Trigger	SQLite	PostgreSQL	Screenshots
playwright.yml	PR with the playwright label	Chromium, Firefox, WebKit (If we want)	Chromium	—
schedule_tests.yml	Daily	Chromium, Firefox, WebKit	Chromium	—
screenshots.yml	PR with the screenshots label	—	—	Chromium

We could at least add Firefox and WebKit to the daily SQLite job first and watch that for a while. If it stays useful, we can add the same browsers to the labeled PR workflow.
PostgreSQL can stay on Chromium only. The core idea is to exercise tests across in browsers. Screenshots can stay on Chromium.

[Varun Kasyap Pentamaraju]

​https://github.com/django/django/pull/21859