Random test failures with parallel selenium tests

[Moritz Sichert]

When running selenium tests in parallel I often get test failures that seem to be some kind of race condition. However with --parallel 1 these errors don't occur and sometimes even when running in parallel all tests pass.

I usually run the tests with xvfb-runner, however I also tested it without xvfb and the errors still occur.

$ chromium --version
Chromium 45.0.2454.101 
$ chromedriver --version
ChromeDriver 2.16
$ firefox --version
Mozilla Firefox 41.0

Here are some of the results of running the test suite multiple times with ./runtests --selenium (running on 8 cores):

======================================================================
ERROR: test_ForeignKey (admin_widgets.tests.AdminRawIdWidgetSeleniumFirefoxTests)
----------------------------------------------------------------------
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"link text","selector":"Bogey Blues"}
Stacktrace:
    at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdriver@googlecode.com/components/driver-component.js:10647)
    at FirefoxDriver.prototype.findElement (file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdriver@googlecode.com/components/driver-component.js:10656)
    at DelayedCommand.prototype.executeInternal_/h (file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdriver@googlecode.com/components/command-processor.js:12617)
    at DelayedCommand.prototype.executeInternal_ (file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdriver@googlecode.com/components/command-processor.js:12622)
    at DelayedCommand.prototype.execute/< (file:///tmp/django_m_oympys/tmpqtuo3nbn/extensions/fxdriver@googlecode.com/components/command-processor.js:12564)

======================================================================
ERROR: test_many_to_many (admin_widgets.tests.AdminRawIdWidgetSeleniumFirefoxTests)
----------------------------------------------------------------------
selenium.common.exceptions.TimeoutException: Message: 


======================================================================
ERROR: test_prepopulated_fields (admin_views.tests.SeleniumAdminViewsChromeTests)
----------------------------------------------------------------------
selenium.common.exceptions.NoSuchElementException: Message: no such element
  (Session info: chrome=45.0.2454.101)
  (Driver info: chromedriver=2.16,platform=Linux 4.1.6-1-ARCH x86_64)


======================================================================
FAIL: test_multiple_locales (i18n.test_compilation.MultipleLocaleCompilationTests)
----------------------------------------------------------------------
AssertionError: False is not true

----------------------------------------------------------------------
Ran 10308 tests in 61.830s

======================================================================
ERROR: test_calendar_nonday_class (admin_widgets.tests.DateTimePickerSeleniumChromeTests)
----------------------------------------------------------------------
selenium.common.exceptions.NoSuchElementException: Message: no such element
  (Session info: chrome=45.0.2454.101)
  (Driver info: chromedriver=2.16,platform=Linux 4.1.6-1-ARCH x86_64)


======================================================================
FAIL: test_add_inlines (admin_inlines.tests.SeleniumChromeTests)
----------------------------------------------------------------------
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 10308 tests in 62.124s

======================================================================
ERROR: test_filter (admin_widgets.tests.HorizontalVerticalFilterSeleniumChromeTests)
----------------------------------------------------------------------
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=45.0.2454.101)
  (Driver info: chromedriver=2.16,platform=Linux 4.1.6-1-ARCH x86_64)


----------------------------------------------------------------------
Ran 10308 tests in 65.913s

[Aymeric Augustin]

I don't know if Selenium is supposed to handle parallel execution of multiple processes.

The easiest solution is probably to serialize execution of these tests with SerializeMixin.

[Moritz Sichert]

I can't reproduce the problem any more. I assume it was fixed by #26048.