Opened 4 months ago

Closed 6 weeks ago

Last modified 7 days ago

#36491 closed Bug (fixed)

AttributeError: '_io.TextIOWrapper' object has no attribute 'getvalue' when setUpTestData raises and tests are run with –parallel –buffer

Reported by: Rafael Carlos Soriano Marmol Owned by: Shubham Akhilesh Singh
Component: Testing framework Version: 5.2
Severity: Normal Keywords: tests, parallel, buffer
Cc: Adam Johnson Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Rafael Carlos Soriano Marmol)

Similar issue reported here: https://code.djangoproject.com/ticket/35794

If a test class raises an exception in setUpTestData(), the failure is reported correctly when tests are executed

  • in sequence, or
  • in parallel without buffering (--parallel only).

However, running the same suite with both --parallel and --buffer causes the test run to abort with an unrelated AttributeError, and the original exception is lost.

Minimal reproducer

# tests/test_issue.py
from django.test import TestCase


class AbortingTest(TestCase):
    @classmethod
    def setUpTestData(cls):
        raise RuntimeError("Boo!")

    def test_pass(self):
        pass


class AnotherAbortingTest(TestCase):
    @classmethod
    def setUpTestData(cls):
        raise RuntimeError("Boo Too!")

    def test_pass(self):
        pass

the errors are reported clearly when running tests in sequence or --parallel without --buffer

======================================================================
ERROR: setUpClass (investment.investment.tests.test_issue.AnotherAbortingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/test/testcases.py", line 1426, in setUpClass
    cls.setUpTestData()
    ^^^^^^^^^^^^^^^^^
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/Indiana/investment/investment/tests/test_issue.py", line 16, in setUpTestData
    raise RuntimeError("Boo Too!")
    ^^^^^^^^^^^^^^^^^
RuntimeError: Boo Too!

======================================================================
ERROR: setUpClass (investment.investment.tests.test_issue.AbortingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/test/testcases.py", line 1426, in setUpClass
    cls.setUpTestData()
    ^^^^^^^^^^^^^^^^^
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/Indiana/investment/investment/tests/test_issue.py", line 7, in setUpTestData
    raise RuntimeError("Boo!")
    ^^^^^^^^^^^^^^^^^
RuntimeError: Boo!

----------------------------------------------------------------------
Ran 0 tests in 4.715s

FAILED (errors=2)

❌ Fails (parallel with buffer)

python manage.py test tests.test_issue --parallel --buffer

Destroying test database for alias 'default'...
Traceback (most recent call last):
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/Indiana/manage.py", line 34, in <module>
    main()
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/Indiana/manage.py", line 29, in main
    execute_from_command_line(sys.argv)
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/core/management/commands/test.py", line 24, in run_from_argv
    super().run_from_argv(argv)
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/core/management/base.py", line 416, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/core/management/commands/test.py", line 63, in handle
    failures = test_runner.run_tests(test_labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/test/runner.py", line 1099, in run_tests
    result = self.run_suite(suite)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/test/runner.py", line 1026, in run_suite
    return runner.run(suite)
           ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/runner.py", line 240, in run
    test(result)
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/test/runner.py", line 562, in run
    self.handle_event(result, tests, event)
  File "/Users/rafaelsm@backbase.com/dell/home/rsoriano/repositorios/docker-indiana/venvs/indiana_3_12/lib/python3.12/site-packages/django/test/runner.py", line 586, in handle_event
    handler(test, *args)
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/runner.py", line 101, in addError
    super(TextTestResult, self).addError(test, err)
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/result.py", line 17, in inner
    return method(self, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/result.py", line 116, in addError
    self.errors.append((test, self._exc_info_to_string(err, test)))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.11/Frameworks/Python.framework/Versions/3.12/lib/python3.12/unittest/result.py", line 195, in _exc_info_to_string
    output = sys.stdout.getvalue()
             ^^^^^^^^^^^^^^^^^^^
AttributeError: '_io.TextIOWrapper' object has no attribute 'getvalue'
  • The traceback contains no reference to the original RuntimeError("Boo!") / RuntimeError("Boo Too!").

Expected behaviour

With --parallel --buffer the test runner should report the same clear RuntimeError stack traces shown when buffering is disabled.

Actual behaviour

The run aborts with an AttributeError deep inside unittest.result, and the real test failures are swallowed.

Additional details

  • tblib installed (latest version == 3.1.0)
  • logging => logging.StreamHandler
  • Removing either --parallel or --buffer avoids the issue.

Similar issue reported here: https://code.djangoproject.com/ticket/35794

Edit:

issue also happens with exceptions inside setUpClass (--parallel --buffer)

class AbortingTest(TestCase):
    @classmethod
    def setUpClass(cls):
        raise RuntimeError("Boo!")

    def test_pass(self):
        pass

Attachments (2)

test_issue.py​ (346 bytes ) - added by Rafael Carlos Soriano Marmol 4 months ago.
issue​ (57.7 KB ) - added by Rafael Carlos Soriano Marmol 4 months ago.

Download all attachments as: .zip

Change History (16)

by Rafael Carlos Soriano Marmol, 4 months ago

Attachment: test_issue.py​ added

by Rafael Carlos Soriano Marmol, 4 months ago

Attachment: issue​ added

comment:1 by Rafael Carlos Soriano Marmol, 4 months ago

Description: modified (diff)

comment:2 by Rafael Carlos Soriano Marmol, 4 months ago

issue also happens with exceptions inside setUpClass (--parallel --buffer)

class AbortingTest(TestCase):
    @classmethod
    def setUpClass(cls):
        raise RuntimeError("Boo!")

    def test_pass(self):
        pass

comment:3 by Simon Charette, 4 months ago

Triage Stage: Unreviewed → Accepted

Managed to reproduce by using unitest.TestCase as well

from unittest import TestCase

class AbortingTest(TestCase):
    @classmethod
    def setUpClass(cls):
        raise RuntimeError("Boo!")

    def test_pass(self):
        pass


class AbortingTest2(TestCase):
    @classmethod
    def setUpClass(cls):
        raise RuntimeError("Boo!")

    def test_pass(self):
        pas

the problem seems due to the fact that the normal unittest ​suite setup makes sure to call TestResutl._setupStdout ​method prior to calling setUpClass which allows its addError ​handler to not crash when it takes its if self.buffer ​branch in _exc_info_to_string.

Given the buffering should happened in spawned worker process and we basically only use the TestResult provided to ParallelTestSuite.run for replicating the events sent from the pool in the main process and there's no test output to capture on the main process (no tests are run in the main process) a potential solution could be to simply disable buffer on the accumulator test result.

  • django/test/runner.py 

    diff --git a/django/test/runner.py b/django/test/runner.py
index 3e5c319ade..a44180b3b6 100644
    a b def run(self, result):  
    561561        ]
    562562        test_results = pool.imap_unordered(self.run_subsuite.__func__, args)
    563563
     564        # Disable buffering on the local test result that will accumulate
     565        # remote suites results as each process will take care of its own
     566        # buffering and there's nothing to capture on the main process.
     567        result.buffer = False
     568
    564569        while True:
    565570            if result.shouldStop:
    566571                pool.terminate()
Last edited 4 months ago by Simon Charette (previous) (diff)

comment:4 by Simon Charette, 4 months ago

Cc: Adam Johnson added

Adam, adding you as you worked on adding --buffer support for parallel tests in #31370.

comment:5 by Shubham Akhilesh Singh, 2 months ago

Owner: set to Shubham Akhilesh Singh
Status: new → assigned

comment:6 by Shubham Akhilesh Singh, 2 months ago

Has patch: set

Fix PR ​https://github.com/django/django/pull/19853

comment:7 by Adam Johnson, 8 weeks ago

Patch needs improvement: set

comment:8 by Jacob Walls, 7 weeks ago

Patch needs improvement: unset
Triage Stage: Accepted → Ready for checkin

comment:9 by Jacob Walls <jacobtylerwalls@…>, 6 weeks ago

Resolution: → fixed
Status: assigned → closed

In be581ff:

Fixed #36491 -- Fixed crash in ParallelTestRunner with --buffer.

Thanks Javier Buzzi and Adam Johnson for reviews.

Co-authored-by: Simon Charette <charette.s@…>

comment:10 by Jacob Walls <jacobtylerwalls@…>, 6 weeks ago

In f8e91d0:

[6.0.x] Fixed #36491 -- Fixed crash in ParallelTestRunner with --buffer.

Thanks Javier Buzzi and Adam Johnson for reviews.

Co-authored-by: Simon Charette <charette.s@…>

Backport of be581ff473e8ade6365975db2df602f295a4cb4b from main.

comment:11 by GitHub <noreply@…>, 6 weeks ago

In 6cb641ba:

Refs #36491 -- Skipped ParallelTestSuiteTest.test_buffer_mode_reports_setupclass_failure() without tblib.

comment:12 by Mariusz Felisiak <felisiak.mariusz@…>, 6 weeks ago

In 85140019:

[6.0.x] Refs #36491 -- Skipped ParallelTestSuiteTest.test_buffer_mode_reports_setupclass_failure() without tblib.

Backport of 6cb641ba75b1e6eace9a46e3cbade70e4af2ff66 from main

comment:13 by Joe Cardali, 7 days ago

Is this going to be backported to 5.2?

comment:14 by Jacob Walls, 7 days ago

No, it doesn't meet the strict bullet points of the ​backport policy.

Note: See TracTickets for help on using tickets.
Back to Top