TestCase inconsistent behavior on Db integrity error

[an-mile]

Hello! I'm filing my first Django bug and hopefully it is meaningful:

When a db integrity error occurs in some test, the exception is captured by the test suite and following code in the test is executed. This seems inconsistent with any other exception that might arise from test code execution.
It might be similar to the closed ticket https://code.djangoproject.com/ticket/14223

models.py:

from django.db import models

# inspired by https://code.djangoproject.com/ticket/14223

class Reporter(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)
    email = models.EmailField(null=True)

    def __unicode__(self):
        return u"%s %s" % (self.first_name, self.last_name)

class Article(models.Model):
    reporter = models.ForeignKey(Reporter, on_delete=models.CASCADE,)
    headline = models.CharField(max_length=100)

tests.py:

from django.test import TestCase

from mod.models import Reporter, Article


class DatabasePsycopgTest(TestCase):

    def test_db_integrity_error_handling(self):
        objs = [Article(reporter_id=123, headline='Hello DB integrity')]
        # NammeErrorWouldRaiseHere
        Article.objects.bulk_create(objs)  # DB Integrity Error captured...
        print('Hello Dear Django Coders!') 
        self.assertEqual(1, 2) 

Here we get following traceback:

Traceback (most recent call last):
  File "/..../python3.9/site-packages/django/test/testcases.py", line 299, in _setup_and_call
    self._post_teardown()
  File "/..../python3.9/site-packages/django/test/testcases.py", line 1199, in _post_teardown
    self._fixture_teardown()
  File "/..../python3.9/site-packages/django/test/testcases.py", line 1461, in _fixture_teardown
    connections[db_name].check_constraints()
  File "/..../python3.9/site-packages/django/db/backends/sqlite3/base.py", line 383, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'mod_article' with primary key '1' has an invalid foreign key: mod_article.reporter_id contains a value '123' that does not have a corresponding value in mod_reporter.id.

  File "/..../codelab/dabase/mod/tests.py", line 12, in test_db_integrity_error_handling
    self.assertEqual(1, 2)
AssertionError: 1 != 2

[Edit: It will also print "Hello Dear Django Coders!"]. But when uncommenting the "NammeErrorWouldRaiseHere" , "Hello Dear Django Coders!'" is not printed, nor is the last "AssertionError" executed. This seems confusing and inconsistent behavior.

It appears on SQLite as well as with PsycoPg / PG. Furthermore, the exception can not be captured within the Code with a regular try-except. Following modification of the test will not print "Hello" :

        try:
            Article.objects.bulk_create(objs)
        except Exception:
            print('Hello')

[Mariusz Felisiak]

Thanks for this report, however I don't see any issue here. You raised NameError before trying to insert a new row to the database (bulk_create() call is where we hit the database), that's why IntegrityError is not raised.

If you're having trouble understanding how Django works, see TicketClosingReasons/UseSupportChannels for ways to get help.

[an-mile]

Here another detail: it does not merrily excessively capturing an django.db.utils.IntegrityError since the following will just fail with an IntegrityError without executing further code :

    def test_db_integrity_error_handling(self):
        rera = Reporter.objects.create(id=124)
        reru = Reporter.objects.create(id=124)
        # test execution interrupted here

[an-mile]

Replying to Mariusz Felisiak:

Thanks for this report, however I don't see any issue here. You raised NameError before trying to insert a new row to the database (bulk_create() call is where we hit the database), that's why IntegrityError is not raised.

hi,

Unfortunately you did not understand the issue: the NameError is not the issue at all, it was just added to illustrate different behavior between the two different situations.
The bulk_create is executed, raises an exception, but is then somehow captured by the test suite or so, and the the execution of the test goes on... Just weird...

[an-mile]

I'm sorry for creating confusion with "NammeErrorWouldRaiseHere", but the whole bug is just sooo weird to me that I just added some elementary NameErrors, print and the like, just to make sure that I am not wrong ...

Replying to an-mile:

Replying to Mariusz Felisiak:

Thanks for this report, however I don't see any issue here. You raised NameError before trying to insert a new row to the database (bulk_create() call is where we hit the database), that's why IntegrityError is not raised.

hi,

Unfortunately you did not understand the issue: the NameError is not the issue at all, it was just added to illustrate different behavior between the two different situations.
The bulk_create is executed, raises an exception, but is then somehow captured by the test suite or so, and the the execution of the test goes on... Just weird...

[Mariusz Felisiak]

Unfortunately you did not understand the issue: the NameError is not the issue at all, it was just added to illustrate different behavior between the two different situations.

That's how you described the issue: "But when uncommenting the "NammeErrorWouldRaiseHere" , "Hello Dear Django Coders!'" is not printed, nor is the last "AssertionError" executed. This seems confusing and inconsistent behavior. ". Which is not an issue, it's the expected behavior.

The bulk_create is executed, raises an exception, but is then somehow captured by the test suite or so, and the the execution of the test goes on... Just weird...

bulk_create() doesn't raise an exception because you use TestCase see ​docs: "Checks deferrable database constraints at the end of each test.". Use TransactionTestCase if you want to raise an exception immediately.

[an-mile]

Ok I took a further look at the issue: the posted traceback shows that we enter and execute _post_teardown() in "django/test/testcases.py", line 299, _setup_and_call

So it seems what is happening is the following :

• At least in the SQLite case, the bulk_create is executed completely, hence creating inconsistent data in the DB.
• The _tear_down runs some DB integrity check and prints a corresponding exception...

Does this make sense?

So I am not so sure anymore if it's a bug or an undocumented feature or so, but it's still confusing...

What also probably created additional confusion is that in the PG case, it did not insert data into DB creating a slightly different situation.

[an-mile]

Replying to Mariusz Felisiak:

That's how you described the issue: "But when uncommenting the "NammeErrorWouldRaiseHere" , "Hello Dear Django Coders!'" is not printed, nor is the last "AssertionError" executed. This seems confusing and inconsistent behavior. ". Which is not an issue, it's the expected behavior.

Well that was surely not what I meant to be the issue but indeed the regular behavior, meant as comparison. Again sorry for confusion.