Django

Code

Ticket #4457 (closed: fixed)

Opened 1 year ago

Last modified 1 year ago

Using a custom test suite, django.test.TestCase-based tests either fail silently (DEBUG = True) or fail with a generic error message (DEBUG = False)

Reported by: Chris Wagner <cw264701@ohio.edu> Assigned to: russellm
Milestone: Component: Testing framework
Version: SVN Keywords:
Cc: Triage Stage: Accepted
Has patch: 0 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

I made a custom test script since I wanted to have tests in more places than just tests.py and models.py. In the script, I set up Django's "stuff" using 

django.test.utils.setup_test_environment()
django.test.utils.create_test_db(verbosity=0, autoclobber=True)

just before auto-detecting and running all tests using the nose testing tool/framework. Under this configuration, I noticed that my django.test.TestCase?-based tests could fail, but there would be no indication of failure when running them.

I looked at Django's code, to see how it ran the same tests, and the only noticeable difference that I saw, was, Django's test runner would set the "DEBUG" setting to False before initializing the test environment. I added that to my script, and now the result is slightly better, as I at least see some indication of failure, but the error message is a generic one ("TemplateDoesNotExist?: 500.html") so I cannot easily determine the reason for failure.

Here's an example traceback: 

ERROR: test_downloading_release_as_zip (tests.TestStuff)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/projects/easyweaze-py/easyweaze/test/web-level-tests/tests.py", line 160, in test_downloading_release_as_zip
    response = self.client.get("/members/zip_release/%d" % r.id)
  File "/usr/local/lib/python2.5/site-packages/django/test/client.py", line 203, in get
    return self.request(**r)
  File "/usr/local/lib/python2.5/site-packages/django/test/client.py", line 168, in request
    response = self.handler(environ)
  File "/usr/local/lib/python2.5/site-packages/django/test/client.py", line 38, in __call__
    response = self.get_response(request)
  File "/usr/local/lib/python2.5/site-packages/django/core/handlers/base.py", line 126, in get_response
    return callback(request, **param_dict)
  File "/usr/local/lib/python2.5/site-packages/django/views/defaults.py", line 88, in server_error
    t = loader.get_template(template_name) # You need to create a 500.html template.
  File "/usr/local/lib/python2.5/site-packages/django/template/loader.py", line 79, in get_template
    source, origin = find_template_source(template_name)
  File "/usr/local/lib/python2.5/site-packages/django/template/loader.py", line 72, in find_template_source
    raise TemplateDoesNotExist, name
TemplateDoesNotExist: 500.html

Let me know if you'd like to see my test script or anything. Thanks for any input on this.

Attachments

Change History

06/02/07 00:05:55 changed by ubernostrum

  • needs_better_patch changed.
  • needs_tests changed.
  • needs_docs changed.

The problem here is that running Django with DEBUG=False requires you to have a template named 500.html to display when an internal error occurs (and, correspondingly, 404.html for URLs that don't match), or to override handler500 and/or handler404 in the root URLConf; not having it swallows the original exception with a TemplateDoesNotExist, as you've found.

This seems to trip some folks up, and I'm not sure what the best course is; documenting it clearly is one option, and we should probably do that (though the traceback does tell you what you need to do if you read through it), but I don't know if we should also be including default templates or, if we do, how we should handle that.

06/02/07 00:07:03 changed by ubernostrum

  • owner changed from adrian to jacob.
  • component changed from Unit test system to Documentation.
  • stage changed from Unreviewed to Design decision needed.

Changing this to "documentation" for now since we need that either way, and since this is not specific to any part of Django -- doesn't matter if you're unit testing or not, you can trip this if you don't have a 500.html.

06/02/07 10:47:11 changed by Chris Wagner <cw264701@ohio.edu>

A) That still doesn't explain why my tests fail silently when DEBUG == True. B) Shouldn't the test client have a special mechanism for catching errors that occur within a view? Everyone is going to have a failing test now and then, and if you have to dig beneath the extra rubbish every time -- well, that sure seems wasteful. And, in my case, I was taking a sort of test-driven development approach; I kinda expected the test to fail (although I was not sure of exactly where or how it would fail in this case).

06/02/07 16:18:22 changed by ubernostrum

Depends on whether your test is expecting to see an exception raised or is checking for an HTTP status code; regardless of DEBUG, Django does its best to catch any exception thrown during a request/response cycle. So if you're using the text client and you want to verify that something failed, check for HTTP 500 in the response. If there's some particular part of the view that you want to test for failure, using the client and doing a full request/response cycle may not be the best option; you can always pass a mock HttpRequest object and other arguments to the view directly, and look to see if it returns an HttpResponse or throws an exception.

(follow-up: ↓ 9 ) 06/07/07 12:43:45 changed by jacob

  • owner changed from jacob to adrian.
  • component changed from Documentation to Unit test system.

I actually don't think this *is* a documentation issue. Under the built-in test suite, a failing view called from a TestClient will re-raise the original exception (which is what Chris wants). I'm changing the component back to unit tests so that Russ will see it.

06/07/07 12:51:21 changed by Chris Wagner <cw264701@ohio.edu>

Coincidentally, I was writing this up just as Jacob posted. I'll go ahead and post it anyway....

According to the Django documentation (at http://www.djangoproject.com/documentation/testing/#test-client): 

Exceptions

If you point the Test Client at a view that raises an exception, that exception will be visible in the test case. You can then use a standard try...catch block, or unittest.TestCase.assertRaises() to test for exceptions.

The only exceptions that are not visible in a Test Case are Http404, PermissionDenied and SystemExit. Django catches these exceptions internally and converts them into the appropriate HTTP responses codes.

Doesn't that contradict with what I'm seeing? I assume that, that means I should see any exception raised from one of my "views" (except for, of course, Http404, PermissionDenied and SystemExit exceptions), regardless of whether I have DEBUG set or not. Furthermore, I shouldn't see a TemplateDoesNotExist exception, because, if Django is already trying to render an error template, then it must have already captured the exception that was raised within my view.

07/12/07 10:03:08 changed by russellm

  • owner changed from adrian to russellm.

07/23/07 08:45:32 changed by russellm

As long as your view is raising a normal exception (e.g., KeyError?), you _should_ be able to catch the exception in a test case as long as settings.DEBUG = False.

I've tried to replicate your problem, but I haven't had any luck. Can you provide an example script that exhibits this problem?

(in reply to: ↑ 5 ) 07/23/07 09:25:36 changed by ubernostrum

Replying to jacob:

I actually don't think this *is* a documentation issue. Under the built-in test suite, a failing view called from a TestClient will re-raise the original exception (which is what Chris wants). I'm changing the component back to unit tests so that Russ will see it.

OK. I still think there's a related documentation issue here (probably worth a separate ticket, though) from the fact that we never warn people that they'll need 500.html and 404.html; as soon as they flip DEBUG of they'll start seeing TemplateDoesNotExist in places they never expected it...

(follow-up: ↓ 11 ) 08/11/07 20:15:32 changed by Chris Wagner <cw264701@ohiou.edu>

I was going to upload an attachment -- a zip archive containing a Django project that seems to portray the problem -- but uploading attachments doesn't seem to be working (?). Let me know if you'd like me to email the archive to you, or something.

I didn't do anything special to reproduce the problem. It was actually less work than I thought it would be; I thought I would have to make a custom test-runner script, but I didn't even need to take it that far. I'm not sure what I'm doing wrong. I am running from Django's trunk and am up-to-date (at revision 5855).

(in reply to: ↑ 10 ) 08/24/07 08:49:34 changed by russellm

Replying to Chris Wagner <cw264701@ohiou.edu>:

I was going to upload an attachment -- a zip archive containing a Django project that seems to portray the problem -- but uploading attachments doesn't seem to be working (?). Let me know if you'd like me to email the archive to you, or something.

Sure. Mail a sample project to me privately at freakboy3742@gmail.com.

08/28/07 07:07:47 changed by russellm

  • stage changed from Design decision needed to Accepted.

Ok; found the problem. The issue is caused by not having a 500.html template. The template isn't actually used - the underlying exception is caught and rethrown, but the template must exist. If it doesn't exist, the request handler throws another exception, which is the error you are reporting. My test environment (and the Django test environment) both provide a 500.html template, so I didn't see the problem.

Once I found the problem, the fix was pretty trivial. I just have to do some final testing; it should be in trunk shortly.

08/28/07 08:03:23 changed by russellm

  • status changed from new to closed.
  • resolution set to fixed.

(In [6023]) Fixed #4457 -- Corrected the handling of exceptions in the test client when the 500.html template is not available. Thanks to Chris Wager <cw264701@ohiou.edu> for his help in tracking down this problem.

08/29/07 00:00:55 changed by Chris Wagner <cw264701@ohio.edu>

Seems to be working for me. Thank you. :)

Add/Change #4457 (Using a custom test suite, django.test.TestCase-based tests either fail silently (DEBUG = True) or fail with a generic error message (DEBUG = False))




Change Properties
Action