Code

Opened 8 years ago

Closed 7 years ago

#2970 closed defect (fixed)

[patch] HttpResponse should treat headers case-insensitively

Reported by: adrian Owned by: PhiR
Component: HTTP handling Version:
Severity: normal Keywords: sprintsept14
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

(Submitted by Bryan O'Sullivan, who had problems with Trac's spam filter.)

The HttpResponse implementation uses a normal Python dict to store
headers. This makes it necessary to have every component that sets
headers in a response agree on the exact case to use for every header
name. I found this problem while trying to glue a WSGI app into a
Django app, only to discover that one believes that a header should be
named "content-type", while the other prefers "Content-Type". This
results in a HTTP response that contains both headers, leading to
undefined (and currently bad) results on the client side.

Probably the HttpResponse class should use a dict-like object that
preserves, but ignores, case, so that "Foo" and "fOo" will map to the
same item.

Attachments (5)

case_insensitive_dict.py (3.0 KB) - added by SmileyChris 8 years ago.
Here's a (2.3 compatible) case insensitive dict I whipped up
http_response_insensitive.patch (1.2 KB) - added by SmileyChris 8 years ago.
Here's the patch
http_response_insensitive.2.patch (1.2 KB) - added by SmileyChris 8 years ago.
more backwardsly-compatible
http_response_insensitive.3.patch (7.2 KB) - added by SmileyChris 8 years ago.
Tests and included CaseInsensitiveDict
2970.diff (1.4 KB) - added by PhiR 7 years ago.
refactoring using a basic python dict with lowercase keys

Download all attachments as: .zip

Change History (18)

Changed 8 years ago by SmileyChris

Here's a (2.3 compatible) case insensitive dict I whipped up

Changed 8 years ago by SmileyChris

Here's the patch

comment:1 Changed 8 years ago by SmileyChris

  • Summary changed from HttpResponse should treat headers case-insensitively to [patch] HttpResponse should treat headers case-insensitively

Patch doesn't contain case_insensitive_dict.py, so you'll need to put that in django.utils yourself.

comment:2 Changed 8 years ago by Bastian Kleineidam <calvin@…>

The second part of your patch modifying has_header() is not backwards compatible, and could break existing applications if they used a plain dictionary for header values

If the patch is applied, this should be noted in the documentation.

Changed 8 years ago by SmileyChris

more backwardsly-compatible

comment:3 Changed 8 years ago by SmileyChris

Good point, Bastian. This patch should keep you happier :)

comment:4 Changed 8 years ago by jacob

Looks like a great idea. Can we get some unit tests for both the new data structure and the HttpResponse behavior? Oh, and a revised patch that moved CaseInsensitiveDict into django.utils.datastructures would be nice.

Changed 8 years ago by SmileyChris

Tests and included CaseInsensitiveDict

comment:5 Changed 8 years ago by SmileyChris

Here you go, Jacob.

I haven't included any tests for the HttpResponse behaviour specifically. If you check out the code, it's just doing a basic property(get, set) to keep the identical behaviour as before. That, and I didn't know exactly what tests you'd want to run - I haven't changed any external behaviour apart from the case insensitivity which is being 100% done in the CaseInsensitiveDict.

comment:6 Changed 8 years ago by jacob

Awesome; thanks. I'll check this in later tonight.

comment:7 Changed 7 years ago by SmileyChris

  • Triage Stage changed from Unreviewed to Ready for checkin

comment:8 Changed 7 years ago by mtredinnick

  • Owner changed from adrian to mtredinnick
  • Triage Stage changed from Ready for checkin to Design decision needed

There's a lot of processing overhead going on in the CaseInsensitiveDict class just to preserve the case of the original key. I can't think of a reason when this is really needed (we will document that the headers are case-insensitive so that people won't be surprised when headers.keys() returns all lower-case). Am I missing some obvious use-case beyond "it might be nice"?

If there isn't a strong reason for keeping the original key case, I'd like to replace CaseInsensitiveDict with something like this Python cookbook recipe (I'll use one of the versions from later in the comments, since they fix problems with the earlier version). That code just looks a bit easier to read.

I'll move this to "design decision needed" for the time being. Chris, if I'm missing a good reason why you have that functionality, feel free to point it out. I'm willing to admit to being dense sometimes. No need to go to all the trouble of updating the patch if we can just make the switch: I'll just drop in the replacement and some tests when I do the commit.

comment:9 Changed 7 years ago by SmileyChris

You are right, preserving case it probably doesn't matter... in which case, yea there's probably a simpler way to do it. I was just blindly following Adrian's suggestion to "preserve":

Probably the HttpResponse? class should use a dict-like object that preserves, but ignores, case, so that "Foo" and "fOo" will map to the same item.

Changed 7 years ago by PhiR

refactoring using a basic python dict with lowercase keys

comment:10 Changed 7 years ago by PhiR

  • Component changed from Core framework to HTTP handling
  • Keywords sprintsept14 added
  • Owner changed from nobody to PhiR

The new patch has less overhead and has_header() should be much faster than before. http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 says that field names are case-insensitive so there's no need at all to preserve case.

comment:11 Changed 7 years ago by jacob

  • Resolution set to fixed
  • Status changed from new to closed

(In [6212]) Fixed #2970: made HttpResponse headers case-insensitive. Thanks to SmileyChris for the original patch and PhiR for the final one.

comment:12 Changed 7 years ago by calvin@…

  • Resolution fixed deleted
  • Status changed from closed to reopened

In my experience changing header names to lower-case will break some clients or servers (let alone bad programmed proxies).

To follow the reference, only key lookup must be case insensitive. Otherwise the application should preserve the original header value. So please consider reverting the __setitem__ change and the changed "Content-Type" case.

comment:13 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from reopened to closed

(In [6546]) Changed the way we handle HTTP headers internally so that they appear
case-insensitive, but the original case is preserved for output. This increases the chances of working with non-compliant clients without changing the external interface. Fixed #2970.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.