Normalize newlines from Textarea widgets

[Claude Paroz]

Currently, newlines from Textarea may come in at least two different formats. Most browsers send them as CRLF. But when sent from JS (AJAX), it depends on the browser (Firefox and WebKit seem to send LF-only).

I would suggest to normalize them to LF in all cases. This may also solve the issue about different char counting between client-side and server-side (​http://groups.google.com/group/django-users/browse_thread/thread/3161244a2f119444/5fad134a89039d7b?lnk=gst&q=textarea%2C+max_length#5fad134a89039d7b).

[Claude Paroz]

Normalize newlines from Textarea

[Aymeric Augustin]

If browsers generally send CRLF and we want to normalize, shouldn't we normalize to CRLF — even if our personal tastes are different — to avoid creating regressions for Windows users? For instance if they're building text files from user input, suddenly their files may not display correctly any longer.

Generally speaking, Django is very careful about not altering data sent by the browsers. For instance it doesn't strip whitespace, even though it's a very common request, and it's the right thing to do 99% of the time.

[Claude Paroz]

Replying to aaugustin:

If browsers generally send CRLF and we want to normalize, shouldn't we normalize to CRLF — even if our personal tastes are different — to avoid creating regressions for Windows users? For instance if they're building text files from user input, suddenly their files may not display correctly any longer.

If browsers would systematically return CRLF, you would be right. But this isn't always true, so the situation is already inconsistent and somewhat broken.

Generally speaking, Django is very careful about not altering data sent by the browsers. For instance it doesn't strip whitespace, even though it's a very common request, and it's the right thing to do 99% of the time.

Stripping whitespace is semantically altering content, as leading/trailing space may be wanted and significant (albeit seldom). Changing new line format does not change the content (still semantically speaking), this is only a technical artefact.

[Aymeric Augustin]

Replying to claudep:

If browsers would systematically return CRLF, you would be right. But this isn't always true, so the situation is already inconsistent and somewhat broken.

I didn't explain it well, but my reasoning was:

• regular requests always use CRLF: if we normalize to CRLF, we're backwards compatible, if we normalize to LF, we aren't.
• AJAX requests are browser dependent: no matter which way we normalize, it will have side effects for some people.

In my opinion this is a basic browser interoperability issue that is better left to browser developers / W3C / etc.; but if you're keen on fixing it in Django that's just fine.

[Aymeric Augustin]

What really happens here is that JS code gets the raw content of a textarea, while Django usually gets the content ​encoded on submit.

In IE, the raw content has newlines represented by CRLF; in other browsers, it has LF.

I still believe the correct solution is to handle this in JavaScript, for example: ​http://blog.sarathonline.com/2009/09/javascript-maxlength-for-textarea-with.html