Code

Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#6387 closed (fixed)

Markdown for Python v1.7 support needed in contrib.markup -- handling of unicode data changed

Reported by: cogg Owned by: cogg
Component: Contrib apps Version: master
Severity: Keywords: unicode markdown
Cc: waylan@…, yatiohi@…, jerojasro@…, roderikk@…, django@… Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

In v1.7rc1 Markdown moved to expecting unicode and always returning unicode. This causes the utf-8 encoding method that django.contrib.markup currently uses to generate incorrect encodings.

I am attaching a patch that changes the Markdown version and correctly passes unicode if the version is high enough.

Attachments (3)

markdown-1.7-support.diff (965 bytes) - added by cogg 6 years ago.
Support for Markdown for Python v1.7rc1 and later
markdown-1.7-support2.diff (1.0 KB) - added by jedie 6 years ago.
markdown-1.7-support2.2.diff (1.0 KB) - added by Dmitri Fedortchenko <zeraien@…> 6 years ago.
Minor syntax error in previous patch, : missing from if statement. Corrected.

Download all attachments as: .zip

Change History (14)

comment:1 Changed 6 years ago by cogg

  • Owner changed from nobody to cogg
  • Status changed from new to assigned

Link to mailing list announcement that describes changes to markdown. Relevant bit:

Additionally, the `encoding` argument has been removed from both `markdown` and
`Markdown`. Markdown expects unicode (or ascii) input and it is the
users responsibility to ensure that's what is provided. Therefore, all
output is in unicode. Only `markdownFromFile` accepts an encoding.

Changed 6 years ago by cogg

Support for Markdown for Python v1.7rc1 and later

comment:2 Changed 6 years ago by wayla

  • Cc waylan@… added

There is a duplicate of this at #6577 with some additional comments.

comment:3 Changed 6 years ago by ctrochalakis

  • Cc yatiohi@… added

comment:4 Changed 6 years ago by anonymous

  • Cc jerojasro@… added

comment:5 Changed 6 years ago by anonymous

  • Cc roderikk@… added

The patch works fine. How can we get this implemented?

comment:6 Changed 6 years ago by jedie

  • Cc django@… added

The patch didn't work for older version of markdown!
The version_info added in v1.6.2rc-2 previous versions has no version_info, see: http://python-markdown.svn.sourceforge.net/viewvc/python-markdown/markdown.py?r1=30&r2=31&pathrev=94

I will attached a new patch.

Changed 6 years ago by jedie

Changed 6 years ago by Dmitri Fedortchenko <zeraien@…>

Minor syntax error in previous patch, : missing from if statement. Corrected.

comment:7 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [7423]) Fixed #6387 -- Updated markdown filter to handle markdown-1.7 as well as
earlier versions. Patch from cogg, jedie and Dmitri Fedortchenko.

comment:8 Changed 6 years ago by italomaia

  • Needs tests set
  • Patch needs improvement set
  • Resolution fixed deleted
  • Status changed from closed to reopened

This last change breaks markdown for me. Using ubuntu hardy, sqlite3 and python-markdown 1.6b1(repository).

return mark_safe(force_unicode(markdown.markdown(smart_str(value), extensions, safe_mode=safe_mode))) 

# unicode support only in markdown v1.7 or above. 
# version_info exist only in markdown v1.6.2rc-2 or above. 
if getattr(markdown, "version_info", None) < (1,7): 
    return mark_safe(force_unicode(markdown.markdown(smart_str(value), extensions, safe_mode=safe_mode))) 
else: 
    return mark_safe(markdown.markdown(force_unicode(value), extensions, safe_mode=safe_mode)) 

If i use

return mark_safe(markdown.markdown(force_unicode(value), extensions, safe_mode=safe_mode)) 

The error occors when i have a string with special characters on it, like ã or ç.

Without

if getattr(markdown, "version_info", None) < (1,7): 
    return mark_safe(force_unicode(markdown.markdown(smart_str(value), extensions, safe_mode=safe_mode))) 
else: 

it works great!

comment:9 Changed 6 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from reopened to closed

@italomaia: Please open a new ticket with the details of your failure (including an example of how to repeat the problem -- it's a little hard to work out what you're doing in this comment).

The main issue in this ticket has been fixed and you're seeing something that's a bit different: possibly due to a specific version, possibly due to the input you're passing... at the moment, it's hard to tell. But it will just get confusing if we mix up that report with the original issue here.

In the new report, please include what version_info reports for you version of markdown. It's a little annoying to have to support a beta release when the final has been out for ages (and subsequent releases fixed some problems in b1 -- I know because I sent in the patches to markdown). I wish Ubuntu would stop shipping stuff like that, or at least upgrade when the real version comes out. But if there's a way of distinguishing the versions, we'll try to handle it.

comment:10 Changed 6 years ago by wayla

@italomaia: It appears that the issue you are having is discussed in #5663. See my comment (9) on that ticket

comment:11 Changed 6 years ago by italomaia

  • Needs tests unset
  • Patch needs improvement unset

Thanks Wayla. That was pretty much what i meant. mtred, i'll send them a email asking for new markdown version. And, by the way, the patch in
ticket 5663 works for me(it was the same solution i did here, but, as it is there, is official! =D)
Sorry if i was not clear.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.