`admindocs` breaks compatibility with `docutils` when rendering reStructuredText

[Malcolm Tredinnick]

Trying to access ​http://www.djangoproject.com/documentation/0.96/tutorial01/ just now gave a traceback. Normal documentation (the equivalent current stuff) works fine.

Traceback (most recent call last):
File "/home/djangoproject.com/django/core/handlers/base.py" in _real_get_response
  81. response = callback(request, *callback_args, **callback_kwargs)
File "/home/djangoproject.com/django_website/apps/docs/views.py" in doc_detail
  40. parts = builder.build_document(client.cat(docpath))
File "/home/djangoproject.com/django_website/apps/docs/builder.py" in build_document
  16. return publish_parts(text, writer=DjangoHTMLWriter(), settings_overrides={'initial_header_level': 2})
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/core.py" in publish_parts
  431. enable_exit_status=enable_exit_status)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/core.py" in publish_programmatically
  612. output = pub.publish(enable_exit_status=enable_exit_status)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/core.py" in publish
  202. self.settings)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/readers/__init__.py" in read
  69. self.parse()
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/readers/__init__.py" in parse
  75. self.parser.parse(self.input, document)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/__init__.py" in parse
  157. self.statemachine.run(inputlines, document, inliner=self.inliner)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in run
  170. input_source=document['source'])
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in run
  232. context, state, transitions)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in check_line
  420. return method(match, context, next_state)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in text
  2883. self.section(title.lstrip(), source, style, lineno + 1, messages)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in section
  308. self.new_subsection(title, lineno, messages)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in new_subsection
  376. node=section_node, match_titles=1)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in nested_parse
  266. node=node, match_titles=match_titles)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in run
  195. results = StateMachineWS.run(self, input_lines, input_offset)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in run
  232. context, state, transitions)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in check_line
  420. return method(match, context, next_state)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in underline
  2658. self.section(title, source, style, lineno - 1, messages)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in section
  308. self.new_subsection(title, lineno, messages)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in new_subsection
  376. node=section_node, match_titles=1)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in nested_parse
  266. node=node, match_titles=match_titles)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in run
  195. results = StateMachineWS.run(self, input_lines, input_offset)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in run
  232. context, state, transitions)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in check_line
  420. return method(match, context, next_state)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in indent
  1081. elements = self.block_quote(indented, line_offset)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in block_quote
  1096. self.nested_parse(blockquote_lines, line_offset, blockquote)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in nested_parse
  266. node=node, match_titles=match_titles)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in run
  195. results = StateMachineWS.run(self, input_lines, input_offset)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in run
  232. context, state, transitions)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in check_line
  420. return method(match, context, next_state)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in bullet
  1187. blank_finish=blank_finish)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in nested_list_parse
  300. node=node, match_titles=match_titles)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in run
  195. results = StateMachineWS.run(self, input_lines, input_offset)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in run
  232. context, state, transitions)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in check_line
  420. return method(match, context, next_state)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in bullet
  2417. listitem, blank_finish = self.list_item(match.end())
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in list_item
  1203. node=listitem)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in nested_parse
  266. node=node, match_titles=match_titles)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in run
  195. results = StateMachineWS.run(self, input_lines, input_offset)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in run
  232. context, state, transitions)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/statemachine.py" in check_line
  420. return method(match, context, next_state)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in text
  2672. paragraph, literalnext = self.paragraph(lines, startline)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in paragraph
  399. textnodes, messages = self.inline_text(text, lineno)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in inline_text
  408. return self.inliner.parse(text, lineno, self.memo, self.parent)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in parse
  492. lineno)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in interpreted_or_phrase_ref
  746. lineno)
File "/usr/lib/python2.5/site-packages/docutils-0.5-py2.5.egg/docutils/parsers/rst/states.py" in interpreted
  802. nodes, messages2 = role_fn(role, rawsource, text, lineno, self)
File "/home/djangoproject.com/django/contrib/admin/utils.py" in default_reference_role
  93. context = inliner.document.settings.default_reference_context

  AttributeError at /documentation/0.96/tutorial01/
  Values instance has no attribute 'default_reference_context'

[Jacob]

Meh, this happens every time I restart Apache. It's something to do with the nasty monkeypatching I'm doing to docutils, but since it goes away quickly I can never reproduce it long enough to figure out what's wrong.

[zeth <theology@…>]

I'm having the same bug as Jacob encountered but I have done no "nasty monkeypatching". I am using plain SVN Django and the default Debian docutils.

Let me be more specific:

I am using the FreeComment from the contrib Comments module, and then filtering the comments through the markup module's restructuredtext filter. Like this:

{{ comment.comment|escape|restructuredtext }}

Certain user input will kill the restructuredtext filter. For example, this piece of markdown:

(http://google.com)[blah]

AttributeError at [absolute URL]

Will give:

Values instance has no attribute 'default_reference_context'

In template [template], error at line 66
Caught an exception while rendering: Values instance has no attribute 'default_reference_context'

Line 66 is our friend: {{ comment.comment|escape|restructuredtext }}

[zeth <theology@…>]

My example user input got filtered by Trac, here it is escaped:

`(http://google.com)[blah]`

[zeth <theology@…>]

The full traceback:

Traceback:
File "/my-django-location/django/template/debug.py" in render_node

71. result = node.render(context)

File "/my-django-location/django/template/debug.py" in render

87. output = force_unicode(self.filter_expression.resolve(context))

File "/my-django-location/django/template/init.py" in resolve

542. new_obj = func(obj, *arg_vals)

File "/my-django-location/django/contrib/markup/templatetags/markup.py" in restructuredtext

88. parts = publish_parts(source=smart_str(value), writer_name="html4css1", settings_overrides=docutils_settings)

File "/usr/lib/python2.4/site-packages/docutils/core.py" in publish_parts

431. enable_exit_status=enable_exit_status)

File "/usr/lib/python2.4/site-packages/docutils/core.py" in publish_programmatically

612. output = pub.publish(enable_exit_status=enable_exit_status)

File "/usr/lib/python2.4/site-packages/docutils/core.py" in publish

204. self.settings)

File "/usr/lib/python2.4/site-packages/docutils/readers/init.py" in read

71. self.parse()

File "/usr/lib/python2.4/site-packages/docutils/readers/init.py" in parse

77. self.parser.parse(self.input, document)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/init.py" in parse

155. self.statemachine.run(inputlines, document, inliner=self.inliner)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in run

171. input_source=documentsource)

File "/usr/lib/python2.4/site-packages/docutils/statemachine.py" in run

240. result = state.eof(context)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in eof

2567. self.blank(None, context, None)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in blank

2558. paragraph, literalnext = self.paragraph(

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in paragraph

400. textnodes, messages = self.inline_text(text, lineno)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in inline_text

409. return self.inliner.parse(text, lineno, self.memo, self.parent)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in parse

493. lineno)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in interpreted_or_phrase_ref

747. lineno)

File "/usr/lib/python2.4/site-packages/docutils/parsers/rst/states.py" in interpreted

803. nodes, messages2 = role_fn(role, rawsource, text, lineno, self)

File "/my-django-location/django/contrib/admin/utils.py" in default_reference_role

94. context = inliner.document.settings.default_reference_context

Exception Type: AttributeError at /absolute/url/of/content/object/
Exception Value: Values instance has no attribute 'default_reference_context'

[eopadoan]

Reproduced on wikiapp too:
​http://pinax.hotcluboffrance.com/tribes/django/wiki/DjangoEvolution/
Traceback and etc:
​http://groups.google.com/group/pinax-updates/browse_thread/thread/a32fe9c4c61e8fee

[Erik Forsberg]

I'm experiencing the same problem. I think I have an idea on what's causing it. It seems like if the bug is triggered by a combination of the following:

1. The apache process is just started.

2. The first (I think it's the first, not sure) request that hits the process is for a page that requires processing by contrib.markup.templatetags.markup.restructuredtext.

3. There's some interpreted text using the default role in the reST. Example:

     `interpreted text using the default role`
   

This text will, in the standard case, be parsed by the reST role named by the variable DEFAULT_INTERPRETED_ROLE in docutils/parsers/rst/roles.py, but django/contrib/admindocs/utils.py (in rev 7967, it's line 100) sets it to 'cmsreference'. I think that's what's causing the problem. Verifying that right now by running my server which got lot's of these errors yesterday with a modified django where the offending line has been disabled.

I have not analyzed why this seems to happen only now and then. I also don't know why admindocs/utils.py is setting DEFAULT_INTERPRETED_ROLE, but it does look like a bad thing to do.

[Erik Forsberg]

Can confirm after about 24h of no-trouble-uptime that my workaround is indeed a valid workaround.

[zenx]

Had the same problem and tried what forsberg did and it works right now correctly. Comment line:
docutils.parsers.rst.roles.DEFAULT_INTERPRETED_ROLE = 'cmsreference'
in django/contrib/admindocs/utils.py

[resplin]

I confirmed that the problem still exists in Django 1.1, and forsberg's workaround solves it.

[HM]

This just bit me as well, I'm on trunk. But shouldn't it be possible to force DEFAULT_INTERPRETED_ROLE back to a working default when not in admin? If admin uses a tag for rst then have them use their own tag, otherwise have another arg on publish_parts. Pff, if push comes to shove, I've managed without the admin before...

[Justin Lilly]

I tracked the offending piece of code through a few moves in SVN and it seems like it entered the codebase in r3.

​http://code.djangoproject.com/browser/django/trunk/django/parts/admin/doc.py?rev=3

Does anyone actually know what this line of code is doing and the ramifications of removing it all together?

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[domguard]

Replying to resplin:

I confirmed that the problem still exists in Django 1.1, and forsberg's workaround solves it.

Same bug, but with django 1.3.1 , I commented the line mentioned in #comment:10 to get rid of it

I just used a blog app using restructured text and contrib.admindocs.
Does that mean I'll always have to patch django if I want to use this code ?

[Russell Keith-Magee]

@quinode: No, but we need a viable patch, first. So far, there's a suggestion as to a workaround, but nobody has answered @justinlilly's question regarding what the purpose of the line was in the first place. I'm not going to commit any patch that removes a line unless someone can verify that the line doesn't have a purpose.

[jgsogo]

I've just written a patch for this bug. I would wan't you to check it out:

​https://github.com/django/django/pull/368

When cmsreference role fails (everywhere but in django admindocs) it tries to exec docutils' default role.

[Peter Poeml]

jgsogo's patch works for me. Thanks.

[anonymous]

Had the same issue, would like to see this fixed in core. This seems to be the relevant commit:

$ git show 7106c68e
commit 7106c68e598e0f2ad2036249122222f1adb3b2d1
Author: Adrian Holovaty <adrian@…>
Date: Wed Dec 7 05:11:19 2005 +0000

Fixed #985 -- Fixed admin docs behavior if docutils isn't installed. Thanks, Tim Keating

git-svn-id: ​http://code.djangoproject.com/svn/django/trunk@1563 bcc190cf-cafb-0310-a4f2-bffc1f526a37

diff --git a/django/contrib/admin/utils.py b/django/contrib/admin/utils.py
index 513067a..80a8f2a 100644
--- a/django/contrib/admin/utils.py
+++ b/django/contrib/admin/utils.py
@@ -90,8 +90,10 @@ def default_reference_role(name, rawtext, text, lineno, inliner, options={}, con

context = inliner.document.settings.default_reference_context
node = docutils.nodes.reference(rawtext, text, refuri=(ROLES[context] % (inliner.document.settings.link_base, text)), options)
return [node], []

-docutils.parsers.rst.roles.register_canonical_role('cmsreference', default_reference_role)
-docutils.parsers.rst.roles.DEFAULT_INTERPRETED_ROLE = 'cmsreference'

-for (name, urlbase) in ROLES.items():

• create_reference_role(name, urlbase)

+if docutils_is_available:
+ docutils.parsers.rst.roles.register_canonical_role('cmsreference', default_reference_role)
+ docutils.parsers.rst.roles.DEFAULT_INTERPRETED_ROLE = 'cmsreference'
+
+ for (name, urlbase) in ROLES.items():
+ create_reference_role(name, urlbase)

[Aymeric Augustin]

django.contrib.markup is deprecated and was recently removed in master.

[Tai Lee]

This is not a problem with django.contrib.markup. This problem still exists for anyone who wants use admindocs and also implement their own restructuredtext filter (or use docutils to render rst for any other purpose).

Django should not assign a *default* interpreted role that *only* works with django.contrib.admindocs.util.parse_rst().

The ideal fix would be for Django to not set a default to begin with, but I do not know of a way to set a default inside parse_rst() only. If Django *must* set a default, then the default should be as compatible with user and 3rd party code as the original default. A refined version of ​https://github.com/django/django/pull/368 should do the trick.

FYI this code appears to have been here from the *very* beginning, when Django was a private framework where it likely didn't need to worry about people wanting to use docutils to render reStructuredText outside of Django's narrow focus.

​https://github.com/django/django/commit/ed114e15106192b22ebb78ef5bf5bce72b419d13

[Tai Lee]

New PR opened.

​https://github.com/django/django/pull/1277

This just sets the default inside parse_rst() and resets it before returning. This appears to be the only function that sets and passes the settings overrides that are required by the "cmsreference" role function.

[Tai Lee]

As apollo13 mentioned on IRC, my approach of setting and unsetting the default inside parse_rst() is not safe. I have updated my PR with a refined version of the approach taken in PR368.

​https://github.com/django/django/pull/1277

[Tai Lee]

Finally figured out how to set the default just for the block of reStructuredText being published in parse_rst(). We don't need to set a global default at all. PR updated.

[Tim Graham <timograham@…>]

In bcd4c3f27d5bcf701337ac7c74c042d90792bdc3:

Fixed #6681 -- Don't break docutils when rendering reStructuredText.

Don't set a global default interpreted role function for reStructuredText.
Instead, use the default-role directive to change the default only within
the parse_rst() function.

Thanks Malcolm Tredinnick for the report.

[Tim Graham]

This causes docutils to throw an error when running the full test suite (or just the webdesign + admin_views tests. I'm not sure the best way to fix it.

$ ./runtests.py django.contrib.webdesign admin_views

<tag:lorem>:2: (ERROR/3) Error in "default-role" directive:
no content permitted.

.. default-role:: cmsreference

    * ``{% lorem %}`` will output the common "lorem ipsum" paragraph
    * ``{% lorem 3 p %}`` will output the common "lorem ipsum" paragraph
      and two random paragraphs each wrapped in HTML ``<p>`` tags
    * ``{% lorem 2 w random %}`` will output two random latin words

The "Example:" bit in the docstring for django.contrib.webdesign.templatetags.webdesign.lorem is parsed as metadata so the bullet list is then interpreted as a new block and unexpectedly indented under the .. default-role directive. Possibly we can fix this particular docstring to avoid the problem, but I'm wondering if other users will encounter the problem with their own docstrings.

[Baptiste Mispelon]

Would dedenting the text work in the general case?
It fixes the current issue and seems like the appropriate thing to do (I'm not very familiar with the surrounding code however so I might be completely wrong).

django/contrib/admindocs/utils.py

@@ -1 +1 @@
 "Misc. utility functions/classes for admin documentation generator."
 
 import re
+from textwrap import dedent
 from email.parser import HeaderParser
 from email.errors import HeaderParseError
 
@@ -76 +77 @@
 
 .. default-role::
 """
-    parts = docutils.core.publish_parts(source % text,
+    parts = docutils.core.publish_parts(source % dedent(text),
                 source_path=thing_being_parsed, destination_path=None,
                 writer_name='html', settings_overrides=overrides)
     return mark_safe(parts['fragment'])

[Baptiste Mispelon]

(sorry for the noise, I forgot to add myself to the Cc list)

[Tim Graham]

Yes, it seems it would fix the issue although not yield the same result as before the first patch was committed. Originally the <ul> would appear in a <blockquote> since it's indented. That's not necessary or really desired in this case - I'm not sure if there's a use case that would benefit from preserving indentation.

[Tim Graham <timograham@…>]

In 975415a8cef38cf2a49c3ce8234c15ef97a69b5f:

Fixed a webdesign template tag docstring to prevent parsing as metadata.

Previously admindocs would throw an error when processing it:
"Error in "default-role" directive: no content permitted."

refs #6681

[Tim Graham]

I decided to fix the webdesign tag docstring. I don't expect normalized docstrings to start with indentation -- clearly the intent here wasn't to parse the "Example:" section as metadata.