Code snippets display without formatting in PDF version of the docs

[adam-glodowski]

In PDF version of documentation (for 1.7) there is a problem with formatting examples (files). In html version it is quite nice presented, but in PDF it is not readable.

[adam-glodowski]

screen from PDF

[Baptiste Mispelon]

Closed #23962 as a duplicate and the reporter there said that the epub is also broken in a similar way.

[Berker Peksag]

This could be related to Sphinx version. Attaching screenshots from both Sphinx 1.2.0 and 1.2.3.

[Berker Peksag]

Sphinx 1.2.0

[Berker Peksag]

Sphinx 1.2.3

[Tim Graham]

It looks like the next step is to generate the PDF with Sphinx 1.2.1 and 1.2.2 to see if we can find the change in Sphinx that caused this to break. Then we can figure out if it's something we need to change or a problem with Sphinx.

[Tim Graham]

Bisected to ​https://github.com/sphinx-doc/sphinx/commit/0103ff2a2b612994b077d9d1e37692ba896d86dc

[Graham Wideman]

FIX FOR THIS ISSUE

I am the poster of issue 25738 which describes this bug in more detail, but was closed as a duplicate of this one here.

The bug revolves around code listings produced with the Sphinx directive 'snippet'. That directive is custom to Django, and is added via [sphinx]/docs/_ext/djangodocs.py.

It turns out that the implementation, in visit_snippet_latex( ) and depart_snippet_latex( ) was hosed up. May never have worked, or inherited sphinx code changed from under it at some point.

Anyhow, I will attach a revised djangodocs.py which produces a correct-looking snippet output into the latex output file, which is then converted into the PDF version. (djangodocs-latex-pdf-fix-gw-2015-11-14.py)

I will also attach a screen cap from a page in the PDF, showing a sample of the broken output, and the fixed output. (DjangoDocsBrokenCodeFormattingFixed01.png)

[Graham Wideman]

Shows effect of fix for this issue.

[Graham Wideman]

djangodocs.py with fixed visit_snippet_latex() and depart_snippet_latex

[Graham Wideman]

For completeness, I note that, to test this fix, I had to set up an environment that could make the docs, and that setup produced a large number of warning and possibly errors. This involves Sphinx, and separately, the giant hairball texlive in order that Sphinx can run latexpdf. In neither case am I likely to be running the same versions as used by the Django project in producing the docs offered online. So it's quite likely some or possibly all of the warnings/errors are attributable to version differences. In any case, I believe that this should not impact the efficacy of the fix I've presented, but it obviously needs testing with the Sphinx pipeline actually used by the Django project.

[Tim Graham]

Thanks for looking at this. I tested the patch and one problem seems to be that the table of contents no longer displays. Could you try to fix that and submit the patch as a pull request if possible?

[Graham Wideman]

Hi Tim,

The PDF I generated had a TOC, but on repeat testing, I see how on a clean build it does not. This seems to apply both using the original djangodocs.py, or my revised one.

After some testing, I see that the proximal cause of the missing TOC is that the TOC is produced _after_ the first pass of producing a PDF (so that pagination has been calculated I guess), and so doesn't get incorporated until a second (non-clean) or subsequent run.

It's not clear whether this behavior is peculiar to the up-to-date sphinx and latex pipeline I installed.

However, it also appears that the Makefile produced in /_build/ for the latex->pdf phase is supposed to repeat that phase (presumably to incorporate the TOC etc), and this may stall if there are errors. And there does seem to be an error associated with the code that I revised. So I will examine that in more detail.

It might also help if you could post some details of the sphinx version and latex->pdf setup that the official docs pipeline uses.

[Graham Wideman]

Revised fix, now builds without errors, and includes TOC.

Issues addressed:

1. In order for the overall 'make latexpdf' to succeed (with TOC), the build must proceed without errors. There is a separate nested docs/_build/latex/Makefile for the latex->pdf phase, which invokes pdflatex three times, in order to iterate toward stable pagination to arrive at page numbers for TOC, index and cross-references. If any of these steps fails, the resulting PDF is incomplete.

So it's essential for all errors to be fixed. [Edit: It's also essential to start with an empty _build directory, so that leftovers from previous build don't lead to false success.]

2. The latex code for the grey box containing the filename must not include unescaped underscores, as that's some kind of a latex error in this context, preventing make from completing. So in the djangodocs.py code that constructs that grey box (the 'colorbox'), I added backslash escape to the the underscores.

3. Make was failing when the latex->pdf process encountered unexpected unicode characters, originally from one of the source files, (/docs/ref/unicode.txt. For example on line 230). For testing purposes, I simply renamed that file so it didn't get included.

The correct fix involves figuring out why the bogus characters are in there in the first place. Are they editing mistakes? Did they get there due to unwanted translation from some previous source form? Or are they intentional, in which case they could be added to the list of DeclareUnicodeCharacter entries in the preample section of the /docs/conf.py Sphinx config file. [Edit: This problem addressed in later comment.]

With these issues taken care of, the build proceeds without errors and the file I will attach produces a good PDF output: djangodocs-latex-pdf-fix-gw-2015-11-14C.py

[Graham Wideman]

djangodocs.py with even better fixed visit_snippet_latex() and depart_snippet_latex

[Graham Wideman]

I was going to upload the "good" PDF as proof of pudding, but it's ~6M and apparently uploads are limited to 256k. Let me know if for some reason you need it. Also, sorry, can't produce a patch at the moment, but the file I uploaded has a very clear region that's modified, limited to two functions only.

[Graham Wideman]

For completeness, here's a way to get 'make latexpdf' to complete, even including the unicode.txt source file that contains problem unicode characters.

The problem characters are in a code box in Chapter 6, section "URI and IRI handling" > "An example to demonstrate:". There's two heart characters and a checkmark. Those characters are legit in RST, but not in Latex without special handling. (So in my view it's an error for Sphinx to emit them into Latex without including the appropriate handling for them.) Later in that chapter there are some Scandinavian characters, but these are in a range that's apparently OK in Latex.

To avoid erroring out on the two characters in question, modify .../docs/conf.py as follows:

latex_elements = {
    'preamble': ('\\DeclareUnicodeCharacter{2264}{\\ensuremath{\\le}}'
                 '\\DeclareUnicodeCharacter{2265}{\\ensuremath{\\ge}}'
                 '\\DeclareUnicodeCharacter{2665}{[unicode-heart]}'
                 '\\DeclareUnicodeCharacter{2713}{[unicode-checkmark]}')

This adds handling for the heart and checkmark characters in the latex->pdf step, and simply places the provided strings in their place. It does not actually cause the proper glyphs to appear in the PDF. I played around a bit with that, but couldn't figure out what latex library needed to be included to make that happen. But at least this avoids a docs make error.

[Tim Graham]

Thanks! Could you please check the ​pull request I created?

[Graham Wideman]

I have looked at the patches and they are pretty much what I submitted, so that looks agreeable.

I've also downloaded the patched conf.py and djangodocs.py files and substituted them into my django docs directory tree, then run 'make latexpdf' into a fresh build directory. This runs fine, no errors, and produces a PDF with properly formatted snippet boxes, no problem with underscore-containing file names, and performs the intended substitution of the heart and checkmark unicode characters.

So, looks good to me!

Thanks for your attention to this,

• Graham

[Tim Graham <timograham@…>]

In e48a5b5:

Fixed #23751 -- Fixed code snippet formatting in docs PDF.

Thanks Graham Wideman for the patch.

[Tim Graham <timograham@…>]

In a80d70dd:

Refs #23751 -- Added some unicode characters to the latex preamble.

Thanks Graham Wideman for the patch.

[Tim Graham <timograham@…>]

In 937ce019:

[1.9.x] Fixed #23751 -- Fixed code snippet formatting in docs PDF.

Thanks Graham Wideman for the patch.

Backport of e48a5b5a03181b3ada6183595eb6e1c599a9657a from master

[Tim Graham <timograham@…>]

In 2315f5f4:

[1.9.x] Refs #23751 -- Added some unicode characters to the latex preamble.

Thanks Graham Wideman for the patch.

Backport of a80d70dd10f3c7d483579734e524c94c62832070 from master

[Tim Graham <timograham@…>]

In 904c47bc:

[1.7.x] Fixed #23751 -- Fixed code snippet formatting in docs PDF.

Thanks Graham Wideman for the patch.

Backport of e48a5b5a03181b3ada6183595eb6e1c599a9657a from master

[Tim Graham <timograham@…>]

In 3bd43598:

[1.7.x] Refs #23751 -- Added some unicode characters to the latex preamble.

Thanks Graham Wideman for the patch.

Backport of a80d70dd10f3c7d483579734e524c94c62832070 from master

[Tim Graham <timograham@…>]

In a3a27de:

[1.8.x] Fixed #23751 -- Fixed code snippet formatting in docs PDF.

Thanks Graham Wideman for the patch.

Backport of e48a5b5a03181b3ada6183595eb6e1c599a9657a from master

[Tim Graham <timograham@…>]

In 583d3c08:

[1.8.x] Refs #23751 -- Added some unicode characters to the latex preamble.

Thanks Graham Wideman for the patch.

Backport of a80d70dd10f3c7d483579734e524c94c62832070 from master