Website docs builder should clean to get rid of old docs

[James Bennett]

Currently it doesn't, which is why we got ticket #18699, and why ​this document still exists on-site even though it was removed from the docs months ago.

A step should be added to clean out old generated HTML, so we don't run into this again.

[Aymeric Augustin <aymeric.augustin@…>]

In 1af4480ae19db8c39b07c28df32a12df72a38f0a/djangoproject.com:

Fix #18930 -- Force a 404 for obsolete documents.

This relies on Document objects being properly deleted when they're
obsolete. The code is designed to do that but I've observed it doesn't
work as expected: duplicate objects get created.

[Aymeric Augustin]

I wiped the Document table as well as the xapian index in production.

Then I did a single rebuild of the docs, and I tried adding a unique index on (version, path).

It failed because of duplicates, which appear to be pages that have been converted into directories at some point (ie. topics/auth.txt => topics/auth/index.txt).

>>> from docs.models import *
>>> from django.db.models import *
>>> sorted(Document.objects.values_list('release', 'path').annotate(count=Count('pk')).filter(count__gt=1).values_list('release__version', 'path', 'count'))
[(u'1.5', u'topics/auth', 2),
 (u'1.5', u'topics/testing', 2),
 (u'dev', u'ref/class-based-views', 2),
 (u'dev', u'ref/contrib/gis/install', 2),
 (u'dev', u'topics/auth', 2),
 (u'dev', u'topics/class-based-views', 2),
 (u'dev', u'topics/testing', 2)]

[Aymeric Augustin <aymeric.augustin@…>]

In d51819ecaab91f2d5b91cdc5d01c22b5aeb3f231/djangoproject.com:

Fix #18930 -- Wipe Sphinx build dir between builds.

This requires serving the docs from another repository, and syncing
the build dir to the serve dir after the build is finished.

[Aymeric Augustin <aymeric.augustin@…>]

In b35966376ac8c1d8caf05a5e882dfb27325a1593/djangoproject.com:

Fix #19809 -- Don't check if docs exist in DB.

The docs database is built from the JSON version of the docs, which
doesn't include special pages such as indexes.

This reverts commit 1af4480ae19db8c39b07c28df32a12df72a38f0a.

d51819ecaab91f2d5b91cdc5d01c22b5aeb3f231 has a better and sufficient
fix.

Refs #18930.