Opened 3 years ago

Closed 4 months ago

#18922 closed Bug (duplicate)

Proliferation of dev docs on search engines confuses newbies

Reported by: dloewenherz Owned by: dloewenherz
Component: *.djangoproject.com Version: 1.4
Severity: Normal Keywords:
Cc: markus.magnuson@… Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by aaugustin)

I brought this issue up earlier today at DjangoCon, but the basic issue can be summarized hence:

  1. User searches to find info on a specific feature, gets directed to dev documentation.
  2. Said feature (on the development version) is backwards incompatible with previous versions of Django.
  3. User does not know better, assumes Django has a bug.
  4. Invalid bug is filed in trac.

EDIT: rejected proposal removed, see comment 4.

Change History (17)

comment:1 Changed 3 years ago by dloewenherz

  • Needs documentation unset
  • Needs tests unset
  • Owner changed from nobody to dloewenherz
  • Patch needs improvement unset

comment:2 Changed 2 years ago by aaugustin

  • Resolution set to wontfix
  • Status changed from new to closed

Thanks for the suggestion.

I understand the idea, but I'm not ready to wipe the dev docs from Google's index. People should be able to find information about an upcoming feature by googling its name.

Besides, we take care to mention in which version new feature are added. And closing invalid tickets is cheap. The most troublesome change was the new {% url %], and the flow of tickets eventually stopped.

tl;dr The cost of the solution seems too high to me compared to the magnitude of the problem.

comment:3 Changed 2 years ago by dloewenherz

This issue wasn't opened with a specific solution in mind--it's just illuminating that this is a problem. I get if the solution I presented is a bit too drastic, but I think closing this ticket ignores the real issue.

comment:4 Changed 2 years ago by aaugustin

  • Description modified (diff)
  • Resolution wontfix deleted
  • Status changed from closed to reopened
  • Triage Stage changed from Unreviewed to Accepted

Yes, we can keep this ticket to track other ideas to resolve this problem.

For the record, the original proposal was:

Hopefully, removing the dev documentation pages from search engines will help solve
this issue, since those who want to read the dev docs can just click through to
them by using the version navigation at the bottom of the docs pages.

I think there is a small subset of actual Django users who run their applications
on trunk. I don't have data to back that up but I can't imagine it's a lot.

I talked with Alex earlier today about how to deal with this issue--he also suggested
redirecting users from dev -> 1.4 (or whatever the latest version is). I initially
thought it was a good idea, but I thought about it and realized you would have to at
least add some sort of referrer check. I then thought whether it would be ok to do
something like

    def conditional_documentation_redirect(request):
        if not request.META['REFERER'].startswith("http://docs.djangoproject.com"):
            return HttpResponseRedirect #... and so on

This felt wrong to me because if that sort of check were in place, users would
no longer be able to permalink to dev docs.

PR @ https://github.com/django/djangoproject.com/pull/43

comment:5 Changed 2 years ago by ptone

I think if there is a choice when addressing this - we need to make special efforts for the tutorials.

Perhaps a special admonition like from #14464

refs: #16954, #19359

comment:6 Changed 2 years ago by Aymeric Augustin <aymeric.augustin@…>

In 248bc06ed8904fff30c648737a97c9a01628686f/djangoproject.com:

Add a sitemap for the docs site.

Current docs have a higher priority than dev docs. Refs #18922.

comment:7 Changed 2 years ago by Aymeric Augustin <aymeric.augustin@…>

In 170944d1f6f554a44e7f6d735a55d13531a9b7cf/djangoproject.com:

Added /stable/ that redirects to the default version.

Refs #18922. Unfortunately this isn't referenced anywhere and I don't
know how search engines could pick this URL.

Refs #19783.

comment:8 Changed 2 years ago by aaugustin

  • Status changed from reopened to new

comment:9 Changed 2 years ago by giuliettamasina

Is it a reasonable request that any version of the documentation other than dev and the latest release should be hidden from search indexes entirely?

Tried searching for "django authentication" on Google now, and it shows dev, 1.5, and 1.4, in that order. I think 1.4 should be hidden to avoid this kind of thing.

comment:10 Changed 2 years ago by giuliettamasina

  • Cc markus.magnuson@… added

comment:11 Changed 2 years ago by aaugustin

1.4 is still supported, it shouldn't be hidden from search engines.

comment:12 Changed 2 years ago by giuliettamasina

Just out of curiosity, how are versions 1.3 and below hidden from search index?

(They are all in the sitemap, so obviously search engines see them there, although with low priority.)

comment:13 Changed 2 years ago by aaugustin

They aren't hidden, and depending on what you're searching Google can still return various more or less recent versions of the docs.

Our main problem is that the docs for each new release duplicate content from previous releases, and Google favors the original URL in this case.

Canonicalizing /en/1.5/ (or whatever the current version is) => /en/stable/ may help.

comment:14 Changed 2 years ago by giuliettamasina

Does that mean that /en/stable/ should be included in sitemap.xml with priority 1, instead of the latest version (i.e. /en/1.5/). Or should both variants be in there? Or are we talking redirects?

Sorry to hassle you with a lot of questions, but I'm looking to hopefully improve the situation, because I'm annoyed by the proliferation too :)

comment:15 Changed 20 months ago by timo

  • Has patch unset

comment:16 Changed 18 months ago by aaugustin

Eric Holscher just pointed me to http://docs.readthedocs.org/en/latest/faq.html#can-i-make-search-engines-only-see-one-version-of-my-docs.

We could use that to point search engines to the stable version.

comment:17 Changed 4 months ago by timgraham

  • Resolution set to duplicate
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.
Back to Top