Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

#24736 closed Cleanup/optimization (fixed)

Document how to customize the 50000 item limit in sitemaps

Reported by: Rich Jones Owned by: Abhaya Agarwal
Component: Documentation Version: dev
Severity: Normal Keywords: sitemap
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: yes UI/UX: no

Description

Hey all!

I recently ran into an interesting and reasonably soluble problem. I've switched from using a static sitemap to Django's Sitemap framework. Now, my database has a ton of items to map (>200,000), so generating the sitemap actually takes quite a long time. Unfortunately, it takes such a long time that my web server actually times out waiting for the request, which is obviously a big problem.

Rather than deal with adding exceptions in my web server or manually creating static sitemaps, I'd greatly prefer to lower the number of items on each page and have a larger number of entries on the index page.

Currently, the pagination uses a magic number of 50000, which is the maximum number of items recommended by Google. That line is here: https://github.com/django/django/blob/c2d5f2903cfaef4f8b17d86f4db706b61073a471/django/contrib/sitemaps/__init__.py#L50

I'd greatly prefer it if this was available as a configurable setting.

This would be a very simple, 2-line fix:

One to add a

SITEMAP_MAX_ITEMS = 50000

to Django's default settings,and another change the line linked about to

limit = settings.SITEMAP_MAX_ITEMS

It's a small change, but this would be greatly useful to me and other Django sites with lots of items that use the Sitemap framework. In the meantime, I'm just going try to override default Sitemap object.

What do you guys think? Does this make sense to you?

Change History (9)

comment:1 by Rich Jones, 10 years ago

Pull request submitted for your gracious consideration.

comment:2 by Markus Holtermann, 10 years ago

Thanks for submitting this issue and patch. However, I'm -1 on adding a global settings variable if you can just override limit on your sitemap class much like protocol: https://docs.djangoproject.com/en/1.8/ref/contrib/sitemaps/#django.contrib.sitemaps.Sitemap.protocol. Or am I missing something?

If we're pursuing this issue I'd prefer to document limit as public API and mention that its value shouldn't exceed 50000.

comment:3 by Markus Holtermann, 10 years ago

Version: 1.8master

comment:4 by Rich Jones, 10 years ago

Fine by me.. that's how I actually ended up solving the problem in my application. I think that makes more sense anyway.

comment:5 by Aymeric Augustin, 10 years ago

Component: contrib.sitemapsDocumentation
Summary: Replace magic 50000 Sitemap item limit with customizable valueDocument how to customize the 50000 item limit in sitemaps
Triage Stage: UnreviewedAccepted

I agree as well. Requalifying as a documentation ticket and accepting.

comment:6 by Abhaya Agarwal, 10 years ago

Owner: changed from nobody to Abhaya Agarwal
Status: newassigned

comment:7 by Claude Paroz, 10 years ago

Has patch: set
Type: New featureCleanup/optimization

comment:8 by Tim Graham <timograham@…>, 10 years ago

Resolution: fixed
Status: assignedclosed

In 9096e2b:

Fixed #24736 -- Documented the Sitemap.limit attribute

comment:9 by Tim Graham <timograham@…>, 10 years ago

In eb00b427:

[1.8.x] Fixed #24736 -- Documented the Sitemap.limit attribute

Backport of 9096e2b5f75abf8e8882937bd3c3d47ccdc24e25 from master

Note: See TracTickets for help on using tickets.
Back to Top