Add support for https to sitemaps

[John Hensley]

I needed to put HTTPS URLs in my sitemaps, and wanted to ping search engines other than Google. The attached patch adds those enhancements.

[Adrian Holovaty]

Sounds like a reasonable enhancement. It'll have to wait until we have a 1.0.x branch. I have not examined this patch.

[miracle2k]

It seems yahoo is pinged twice by this patch, once with the sitemap url (super call), and then again with the base url. Any particular reason for this?

[John Hensley]

Replying to miracle2k:

It seems yahoo is pinged twice by this patch, once with the sitemap url (super call), and then again with the base url. Any particular reason for this?

Well, I'd like to think it was because in trying to get Yahoo! to crawl my sites I found it necessary, but it's been so long I don't remember, and now it does seem redundant and spammy. The overridden ping method can probably be removed entirely, so just the sitemap is submitted.

[John Hensley]

I updated the patch to work with recent code. For now I just removed Yahoo, as the ping service seems to be broken and is always returning 403 Forbidden. It's not possible that I reached their rate limit, I tried from multiple IPs, and it seems to be reported pretty widely. Some claim you now need to get an app ID and use the update notification service instead. If that's true, Yahoo hasn't updated their docs.

At any rate, I'm afraid that's all the time I have for this at the moment, so Yahoo's out.

[John Hensley]

Patch against r13357 for sitemap enhancements.

[John Hensley]

Ignore this.

[Jannis Leidel]

Those are really two features in one ticket, please decide which you want to cover here and open a new ticket for the other feature. Thanks.

[John Hensley]

Sorry, don't have time to spend on this one now. After three years, it's clearly not needed; let's just get it off the list.

[Jannis Leidel]

I think the protocol attribute is definitely useful, IMO.

[Mitar]

+1

[Aymeric Augustin]

Let's decide this ticket is only about HTTPS URLs in sitemaps (something I've missed in the past...) and forget about Yahoo, which no longer offers a "ping" feature.

[Jannis Leidel]

Changing the title..

[John Hensley]

Patch to support HTTPS URLs in sitemaps

[John Hensley]

All right, that's easier. This latest patch just adds HTTPS support.

[vsemenov]

Thanks for the patch! This will fix "URLs not followed" warning in Google Webmaster Tools for HTTPS locations in sitemaps.

[Chris Beaven]

One question, the current patch changes the index view to use the sitemap's protocol for the site map URL locations rather than just relying on request.is_secure(). Isn't this slightly backwards incompatible?

[Jannis Leidel]

That's backwards incompatible indeed, but the feature request is valid. The sitemap class should check if a protocol is specified as a class attribute and fallback to the request.is_secure check. In doubt this could all be a method called get_protocol or similar to allow the users to customize things.

[Marco Paolini]

documentation

[Marco Paolini]

added some documentation for this new feature

[John Hensley]

Backward-compatible patch with docs

[John Hensley]

I've added a patch that includes the docs (thanks, Marco) and falls back to the request protocol if the sitemap doesn't specify it.

[Aymeric Augustin]

There are several major issues with the current patch:

• it isn't acceptable to slap a request attribute on the Sitemap object and then magically use it within Sitemap.get_urls
• duplicating all the tests isn't a sustainable approach
• it seems to me that the patch doesn't actually work because the request attribute is only added in views.index, not views.sitemap

I'm currently working on a more invasive patch that should eventually result in cleaner code. I'll upload a patch when it is ready.

[Aymeric Augustin]

In [17409]:

Fixed #8995 -- Added support for HTTPS in sitemaps.

Modularized tests and did a bit of cleanup while I was in the area.