Sitemaps with i18n=True load the whole table in memory

[Julien Palard]

Hi!

It's a bit like: https://code.djangoproject.com/ticket/11572

when enabling i18n on a sitemap, _items uses the following code:

            # Create (item, lang_code) tuples for all items and languages.
            # This is necessary to paginate with all languages already considered.
            items = [
                (item, lang_code)
                for item in self.items()
                for lang_code in self.get_languages_for_item(item)
            ]

The list comprehension loads the whole table times the number of languages in memory.

We tried with a table containing two millions elements and two languages: it does not fit in 16GB of memory and the process gets killed.

[Julien Palard]

While:

    def get_languages_for_item(self, item):
        """Languages for which this item is displayed."""
        return self._languages()

does not use item, something (non trivial) could be implemented to slice the queryset at the right spot (depending on the number of languages), and the total number of elements could be computed by a simple multiplication.

But as long as get_languages_for_item uses item all of this break.