Ticket #12308: 12308-with-mysql-support-dont-use.patch

docs/index.txt

@@ -66 +66 @@
       :doc:`Transactions <topics/db/transactions>` |
       :doc:`Aggregation <topics/db/aggregation>` |
       :doc:`Custom fields <howto/custom-model-fields>` |
-      :doc:`Multiple databases <topics/db/multi-db>`
+      :doc:`Multiple databases <topics/db/multi-db>` |
+      :doc:`Tablespaces <topics/db/tablespaces>`
 
     * **Other:**
       :doc:`Supported databases <ref/databases>` |

docs/howto/custom-model-fields.txt

@@ -218 +218 @@
     * :attr:`~django.db.models.Field.choices`
     * :attr:`~django.db.models.Field.help_text`
     * :attr:`~django.db.models.Field.db_column`
-    * :attr:`~django.db.models.Field.db_tablespace`: Currently only used with
-      the Oracle backend and only for index creation. You can usually ignore
-      this option.
+    * :attr:`~django.db.models.Field.db_tablespace`: Only for index creation,
+      if the backend supports :doc:`tablespaces </topics/db/tablespaces>` for
+      indexes. You can usually ignore this option.
     * :attr:`~django.db.models.Field.auto_created`: True if the field was
       automatically created, as for the `OneToOneField` used by model
       inheritance. For advanced use only.

docs/topics/db/tablespaces.txt

@@ +1 @@
+===========
+Tablespaces
+===========
+
+A common paradigm for optimizing performance in database systems is the use of
+`tablespaces`_ to organize disk layout.
+
+.. _`tablespaces`: http://en.wikipedia.org/wiki/Tablespace
+
+.. warning::
+    Django does not create the tablespaces for you. Please refer to your
+    database engine's documentation for details on creating and managing
+    tablespaces.
+
+
+Declaring tablespaces for tables
+--------------------------------
+
+A tablespace can be specified for the table generated by a model by supplying
+the :attr:`~django.db.models.Options.db_tablespace` option inside the model's
+``class Meta``. This option also affects tables automatically created for
+:class:`~django.db.models.ManyToManyField`\ s in the model.
+
+You can use the :setting:`DEFAULT_TABLESPACE` setting to specify a default value
+for :attr:`~django.db.models.Options.db_tablespace`. These is useful for setting
+a tablespace for the built-in Django apps and other applications whose code you
+cannot control.
+
+Declaring tablespaces for indexes
+---------------------------------
+
+You can pass the :attr:`~django.db.models.Field.db_tablespace` option to a
+``Field`` constructor to specify an alternate tablespace for the ``Field``'s
+column index. If no index would be created for the column, the option is
+ignored.
+
+You can use the :setting:`DEFAULT_INDEX_TABLESPACE` setting to specify
+a default value for :attr:`~django.db.models.Field.db_tablespace`.
+
+If :attr:`~django.db.models.Field.db_tablespace` isn't specified and you didn't
+set :setting:`DEFAULT_INDEX_TABLESPACE`, the index is created in the same
+tablespace as the tables.
+
+An example
+----------
+
+.. code-block:: python
+
+    class TablespaceExample(models.Model):
+        name = models.CharField(max_length=30, db_index=True, db_tablespace="indexes")
+        data = models.CharField(max_length=255, db_index=True)
+        edges = models.ManyToManyField(to="self", db_tablespace="indexes")
+
+        class Meta:
+            db_tablespace = "tables"
+
+In this example, the tables generated by the ``TablespaceExample`` model (i.e.
+the model table and the many-to-many table) would be stored in the ``tables``
+tablespace. The index for the name field and the indexes on the many-to-many
+table would be stored in the ``indexes`` tablespace. The ``data`` field would
+also generate an index, but no tablespace for it is specified, so it would be
+stored in the model tablespace ``tables`` by default.
+
+Database support
+----------------
+
+When you use a backend that lacks support for tablespaces, Django ignores all
+tablespace-related options.
+
+Here is the current support matrix:
+
+=================== =================== ===========
+Database            Tables              Indexes
+=================== =================== ===========
+SQLite              no                  no
+PostgreSQL          yes                 yes
+MySQL               partial, see below  no
+Oracle              yes                 yes
+=================== =================== ===================
+
+MySQL supports tablespaces since version 5.1.6. They're only useful in
+combination with MySQL Cluster NDB.
+
+.. versionchanged:: 1.4
+    Until Django 1.4, tablespaces were only supported by the Oracle backend.

docs/topics/db/index.txt

@@ -17 +17 @@
    sql
    transactions
    multi-db
+   tablespaces
    optimization

docs/releases/1.4.txt

@@ -245 +245 @@
   code are slightly emphasized. This change makes it easier to scan a stacktrace
   for issues in user code.
 
+* :doc:`Tablespace support </topics/db/tablespaces>` in PostgreSQL and MySQL.
+
 * Customizable names for :meth:`~django.template.Library.simple_tag`.
 
 * In the documentation, a helpful :doc:`security overview </topics/security>`

docs/ref/models/fields.txt

@@ -178 +178 @@
 
 .. attribute:: Field.db_tablespace
 
-The name of the database tablespace to use for this field's index, if this field
-is indexed. The default is the project's :setting:`DEFAULT_INDEX_TABLESPACE`
-setting, if set, or the :attr:`~Field.db_tablespace` of the model, if any. If
-the backend doesn't support tablespaces, this option is ignored.
+The name of the :doc:`database tablespace </topics/db/tablespaces>` to use for
+this field's index, if this field is indexed. The default is the project's
+:setting:`DEFAULT_INDEX_TABLESPACE` setting, if set, or the
+:attr:`~Options.db_tablespace` of the model, if any. If the backend doesn't
+support tablespaces for indexes, this option is ignored.
 
 ``default``
 -----------

docs/ref/models/options.txt

@@ -66 +66 @@
 
 .. attribute:: Options.db_tablespace
 
-    The name of the database tablespace to use for the model. If the backend
-    doesn't support tablespaces, this option is ignored.
+    The name of the :doc:`database tablespace </topics/db/tablespaces>` to use
+    for this model. The default is the project's :setting:`DEFAULT_TABLESPACE`
+    setting, if set. If the backend doesn't support tablespaces, this option is
+    ignored.
 
 ``get_latest_by``
 -----------------

docs/ref/databases.txt

@@ -620 +620 @@
 In this case, the Oracle backend will use a separate ``SELECT`` query to
 retrieve AutoField values.
 
-Tablespace options
-------------------
-
-A common paradigm for optimizing performance in Oracle-based systems is the
-use of `tablespaces`_ to organize disk layout. The Oracle backend supports
-this use case by adding ``db_tablespace`` options to the ``Meta`` and
-``Field`` classes.  (When you use a backend that lacks support for tablespaces,
-Django ignores these options.)
-
-.. _`tablespaces`: http://en.wikipedia.org/wiki/Tablespace
-
-A tablespace can be specified for the table(s) generated by a model by
-supplying the ``db_tablespace`` option inside the model's ``class Meta``.
-Additionally, you can pass the ``db_tablespace`` option to a ``Field``
-constructor to specify an alternate tablespace for the ``Field``'s column
-index. If no index would be created for the column, the ``db_tablespace``
-option is ignored::
-
-    class TablespaceExample(models.Model):
-        name = models.CharField(max_length=30, db_index=True, db_tablespace="indexes")
-        data = models.CharField(max_length=255, db_index=True)
-        edges = models.ManyToManyField(to="self", db_tablespace="indexes")
-
-        class Meta:
-            db_tablespace = "tables"
-
-In this example, the tables generated by the ``TablespaceExample`` model
-(i.e., the model table and the many-to-many table) would be stored in the
-``tables`` tablespace. The index for the name field and the indexes on the
-many-to-many table would be stored in the ``indexes`` tablespace. The ``data``
-field would also generate an index, but no tablespace for it is specified, so
-it would be stored in the model tablespace ``tables`` by default.
-
-Use the :setting:`DEFAULT_TABLESPACE` and :setting:`DEFAULT_INDEX_TABLESPACE`
-settings to specify default values for the db_tablespace options.
-These are useful for setting a tablespace for the built-in Django apps and
-other applications whose code you cannot control.
-
-Django does not create the tablespaces for you. Please refer to `Oracle's
-documentation`_ for details on creating and managing tablespaces.
-
-.. _`Oracle's documentation`: http://download.oracle.com/docs/cd/B19306_01/server.102/b14200/statements_7003.htm#SQLRF01403
-
 Naming issues
 -------------
 

docs/ref/settings.txt

@@ -857 +857 @@
 Default: ``''`` (Empty string)
 
 Default tablespace to use for indexes on fields that don't specify
-one, if the backend supports it.
+one, if the backend supports it (see :doc:`/topics/db/tablespaces`).
 
 .. setting:: DEFAULT_TABLESPACE
 
@@ -867 +867 @@
 Default: ``''`` (Empty string)
 
 Default tablespace to use for models that don't specify one, if the
-backend supports it.
+backend supports it (see :doc:`/topics/db/tablespaces`).
 
 .. setting:: DISALLOWED_USER_AGENTS
 

tests/modeltests/tablespaces/tests.py

@@ +1 @@
+from django.db import connection
+from django.db import models
+from django.core.management.color import no_style 
+from django.test import TestCase, skipIfDBFeature, skipUnlessDBFeature
+
+from models import Article, ArticleRef, Scientist, ScientistRef
+
+# Automatically created models
+Authors = Article._meta.get_field('authors').rel.through
+Reviewers = Article._meta.get_field('reviewers').rel.through
+
+# We can't test the DEFAULT_TABLESPACE and DEFAULT_INDEX_TABLESPACE settings
+# because they're evaluated when the model class is defined. As a consequence,
+# @override_settings won't work.
+
+def sql_for_table(model):
+    return '\n'.join(connection.creation.sql_create_model(model, no_style())[0])
+
+def sql_for_index(model):
+    return '\n'.join(connection.creation.sql_indexes_for_model(model, no_style()))
+
+
+class TablespacesTests(TestCase):
+
+    def setUp(self):
+        for model in Article, Authors, Reviewers, Scientist:
+            model._meta.managed = True
+
+    def tearDown(self):
+        for model in Article, Authors, Reviewers, Scientist:
+            model._meta.managed = False
+
+    def assertNumContains(self, haystack, needle, count):
+        real_count = haystack.count(needle)
+        self.assertEqual(real_count, count, "Found %d instances of '%s', "
+                "expected %d" % (real_count, needle, count))
+
+    @skipUnlessDBFeature('supports_tablespaces')
+    @skipUnlessDBFeature('supports_index_tablespaces')
+    def test_tablespace_for_model(self):
+        # 1 for the table + 1 for the index on the primary key
+        self.assertNumContains(sql_for_table(Scientist), 'tbl_tbsp', 2)
+
+    @skipUnlessDBFeature('supports_tablespaces')
+    @skipIfDBFeature('supports_index_tablespaces')
+    def test_tablespace_for_model_without_index(self):
+        # 1 for the table
+        self.assertNumContains(sql_for_table(Scientist), 'tbl_tbsp', 1)
+
+    @skipIfDBFeature('supports_tablespaces')
+    def test_tablespace_ignored_for_model(self):
+        # No tablespace-related SQL
+        self.assertEqual(sql_for_table(Scientist),
+                         sql_for_table(ScientistRef).replace('ref', ''))
+
+    @skipUnlessDBFeature('supports_tablespaces')
+    @skipUnlessDBFeature('supports_index_tablespaces')
+    def test_tablespace_for_indexed_field(self):
+        # 1 for the table + 1 for the primary key + 1 for the index on name
+        self.assertNumContains(sql_for_table(Article), 'tbl_tbsp', 3)
+        # 1 for the index on reference
+        self.assertNumContains(sql_for_table(Article), 'idx_tbsp', 1)
+
+    @skipUnlessDBFeature('supports_tablespaces')
+    @skipIfDBFeature('supports_index_tablespaces')
+    def test_tablespace_for_indexed_field(self):
+        # 1 for the table
+        self.assertNumContains(sql_for_table(Article), 'tbl_tbsp', 1)
+        self.assertNumContains(sql_for_table(Article), 'idx_tbsp', 0)
+
+    @skipIfDBFeature('supports_tablespaces')
+    def test_tablespace_ignored_for_indexed_field(self):
+        # No tablespace-related SQL
+        self.assertEqual(sql_for_table(Article),
+                         sql_for_table(ArticleRef).replace('ref', ''))
+
+    @skipUnlessDBFeature('supports_tablespaces')
+    @skipUnlessDBFeature('supports_index_tablespaces')
+    def test_tablespace_for_many_to_many_field(self):
+        # The join table of the ManyToManyField always goes to the tablespace
+        # of the model.
+        self.assertNumContains(sql_for_table(Authors), 'tbl_tbsp', 2)
+        self.assertNumContains(sql_for_table(Authors), 'idx_tbsp', 0)
+        # The ManyToManyField declares no db_tablespace, indexes for the two
+        # foreign keys in the join table go to the tablespace of the model.
+        self.assertNumContains(sql_for_index(Authors), 'tbl_tbsp', 2)
+        self.assertNumContains(sql_for_index(Authors), 'idx_tbsp', 0)
+
+        # The join table of the ManyToManyField always goes to the tablespace
+        # of the model.
+        self.assertNumContains(sql_for_table(Reviewers), 'tbl_tbsp', 2)
+        self.assertNumContains(sql_for_table(Reviewers), 'idx_tbsp', 0)
+        # The ManyToManyField declares db_tablespace, indexes for the two
+        # foreign keys in the join table go to this tablespace.
+        self.assertNumContains(sql_for_index(Reviewers), 'tbl_tbsp', 0)
+        self.assertNumContains(sql_for_index(Reviewers), 'idx_tbsp', 2)

tests/modeltests/tablespaces/models.py

@@ +1 @@
+from django.db import models
+
+# Since the test database doesn't have tablespaces, it's impossible for Django
+# to create the tables for models where db_tablespace is set. To avoid this
+# problem, we mark the models as unmanaged, and temporarily revert them to
+# managed during each test (see setUp and tearDown).
+
+class ScientistRef(models.Model):
+    name = models.CharField(max_length=50)
+
+class ArticleRef(models.Model):
+    title = models.CharField(max_length=50, unique=True)
+    code = models.CharField(max_length=50, unique=True)
+    authors = models.ManyToManyField(ScientistRef, related_name='articles_written_set')
+    reviewers = models.ManyToManyField(ScientistRef, related_name='articles_reviewed_set')
+
+class Scientist(models.Model):
+    name = models.CharField(max_length=50)
+    class Meta:
+        db_tablespace = 'tbl_tbsp'
+        managed = False
+
+class Article(models.Model):
+    title = models.CharField(max_length=50, unique=True)
+    code = models.CharField(max_length=50, unique=True, db_tablespace='idx_tbsp')
+    authors = models.ManyToManyField(Scientist, related_name='articles_written_set')
+    reviewers = models.ManyToManyField(Scientist, related_name='articles_reviewed_set', db_tablespace='idx_tbsp')
+    class Meta:
+        db_tablespace = 'tbl_tbsp'
+        managed = False

django/db/models/fields/related.py

@@ -988 +988 @@
         'managed': managed,
         'auto_created': klass,
         'app_label': klass._meta.app_label,
+        'db_tablespace': klass._meta.db_tablespace,
         'unique_together': (from_, to),
         'verbose_name': '%(from)s-%(to)s relationship' % {'from': from_, 'to': to},
         'verbose_name_plural': '%(from)s-%(to)s relationships' % {'from': from_, 'to': to},
@@ -996 +997 @@
     return type(name, (models.Model,), {
         'Meta': meta,
         '__module__': klass.__module__,
-        from_: models.ForeignKey(klass, related_name='%s+' % name),
-        to: models.ForeignKey(to_model, related_name='%s+' % name)
+        from_: models.ForeignKey(klass, related_name='%s+' % name, db_tablespace=field.db_tablespace),
+        to: models.ForeignKey(to_model, related_name='%s+' % name, db_tablespace=field.db_tablespace)
     })
 
 class ManyToManyField(RelatedField, Field):

django/db/backends/sqlite3/base.py

@@ -58 +58 @@
     supports_unspecified_pk = True
     supports_1000_query_parameters = False
     supports_mixed_date_datetime_comparisons = False
+    supports_index_tablespaces = False
 
     def _supports_stddev(self):
         """Confirm support for STDDEV and related stats functions
@@ -77 +78 @@
         cursor.execute('DROP TABLE STDDEV_TEST')
         return has_support
 
+    def _supports_tablespaces(self):
+        return False
+
 class DatabaseOperations(BaseDatabaseOperations):
     def date_extract_sql(self, lookup_type, field_name):
         # sqlite doesn't support extract, so we fake it with the user-defined
@@ -129 +133 @@
         # sql_flush() implementations). Just return SQL at this point
         return sql
 
+    def tablespace_sql(self, tablespace, inline=False, index=False):
+        return ''
+
     def year_lookup_bounds(self, value):
         first = '%s-01-01'
         second = '%s-12-31 23:59:59.999999'

django/db/backends/mysql/base.py

@@ -134 +134 @@
     supports_timezones = False
     requires_explicit_null_ordering_when_grouping = True
     allows_primary_key_0 = False
+    supports_index_tablespaces = False
 
     def _can_introspect_foreign_keys(self):
         "Confirm support for introspected foreign keys"
@@ -148 +149 @@
         cursor.execute('DROP TABLE INTROSPECT_TEST')
         return result[1] != 'MyISAM'
 
+    def _supports_tablespaces(self):
+        "Confirm support for tablespaces"
+        return self.connection.get_server_version() >= (5, 1, 6)
+
 class DatabaseOperations(BaseDatabaseOperations):
     compiler_module = "django.db.backends.mysql.compiler"
 
@@ -232 +237 @@
         else:
             return []
 
+    def tablespace_sql(self, tablespace, inline=False, index=False):
+        # MySQL doesn't support setting tablespaces for indexes, and it only
+        # supports setting tablespaces for table since version 5.1.6
+        if inline or index or self.connection.get_server_version() < (5, 1, 6):
+            return ''
+        return super(DatabaseOperations, self).tablespace_sql(tablespace,
+                inline=inline, index=index)
+
     def value_to_db_datetime(self, value):
         if value is None:
             return None

django/db/backends/oracle/base.py

@@ -324 +324 @@
     def start_transaction_sql(self):
         return ''
 
-    def tablespace_sql(self, tablespace, inline=False):
-        return "%sTABLESPACE %s" % ((inline and "USING INDEX " or ""),
-            self.quote_name(tablespace))
-
     def value_to_db_datetime(self, value):
         # Oracle doesn't support tz-aware datetimes
         if getattr(value, 'tzinfo', None) is not None:

django/db/backends/__init__.py

@@ -364 +364 @@
     # date_interval_sql can properly handle mixed Date/DateTime fields and timedeltas
     supports_mixed_date_datetime_comparisons = True
 
+    # Does the backend support tablespaces for indexes?
+    # Requires supports_tablespaces = True.
+    supports_index_tablespaces = True
+
     # Features that need to be confirmed at runtime
+
     # Cache whether the confirmation has been performed.
     _confirmed = False
     supports_transactions = None
     supports_stddev = None
     can_introspect_foreign_keys = None
+    # Does the backend support tablespaces for tables?
+    supports_tablespaces = None
 
     def __init__(self, connection):
         self.connection = connection
@@ -380 +387 @@
         self.supports_transactions = self._supports_transactions()
         self.supports_stddev = self._supports_stddev()
         self.can_introspect_foreign_keys = self._can_introspect_foreign_keys()
+        self.supports_tablespaces = self._supports_tablespaces()
 
     def _supports_transactions(self):
         "Confirm support for transactions"
@@ -410 +418 @@
         # which can't do it for MyISAM tables
         return True
 
+    def _supports_tablespaces(self):
+        "Confirm support for tablespaces"
+        # On MySQL, this depends on the version
+        return True
 
+
 class BaseDatabaseOperations(object):
     """
     This class encapsulates all backend-specific differences, such as the way
@@ -693 +706 @@
             return "ROLLBACK;"
         return "COMMIT;"
 
-    def tablespace_sql(self, tablespace, inline=False):
+    def tablespace_sql(self, tablespace, inline=False, index=False):
         """
-        Returns the SQL that will be appended to tables or rows to define
-        a tablespace. Returns '' if the backend doesn't use tablespaces.
+        Returns the SQL that will be used in a query to define the tablespace.
+
+        Returns '' if the backend doesn't support tablespaces.
+
+        If inline is True, the SQL is appended to a row; otherwise it's appended
+        to the entire statement. If index is True, the statement creates an
+        index; otherwise it creates a table.
+
+        SQL added to a row applies to the index created by this row. When inline
+        is True, it implies that index is True, even if the caller doesn't pass
+        the index parameter.
         """
-        return ''
+        if inline:
+            return "USING INDEX TABLESPACE %s" % self.quote_name(tablespace)
+        else:
+            return "TABLESPACE %s" % self.quote_name(tablespace)
 
     def prep_for_like_query(self, x):
         """Prepares a value for use in a LIKE query."""

django/db/backends/postgresql_psycopg2/creation.py

@@ -44 +44 @@
             db_table = model._meta.db_table
             tablespace = f.db_tablespace or model._meta.db_tablespace
             if tablespace:
-                sql = self.connection.ops.tablespace_sql(tablespace)
-                if sql:
-                    tablespace_sql = ' ' + sql
-                else:
-                    tablespace_sql = ''
+                tablespace_sql = self.connection.ops.tablespace_sql(tablespace, index=True)
+                if tablespace_sql:
+                    tablespace_sql = ' ' + tablespace_sql
             else:
                 tablespace_sql = ''
 

django/db/backends/creation.py

@@ -57 +57 @@
             if tablespace and f.unique:
                 # We must specify the index tablespace inline, because we
                 # won't be generating a CREATE INDEX statement for this field.
-                field_output.append(self.connection.ops.tablespace_sql(tablespace, inline=True))
+                tablespace_sql = self.connection.ops.tablespace_sql(tablespace, inline=True)
+                if tablespace_sql:
+                    field_output.append(tablespace_sql)
             if f.rel:
                 ref_output, pending = self.sql_for_inline_foreign_key_references(f, known_models, style)
                 if pending:
@@ -74 +76 @@
             full_statement.append('    %s%s' % (line, i < len(table_output)-1 and ',' or ''))
         full_statement.append(')')
         if opts.db_tablespace:
-            full_statement.append(self.connection.ops.tablespace_sql(opts.db_tablespace))
+            tablespace_sql = self.connection.ops.tablespace_sql(opts.db_tablespace)
+            if tablespace_sql:
+                full_statement.append(tablespace_sql)
         full_statement.append(';')
         final_output.append('\n'.join(full_statement))
 
@@ -149 +153 @@
             qn = self.connection.ops.quote_name
             tablespace = f.db_tablespace or model._meta.db_tablespace
             if tablespace:
-                sql = self.connection.ops.tablespace_sql(tablespace)
-                if sql:
-                    tablespace_sql = ' ' + sql
-                else:
-                    tablespace_sql = ''
+                tablespace_sql = self.connection.ops.tablespace_sql(tablespace, index=True)
+                if tablespace_sql:
+                    tablespace_sql = ' ' + tablespace_sql
             else:
                 tablespace_sql = ''
             i_name = '%s_%s' % (model._meta.db_table, self._digest(f.column))