An Open Source Column-Store Backend

[anonymous]

Adding a ​column-store database backend would, transparently, allow Django to become applicable to a wider array of problems. Column-stores are being ​touted by some leading pioneers in the database community as a solution to some of the prevalent problems in database management specifically exacerbated by row-based storage (i.e. MySQL, postgrs, Oracle, etc.) of large sets.

The shortcomings of column-store backend are both general and specific:

• Column stores are less efficient at inserts and updates due to their aggressive compression and indexing.
• The below implementations are not in the usual palette of database solutions provided by shared web hosts, nor have most web developers paid much attention to the ​VLDB world.
• Little legacy data is stored in databases of this type, reducing the effectiveness of inspectdb, one of the killer Django hooks.

[anonymous]

Several open source column store databases are available. An alphabetical selection of data storage solutions this observer has investigated for a problem to solve with Django:

• ​FastBit is implemented in C++, has no python interface I can find and kinda supports some sort of SQL.
• ​HBase is a BigTable clone, is implemented in Java, sits atop Hadoop, and is aimed at large-scale distributed applications. It has jython support and does not have an SQL implementation.
• ​HyperTable is another BigTable clone, and is implemented in C++. Python support is planned and supports something like SQL.
• ​LucidDB is implemented in java, and has no python interface I can find, though it does support the SQL2003 standard.
• ​Metakit appears to have stagnated despite use in production applications, has a python library but does not support an SQL standard.
• ​MonetDB/SQL is implemented in C and boasts a python library based on MAPI. It supports the SQL99 and SQL2003 standards.

As the first several seem to be aimed more at special purposes (research, embedded, distributed) databases (and therefore applications), MonetDB appears to this observer as the most likely candidate for further investigation as a backend; the python interface (and its similarity to MySQLdb) is icing.

[James Bennett]

The general pattern for proposed database backends is for them to be developed independently as third-party projects; they don't need to be shipped with Django itself (since Django can use anything on your Python path).