Opened 8 years ago

Closed 8 years ago

#5106 closed (fixed)

django.db.backend.*.base refactoring

Reported by: (removed) Owned by: mtredinnick
Component: Database layer (models, ORM) Version: master
Severity: Keywords:
Cc: Triage Stage: Design decision needed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Splitting connection pooling up into two patches; this particular ticket holds refactoring of django.db.backend.*.base.DatabaseWrapper classes to remove redundancy, and give an easier spot to control connection pooling.

Very least, the misc backend implementation are a bit redundant- deriving from a common base class is a bit saner in this case for debugging and extension reasons.

Attachments (3)

backend-refactoring.patch (15.5 KB) - added by (removed) 8 years ago.
backend-refactoring-v2.patch (70.3 KB) - added by (removed) 8 years ago.
version/stage 2 of the refactoring; api breaking, but molds existing old style layout into new automatically, and allows for actual OOP trickery.
backend-refactoring-v2-correct-bitrot.patch (72.2 KB) - added by (removed) 8 years ago.
re-cut v2, since adrians core/management horked v2 from being applied.

Download all attachments as: .zip

Change History (37)

Changed 8 years ago by (removed)

comment:1 Changed 8 years ago by SmileyChris

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

The whole backends area is a bit messy. This is a good start to tidying it up.

I was tempted to push straight to checkin, but I'll let some more eyes look at it before then.

comment:2 Changed 8 years ago by (removed)

via http://groups.google.com/group/django-developers/browse_thread/thread/ab7649e4abd4e589 , might suggest getting the first patch posted into vcs; the second is following and is far more intrusive; actual api changes (no more django.db.backend access primarily), but constricts access to backend capabilities/ops/client/introspection through the wrapper object, thus allowing for far easier/intrusive hooking/override- specifically to just layer connection pooling over a backend, rather then having to write a pooler specific to each backend.

comment:3 follow-up: Changed 8 years ago by anonymous

If you mean that code won't have access to django.db.backend any longer (or shouldn't be accessing it), you will want to start a design discussion on django-dev first. That's a very intrusive change and will break a lot of code, so we need to discuss the change in abstract first before wading through lots of lines of code.

comment:4 Changed 8 years ago by mtredinnick

  • Owner changed from adrian to mtredinnick

Oops... last comment was me, by the way.

I'm generally +1 on the current patch, since it looks clean. Still want to look at it once or twice more, but it'll go in soon, I would expect.

Changed 8 years ago by (removed)

version/stage 2 of the refactoring; api breaking, but molds existing old style layout into new automatically, and allows for actual OOP trickery.

comment:5 in reply to: ↑ 3 Changed 8 years ago by (removed)

Replying to mtredinnick:

If you mean that code won't have access to django.db.backend any longer (or shouldn't be accessing it), you will want to start a design discussion on django-dev first. That's a very intrusive change and will break a lot of code, so we need to discuss the change in abstract first before wading through lots of lines of code.

Still has access to it, just has access to a compartmentalized version of it. Already converted existing code over to it w/in django; atm, getting v2 up in the tracker (upstream for the patch is bzr branch at http://pkgcore.org/~ferringb/django/connection-pooling) prior to kicking off the discussion; it's worth noting that with a rather simple getattr, it's possible to preserve the old api (preferably firing off a warning on access).

comment:6 Changed 8 years ago by George Vilches <gav@…>

Since we don't have an official voting system yet, just adding a comment that I'm +1 for this patch (the first or the second, I'll be happy with either), it would make the SQL logging additions much cleaner.

Changed 8 years ago by (removed)

re-cut v2, since adrians core/management horked v2 from being applied.

comment:7 Changed 8 years ago by adrian

I'm inclined to check in the first patch, as it's a simpler / more atomic change, and I'd rather not check in substantial patches like the second patch in a single commit (mostly because they're hard to follow). I'm on it.

comment:8 Changed 8 years ago by adrian

(In [5949]) Refactored all database backends to inherit from a common base class to remove quite a bit of duplicated code. Thanks for the patch, Brian Harring. Refs #5106

comment:9 Changed 8 years ago by adrian

First patch is checked into trunk. Working on the second patch now; I'm going to split it over several commits, for sanity's sake.

comment:10 Changed 8 years ago by adrian

Continued refactoring in [5950].

comment:11 Changed 8 years ago by adrian

(In [5951]) Refactored get_date_extract_sql() to DatabaseOperations.date_extract_sql(). Refs #5106

comment:12 Changed 8 years ago by adrian

(In [5952]) Refactored get_date_trunc_sql() to DatabaseOperations.date_trunc_sql(). Refs #5106

comment:13 Changed 8 years ago by adrian

(In [5953]) Refactored get_datetime_cast_sql() to DatabaseOperations.datetime_cast_sql(). Refs #5106

comment:14 Changed 8 years ago by adrian

(In [5955]) Refactored get_deferrable_sql() to DatabaseOperations.deferrable_sql(). Refs #5106

comment:15 Changed 8 years ago by adrian

(In [5956]) Refactored get_drop_foreignkey_sql() to DatabaseOperations.drop_foreignkey_sql(). Refs #5106

comment:16 Changed 8 years ago by adrian

(In [5957]) Refactored get_fulltext_search_sql() to DatabaseOperations.fulltext_search_sql(). Refs #5106

comment:17 Changed 8 years ago by adrian

(In [5958]) Refactored get_last_insert_id() to DatabaseOperations.last_insert_id(). Refs #5106

comment:18 Changed 8 years ago by adrian

(In [5959]) Refactored get_limit_offset_sql() to DatabaseOperations.limit_offset_sql(). Refs #5106

comment:19 Changed 8 years ago by adrian

(In [5960]) Refactored get_max_name_length() to DatabaseOperations.max_name_length(). Refs #5106

comment:20 Changed 8 years ago by adrian

(In [5961]) Refactored get_pk_default_value() to DatabaseOperations.pk_default_value(). Refs #5106

comment:21 Changed 8 years ago by adrian

(In [5962]) Refactored get_random_function_sql() to DatabaseOperations.random_function_sql(). Refs #5106

comment:22 Changed 8 years ago by adrian

(In [5963]) Refactored get_sql_flush() to DatabaseOperations.sql_flush(). Refs #5106

comment:23 Changed 8 years ago by adrian

(In [5964]) Refactored get_sql_sequence_reset() to DatabaseOperations.sequence_reset_sql(). Refs #5106

comment:24 Changed 8 years ago by adrian

(In [5965]) Refactored get_start_transaction_sql() to DatabaseOperations.start_transaction_sql(). Refs #5106

comment:25 Changed 8 years ago by adrian

(In [5966]) Refactored get_tablespace_sql() to DatabaseOperations.tablespace_sql(). Refs #5106

comment:26 Changed 8 years ago by adrian

(In [5967]) Refactored quote_name() to DatabaseOperations.quote_name(). Refs #5106

comment:27 Changed 8 years ago by adrian

(In [5972]) Moved postgresql backend DatabaseOperations class into a new module, postgresql/operations.py, so that it can be imported by both the postgresql and postgresql_psycopg2 backends. Hence the two backends no longer have a duplicated DatabaseOperations class

comment:28 Changed 8 years ago by adrian

(In [5974]) Implemented BaseDatabaseFeatures and changed all code to access it -- connection.features.foo instead of backend.foo

comment:29 Changed 8 years ago by adrian

Note that I used "features" instead of "capabilities" because the latter just takes too long to type.

comment:30 Changed 8 years ago by adrian

(In [5976]) Refactored get_query_set_class() to DatabaseOperations.query_set_class(). Also added BaseDatabaseFeatures.uses_custom_queryset. Refs #5106

comment:31 Changed 8 years ago by adrian

(In [5977]) Refactored get_field_cast_sql() to DatabaseOperations.field_cast_sql(). Refs #5106

comment:32 Changed 8 years ago by adrian

(In [5978]) Refactored get_drop_sequence() to DatabaseOperations.drop_sequence_sql(). Refs #5106

comment:33 Changed 8 years ago by adrian

(In [5982]) Refactored OPERATOR_MAPPING so that it exists as django.db.connection.operators instead of django.db.backend.OPERATOR_MAPPING. Refs #5106

comment:34 Changed 8 years ago by adrian

  • Resolution set to fixed
  • Status changed from new to closed

OK, I think we can safely mark this as fixed. I've made 95% of the changes from the patch, with the exception of making DatabaseError, IntegrityError and the introspection and creation functionality accessible via django.db.connection object. Those changes can happen in the near future, if they need to be made. (Please open separate tickets w/ patches for those changes if you think they're worth doing.)

I ended up making small naming/implementation changes in places throughout this fix, but Brian's core design for this refactoring remained intact. Thanks for the great suggestions and patch, Brian!

Note: See TracTickets for help on using tickets.
Back to Top