Add IntervalField to database models

[ben.tucker@…]

Although not all backends support the interval data type, it can be implemented using some other data type. This field is quite useful and it would be nice if it were available.

[Simon G. <dev@…>]

I'm tentatively marking this as wontfix, as surely a combination of a "start-interval" and "end-interval" fields is better normalisation.

Feel free to correct me if I'm wrong though!

[anonymous]

start-interval and end-interval doesn't work for all situations. Suppose you want to track "Amount of time spent driving in the past week" Unless I'm misunderstanding your use of start-interval and end-interval, there isn't an easy way to store this value with date/time fields. The real bonus of intervals comes in when you are able to do computations with them (such as "Amount of time spent driving in May" + "Amount of time spent driving in June")

[Gary Wilson <gary.wilson@…>]

patches are welcome.

[Marty Alchin <gulopine@…>]

DurationField

[Marty Alchin <gulopine@…>]

This simple patch adds a DurationField, which, in my opinion, better encapsulates the use cases described by anonymous above. Calling it a duration implies that it's not tied to any specific points in time, but rather just the passage of a certain amount of time. This makes more sense for things like the length (duration!) of audio and video content, for instance.

The patch itself is fairly simple, based on a FloatField, whose value represents the number of seconds to be represented. This seems sufficient for database storage, since all supported backends already support the equivalent of a FloatField, while max_digits and decimal_places allow for a number large enough to represent timedelta.max, complete with a full set of microseconds. It doesn't take calendars into account at all, but neither does timedelta, which is also in keeping with the duration concept as opposed to intervals.

The one problem I have with it so far is that it seems like Django is relying on backend database connection modules to handle coercion into Python types, rather than using to_python, so the value isn't changed into a timedelta unless you manually call .validate() on the model. This also prevents the validation errors from occurring properly when editing in the admin. That may be another issue for another day, however. I've marked the issue as "Patch needs improvement" in case I'm missing something that would make this happen.

P.S. Pardon the WikiFormatting on the attachment comment. That is, unless it gets a wiki page, in which case it'll be fine.

[Marty Alchin <gulopine@…>]

I should also add, under the heading of "Patch needs improvement," that typing in decimals as a float value is hardly intuitive, and a better UI should be used. I, unfortunately, haven't thought of a good way to visually represent it, so I don't have any suggestions.

[Marty Alchin <gulopine@…>]

Updated DurationField location to actually work (I hadn't tested the other one)

[Marty Alchin <gulopine@…>]

Complete patch with widget, supporting oldforms and newforms, including newforms-admin

[Marty Alchin <gulopine@…>]

This most recent patch covers many more aspects of DurationField, including a newforms field and widget, fully compatible with newforms-admin. This new widget implements a bit better of a UI, separating the value into its components (days, hours, minutes, seconds, microseconds).

This new widget can't reasonably be done with oldforms, and since it's going out anyway, oldforms (and the current admin) will simply use a FloatField describing the number of seconds (including microseconds).

Also, this version of the patch relies on #3982 to enable support of the coerce method.

[Marty Alchin <gulopine@…>]

New patch, complete with documentation

[Marty Alchin <gulopine@…>]

Once again, this patch supercedes all others, and once again relies on a new patch from #3982. However, it now uses lazy instantiation of timedelta objects, which should improve performance. Documentation is also included for trunk, with a few minor changes needed for the newforms-admin branch (due to the newforms widget included in the patch). Tests will come shortly, once I have a few other patches ironed out.

[Marty Alchin]

Updated to work with latest trunk, and to no longer rely on #3982

[Marty Alchin]

This latest patch is missing a few things before it can be considered final, but it's getting close. It no longer relies on #3982, but it doesn't yet support the recent error_messages stuff, and fails with the JSON serializer. It works with the XML serializer though, and should work in newforms-admin as is.

[Marty Alchin]

Fixed a problem with creating objects by hand, and removed a couple debug lines

[Stefan Foulis]

I've had some cases (mainly in the admin-interface) where get_db_prep_save(self, value) gets called with a value of type Decimal. I don't know why that happens, but I added this code to fix DurationField in these circumstances:

def get_db_prep_save(self, value):
    if value is None:
        return None
    if (type(value) == decimal.Decimal) or (type(value)==float):
        return str(value)
    return str(value.days * 24 * 3600 + value.seconds + float(value.microseconds) / 1000000)

If this is the correct way to fix this, please add it to the next diff you make.

[Marty Alchin]

I'm more interested to know why it's getting a Decimal, because that would probably break other expectations as well. Do you have any code that reliably shows that to happen? If you don't know exactly where it's going wrong, that's fine, just being able to reproduce it will help me figure it out.

[Stefan Foulis]

OK... I did some testing and was able to reproduce the condition. In a simple Model with a DurationField create a record in Admin and enter 0 as the value. django-admin will throw a AttributeError: 'Decimal' object has no attribute 'days' and will prevent saving the value.
If you enter 0 directly into the database or save a datetime.timedelta(0) from the django shell (which works, unlike in the admin) that attribute will return a Decimal("0") instead of the timedelta object (once you re-fetch it from the database).
So my proposed patch to get_db_prep_save is the wrong place to fix this problem. It seems this condition is rooted where the data is converted from to database to the native python object.
Could this have something to do with the backend driver doing the coercion to python types that was reported in #3982?
I've been testing this with sqlite3 btw.

[Stefan Foulis]

In DurationProxy.__get__ the value that instance.__dict__[self.field_name] returns is a Decimal object (if the value in the database is 0). With other values it is a TimeDelta

[Jacob]

This needs some UI help, but it's otherwise perfect.

[RahmCoff+dj@…]

Is there any plan to use PostgreSQL’s Interval type with this?

[Marty Alchin]

I have no plans to support interval, since DecimalField allows for compatibility with all of Django's backends. The only real reason to support interval is for inspecting existing databases that use it. If you'd like support for that, feel free to add in whatever code it would take to do it. I'm just too bogged down with other things at the moment, and supporting interval isn't high on my priority list.

[Rob Hudson <treborhudson@…>]

I tried to apply this patch and it no longer applies cleanly. I believe django/newforms/fields.py and django/newforms/widgets.py had a reject and some fuzz.

When I did get it applied I was getting some errors when I went into the admin and tried to add data to my model...

    Traceback:
    File ".../django_trunk/django/contrib/admin/views/main.py" in add_stage
      287.         new_data = manipulator.flatten_data()
    Exception Value: 'str' object has no attribute 'items'

[Thomas Güttler]

Related: ​http://www.djangosnippets.org/snippets/1060/

[Jerome Leclanche]

Updated patch for current trunk, without docs (no idea how much they've changed)

[Jerome Leclanche]

What's holding DurationField from going upstream (at least in contrib)? I can help fix whatever is necessary.

[Marty Alchin]

Well, you'll notice that yours is the first patch to hit this ticket since Jacob said it "needs some UI help." That criticism still stands, since a half-dozen input fields don't make for a very good user experience.

[Jerome Leclanche]

I had a look through the patch today. Using a DecimalField and playing with commas seems unreasonably hacky. Is using a 64-bit int an option? It shouldn't be a problem under postgres, but I don't know about the other db backends.

[Jerome Leclanche]

For the interface part, there's two decent choices.
Either go for a parse of the values in a single field, for example (input) "3d 12h 4m 7s", "7d 10m", "150000 ms"; output would be highest value possible first - 60000ms would be "1m", 75000ms would be "1m 15s" (abbreviations are up for change, would go for the standard ones).

Or, go for a single field with a <select> box, with some js shortcuts. [_] [ Unit ... ] (X)

In both cases, the possible units would be years, months, weeks, days, hours, minutes, seconds, milliseconds, microseconds (assuming microsecond precision). Thoughts?

[Marty Alchin]

I'm a bit confused as to what you mean by "playing with commas," but thus far, I don't see anything at all hacky with using a DecimalField. Going with a 64-bit int would only be an option if Django supports a standalone cross-database 64-bit integer field. I don't want to invent one just for this. As for being hacky, I find it extremely elegant that the current approach stores seconds in the database, allowing for super-simple conversion: value = datetime.timedelta(seconds=float(value)). Plus, it works with all existing backends and maintains the ability to order records as you would expect (though, admittedly, your 64-bit int would also preserve that feature). What's exactly is hacky about it?

On the interface front, there was (once upon a time) a prevailing notion of using two text boxes: one for days and one for everything else. Basically, the days would be cleaned as a standard integer field, while the other would be cleaned as a time field, so the syntax for stuff less than a day would follow the "hh:mm:ss" format. The only thing that held me back from actually implementing that was that it would also be great to have a JavaScript helper similar to that of TimeField, showing a few useful options to get you started. Something like: 30 seconds, 2 hours, 1 day, 6 weeks (or some such similar options).

I obviously haven't gotten back around to this ticket in quite some time, but I'll see if I can devote some time to it soon to get the original UI proposal up and running.

[Jerome Leclanche]

It feels wrong to "store" as seconds. Then again, just my opinion.

I was thinking the same thing about a js helper, and a friend came up with the idea of just parsing the field for units. I'll see if I can get a few mins to work on it tonight, I really like the idea. Three big advantages:

• No need for javascript (6 weeks would be "6w", 30 seconds would be "30s", 2 hours would be "2h", etc)
  
• Easy to understand, very user friendly (parsing "3h 10d" is okay, I suppose we'd give validationerror on "1h 10m 2h" or similar tho)
  
• No need for multiple input boxes, avoids confusion
  

On a slightly-related note, I don't know what to do about microseconds. Standard display is µs, but that could cause problems for users with special keyboard layouts etc.

[Michał Sałaban]

I think the year and month units are misleading, as in real life they have no fixed length. This is probably why the timedelta type does accept day as the longest unit possible.

We can assume that year is an astronomical unit, but it's almost never useful in real life. A year has to round to days and it's length is known only when applied to a certain start date. The same (or even worse) happens with a month. In Gregorian calendar there are 4 different lengths possible: 28, 29, 30, 31 days. We could assume that month is astronomical unit again (moon cycle) but it will roughly match only the Muslim calendar.

I'd prefer to leave a week as the longest unit possible.

[Jerome Leclanche]

Emes, I agree with you. I just wanted to get the implementation in. It's easier to remove than to add-on.

I sent the Dev list a mail a couple of days ago: ​http://groups.google.com/group/django-developers/browse_thread/thread/612e6c641d109f40?hl=en

Another question I bumped into this morning: Should we allow negative values?

[Jerome Leclanche]

cc michal@…, sorry for the spam, Firefox is horrible.

[Jerome Leclanche]

New DurationField implementation (try 3)

[Jerome Leclanche]

Any feedback on the DurationField patch? I still need to figure out the admin output.

[Jerome Leclanche]

Fully working DurationField implementation with changes and fixes from Yuri Baburov

[Jerome Leclanche]

Yuri did some updates to the patch posted above; I'll post a rebase by tomorrow. This rebase also comments out support for months and years as per list discussion; since it's just two values, might as well leave them commented.

1.1's almost out if I am to trust the targeted bugs. I suppose we could start looking at DurationField implementation for 1.2? This ticket has been open long enough.

Reassigning to myself.

[Jerome Leclanche]

Ok, 1.1 is out, it's time for an update.
This ticket has been open long enough. I'm decided to get everything closed for 1.2. I'm going to upload a rebased patch today or tomorrow. Right now, there are three outstanding issues with it:

• In the admin, a durationfield appears as a Decimal's unicode in filters, and when it's equal to zero on the admin detail page.
  
• Validation isn't getting handled correctly in the admin - a 500 is raised rather than a ValidationError.
  
• Lack of docs/tests
  

First issue is something I've tried to fix but came up blank. Second one should be easy enough to fix (I'll have a look before rebasing hopefully), third one I'll leave until the former two are fixed.
I'm still *very* itchy about using a DecimalField as backend. But hey...

[Jerome Leclanche]

Updated patch using the new BigIntegerField as backend

[Jerome Leclanche]

There, new updated patch.
See: ​http://groups.google.com/group/django-developers/browse_thread/thread/d32316ce7964dbaf?hl=en

Unless there are bugs, I will not work on this anymore. The core devs seem to hate the idea of having a DurationField... meanwhile I'm perfectly happy keeping my core patched.

[Russell Keith-Magee]

Replying to Adys:

Unless there are bugs, I will not work on this anymore. The core devs seem to hate the idea of having a DurationField...

Yes, the core devs clearly hate the idea of DurationField. That's why this ticket was marked accepted by Jacob, and is on the low prioirty feature list for v1.2.

There is a difference between "don't like an idea" and "haven't had time to apply a patch because we've been busy with one or two other things". If we didn't like the idea, we'd close the ticket.

meanwhile I'm perfectly happy keeping my core patched.

I'll also point out that you don't need to patch core to write your own field definition. You can define fields in your own code base, and import/use them as you wish.

[Sebastian Żurek]

Adys, I liked the DurationField idea and I hope You were not seorious saying that You're going to leave it? I've tried it in my my project - there are some issues with it which I'm going to address soon - but all in all it works OK and I will be happy to help maintain it.

It would be great to see it in django itself in the future. Up to this time, I must agree with russellm that it would be easier to use/test it 'externally'. Right now, I had to create another django version (along side with my default system one from Gentoo) and patch it to use DurationField... I want to stay in-sync with Your code - that's why I didn't extract it.

Please, consider such an extraction, I belive it would made the DurationField more attractive (while it's not in django).

[Alex Gaynor]

We are now passed the 1.2 feature freeze.

[John Paulett]

I am not sure if this is appropriate, but I took the latest patch by Adys and turned it into a reusable application, ​django-durationfield. If anyone has an issue with turning the patch into an external app, please let me know.

I am viewing the django-durationfield app as a temporary solution since this patched missed the 1.2 feature deadline. I will try to keep the app in sync with latest patch for this issue (and contribute any fixes, documentation, and tests to this issue).

Assuming that a DurationField is eventually committed to Django core, the switch from the django-durationfield app to the core version should just be a matter of switching a few import statements.

[James Bennett]

1.3 is feature-frozen now.

[Karen Rustad]

This is a vote to get a DurationField (or whatever you want to call it) into Django 1.4. I'm currently running into this problem in that I need to represent an iteration length in some task management software I'm writing, where the length is set by the user and could be anything from a week to a month to a couple months (even more hilarity since months aren't consistent lengths). Of course, timedeltas are super-conveniently built into Python and can handle all of this in a snap...but in Django I have no way to store them. It's driving me nuts.

Like, come on. If Django is Python, timedelta needs a model field. Without, it's incomplete.

[John Paulett]

Karen,

While Python does provide the timedelta class, there is no common SQL implementation for storing that object in the database. PostgreSQL does provide the INTERVAL data type, MySQL and SQLite do not have any interval data type (I am not sure about Oracle, MSSQL, etc.).

Luckily Django makes it relatively simple to create ​custom model fields. If you are willing to be PostgreSQL-specific, you could store timedeltas as INTERVALs (I believe the psycopg driver handles the conversion for you).

You could also convievably pickle the timedelta instance and store that blob in the database. This approach breaks the sorting semantics, potentially corrupts the data across Python versions, and is completely opaque from a pure database perspective.

Another approach is to store the data in the database as a BIGINT. This approach has the advantage of working across databases and retains the the expected sorting semantics. The downside to this approach from a pure database perspective, is that you now must know the function to convert from a BIGINT into an interval. Based upon the patches included in this ticket, myself and several other developers created ​django-durationfield. One advantage of being a standalone app as opposed to being in core is that we can release new features & bug fixes without having to wait for a major Django point release.

I do not mean to speak for the core developers, but since there is no ideal solution and since using a standalone app is extremely easy, I doubt we will see this feature included in core anytime soon.

[smcoll]

Replying to jpaulett:

While Python does provide the timedelta class, there is no common SQL implementation for storing that object in the database. PostgreSQL does provide the INTERVAL data type, MySQL and SQLite do not have any interval data type (I am not sure about Oracle, MSSQL, etc.).

What about MySQL's TIME type?

"MySQL retrieves and displays TIME values in 'HH:MM:SS' format (or 'HHH:MM:SS' format for large hours values). TIME values may range from '-838:59:59' to '838:59:59'. The hours part may be so large because the TIME type can be used not only to represent a time of day (which must be less than 24 hours), but also elapsed time or a time interval between two events (which may be much greater than 24 hours, or even negative)."

​http://dev.mysql.com/doc/refman/5.0/en/time.html

[Kal Sze]

I think this ticket is mostly moot now that there is django-timedeltafield, which works quite nicely. See links below:

​https://pypi.python.org/pypi/django-timedeltafield
​https://bitbucket.org/schinckel/django-timedelta-field/

[Tim Graham]

​WIP from Marc.

[Anssi Kääriäinen]

I think we need support for qs.filter(event_startdate__gte=datetime.now() - F('event_duration)). I think this works on PostgreSQL, but at least on Oracle it produces wrong results. The event_duration is in microseconds, but in oracle the date arithmetic interprets integers as days. Actually, I am not sure if this query works at all.

In any case usages of F('duration_field') need tests. Raising errors from F() usage is acceptable, but producing wrong results is clearly not OK.

Another issue that keeps nagging me is the use of SubfieldBase to support value conversion (SubfieldBase basically calls to_python() anytime the field is set on the object). Other fields in Django core do not use this, instead they let the backend convert the values. I am not sure if usage of SubfieldBase leads to any problems, but I keep getting the feeling we should be consistent in not using SubfieldBase. One case where usage of backend specific converters differ from usage of to_python + SubfieldBase is that SubfieldBase does value conversions on assignment. So:

   >>> obj.duration_field = '12:34:56'
   >>> obj.duration_field
   OUT: '12:34:56' if not using SubfieldBase
   OUT: timedelta(12h, 34m, 56s)

Currently no local field in Django core do value conversions on assignment. In addition there is clear overhead in model __init__ when using SubfieldBase, I'd guess the overhead is somewhere around 20% per field that uses SubfieldBase, though I haven't benchmarked this.

[Marc Tamlyn]

Changes have been made to:

• remove SubFieldBase (completely!)
• Support arithmetic with DurationField

I believe this is now ready for checkin, I just need a final check from someone for the arithmetic changes.

[Tim Graham]

There's a test failure that needs fixing.

[Marc Tamlyn <marc.tamlyn@…>]

In 57554442fe3e209c135e15dda4ea45123e579e58:

Fixed #2443 -- Added DurationField.

A field for storing periods of time - modeled in Python by timedelta. It
is stored in the native interval data type on PostgreSQL and as a bigint
of microseconds on other backends.

Also includes significant changes to the internals of time related maths
in expressions, including the removal of DateModifierNode.

Thanks to Tim and Josh in particular for reviews.

[Marc Tamlyn <marc.tamlyn@…>]

In 07728a2c2c78b030b7de9c6b20ef028f82d6134a:

Refs #2443 -- Fix Oracle tests for DurationField.

It helps if there are the correct number of microseconds in a second.

[Marc Tamlyn <marc.tamlyn@…>]

In 962bb9b6bd1deb042e139316108ec0da44cb22a9:

Refs #2443 -- Move the durationfield converter logic.

This reduces how frequently this logic is run significantly.

Thanks to Anssi for the suggestion.

[Tim Graham <timograham@…>]

In 227a4c48be18a439f0a08a49bf1f3062babeee82:

Refs #2443 -- fixed format_dtdelta on SQLite

A test failed on Python 2 32-bit.

[Tim Graham <timograham@…>]

In ee86e590516250a7db7bf7c9a979ced4faf4edd7:

Refs #2443 -- Allowed creation of objects with NULL DurationFields