Code

Opened 8 years ago

Last modified 8 months ago

#2443 new New feature

Add IntervalField to database models

Reported by: ben.tucker@… Owned by: Adys
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: IntervalField interval duration DurationField feature
Cc: jm.bugtracking@…, treborhudson@…, patrick.lauber@…, hv@…, michal@…, john@…, flavio.curella@…, slafs, fckin.spam@…, evgenpioneer@…, sorin, mike@…, net147, aaugustin, styne666 Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: yes
Easy pickings: no UI/UX: yes

Description

Although not all backends support the interval data type, it can be implemented using some other data type. This field is quite useful and it would be nice if it were available.

Attachments (10)

durationfield.diff (1009 bytes) - added by Marty Alchin <gulopine@…> 7 years ago.
DurationField
durationfield.2.diff (1.1 KB) - added by Marty Alchin <gulopine@…> 7 years ago.
Updated DurationField location to actually work (I hadn't tested the other one)
durationfield.3.diff (5.4 KB) - added by Marty Alchin <gulopine@…> 7 years ago.
Complete patch with widget, supporting oldforms and newforms, including newforms-admin
durationfield.4.diff (6.5 KB) - added by Marty Alchin <gulopine@…> 7 years ago.
New patch, complete with documentation
durationfield.5.diff (9.0 KB) - added by Gulopine 7 years ago.
Updated to work with latest trunk, and to no longer rely on #3982
durationfield.6.diff (8.9 KB) - added by Gulopine 7 years ago.
Fixed a problem with creating objects by hand, and removed a couple debug lines
durationfield.7.diff (6.9 KB) - added by Adys 5 years ago.
Updated patch for current trunk, without docs (no idea how much they've changed)
durationfield-new.diff (6.7 KB) - added by Adys 5 years ago.
New DurationField implementation (try 3)
durationfield-new.2.diff (6.4 KB) - added by Adys 5 years ago.
Fully working DurationField implementation with changes and fixes from Yuri Baburov
durationfield.patch (9.8 KB) - added by Adys 4 years ago.
Updated patch using the new BigIntegerField as backend

Download all attachments as: .zip

Change History (65)

comment:1 Changed 8 years ago by adrian

  • priority changed from normal to lowest
  • Severity changed from normal to minor

comment:2 Changed 7 years ago by Simon G. <dev@…>

  • Resolution set to wontfix
  • Status changed from new to closed

I'm tentatively marking this as wontfix, as surely a combination of a "start-interval" and "end-interval" fields is better normalisation.

Feel free to correct me if I'm wrong though!

comment:3 Changed 7 years ago by anonymous

  • Resolution wontfix deleted
  • Status changed from closed to reopened

start-interval and end-interval doesn't work for all situations. Suppose you want to track "Amount of time spent driving in the past week" Unless I'm misunderstanding your use of start-interval and end-interval, there isn't an easy way to store this value with date/time fields. The real bonus of intervals comes in when you are able to do computations with them (such as "Amount of time spent driving in May" + "Amount of time spent driving in June")

comment:4 Changed 7 years ago by Gary Wilson <gary.wilson@…>

  • Triage Stage changed from Unreviewed to Design decision needed

patches are welcome.

Changed 7 years ago by Marty Alchin <gulopine@…>

comment:5 Changed 7 years ago by Marty Alchin <gulopine@…>

  • Has patch set
  • Keywords DurationField added
  • Needs tests set
  • Patch needs improvement set

This simple patch adds a DurationField, which, in my opinion, better encapsulates the use cases described by anonymous above. Calling it a duration implies that it's not tied to any specific points in time, but rather just the passage of a certain amount of time. This makes more sense for things like the length (duration!) of audio and video content, for instance.

The patch itself is fairly simple, based on a FloatField, whose value represents the number of seconds to be represented. This seems sufficient for database storage, since all supported backends already support the equivalent of a FloatField, while max_digits and decimal_places allow for a number large enough to represent timedelta.max, complete with a full set of microseconds. It doesn't take calendars into account at all, but neither does timedelta, which is also in keeping with the duration concept as opposed to intervals.

The one problem I have with it so far is that it seems like Django is relying on backend database connection modules to handle coercion into Python types, rather than using to_python, so the value isn't changed into a timedelta unless you manually call .validate() on the model. This also prevents the validation errors from occurring properly when editing in the admin. That may be another issue for another day, however. I've marked the issue as "Patch needs improvement" in case I'm missing something that would make this happen.

P.S. Pardon the WikiFormatting on the attachment comment. That is, unless it gets a wiki page, in which case it'll be fine.

comment:6 Changed 7 years ago by Marty Alchin <gulopine@…>

I should also add, under the heading of "Patch needs improvement," that typing in decimals as a float value is hardly intuitive, and a better UI should be used. I, unfortunately, haven't thought of a good way to visually represent it, so I don't have any suggestions.

Changed 7 years ago by Marty Alchin <gulopine@…>

Updated DurationField location to actually work (I hadn't tested the other one)

Changed 7 years ago by Marty Alchin <gulopine@…>

Complete patch with widget, supporting oldforms and newforms, including newforms-admin

comment:7 Changed 7 years ago by Marty Alchin <gulopine@…>

  • Needs documentation set
  • Patch needs improvement unset

This most recent patch covers many more aspects of DurationField, including a newforms field and widget, fully compatible with newforms-admin. This new widget implements a bit better of a UI, separating the value into its components (days, hours, minutes, seconds, microseconds).

This new widget can't reasonably be done with oldforms, and since it's going out anyway, oldforms (and the current admin) will simply use a FloatField describing the number of seconds (including microseconds).

Also, this version of the patch relies on #3982 to enable support of the coerce method.

comment:8 Changed 7 years ago by anonymous

  • Cc jm.bugtracking@… added

Changed 7 years ago by Marty Alchin <gulopine@…>

New patch, complete with documentation

comment:9 Changed 7 years ago by Marty Alchin <gulopine@…>

  • Needs documentation unset

Once again, this patch supercedes all others, and once again relies on a new patch from #3982. However, it now uses lazy instantiation of timedelta objects, which should improve performance. Documentation is also included for trunk, with a few minor changes needed for the newforms-admin branch (due to the newforms widget included in the patch). Tests will come shortly, once I have a few other patches ironed out.

Changed 7 years ago by Gulopine

Updated to work with latest trunk, and to no longer rely on #3982

comment:10 Changed 7 years ago by Gulopine

  • Keywords duration added
  • Owner changed from nobody to Gulopine
  • Patch needs improvement set
  • Status changed from reopened to new

This latest patch is missing a few things before it can be considered final, but it's getting close. It no longer relies on #3982, but it doesn't yet support the recent error_messages stuff, and fails with the JSON serializer. It works with the XML serializer though, and should work in newforms-admin as is.

Changed 7 years ago by Gulopine

Fixed a problem with creating objects by hand, and removed a couple debug lines

comment:11 Changed 7 years ago by stefanfoulis

I've had some cases (mainly in the admin-interface) where get_db_prep_save(self, value) gets called with a value of type Decimal. I don't know why that happens, but I added this code to fix DurationField in these circumstances:

def get_db_prep_save(self, value):
    if value is None:
        return None
    if (type(value) == decimal.Decimal) or (type(value)==float):
        return str(value)
    return str(value.days * 24 * 3600 + value.seconds + float(value.microseconds) / 1000000)

If this is the correct way to fix this, please add it to the next diff you make.

comment:12 Changed 7 years ago by Gulopine

I'm more interested to know why it's getting a Decimal, because that would probably break other expectations as well. Do you have any code that reliably shows that to happen? If you don't know exactly where it's going wrong, that's fine, just being able to reproduce it will help me figure it out.

comment:13 Changed 7 years ago by stefanfoulis

OK... I did some testing and was able to reproduce the condition. In a simple Model with a DurationField create a record in Admin and enter 0 as the value. django-admin will throw a AttributeError: 'Decimal' object has no attribute 'days' and will prevent saving the value.
If you enter 0 directly into the database or save a datetime.timedelta(0) from the django shell (which works, unlike in the admin) that attribute will return a Decimal("0") instead of the timedelta object (once you re-fetch it from the database).
So my proposed patch to get_db_prep_save is the wrong place to fix this problem. It seems this condition is rooted where the data is converted from to database to the native python object.
Could this have something to do with the backend driver doing the coercion to python types that was reported in #3982?
I've been testing this with sqlite3 btw.

comment:14 Changed 7 years ago by stefanfoulis

In DurationProxy.__get__ the value that instance.__dict__[self.field_name] returns is a Decimal object (if the value in the database is 0). With other values it is a TimeDelta

comment:15 Changed 7 years ago by jacob

  • Triage Stage changed from Design decision needed to Accepted

This needs some UI help, but it's otherwise perfect.

comment:16 Changed 6 years ago by RahmCoff+dj@…

Is there any plan to use PostgreSQL’s Interval type with this?

comment:17 Changed 6 years ago by Gulopine

I have no plans to support interval, since DecimalField allows for compatibility with all of Django's backends. The only real reason to support interval is for inspecting existing databases that use it. If you'd like support for that, feel free to add in whatever code it would take to do it. I'm just too bogged down with other things at the moment, and supporting interval isn't high on my priority list.

comment:18 Changed 6 years ago by Rob Hudson <treborhudson@…>

  • Cc treborhudson@… added

I tried to apply this patch and it no longer applies cleanly. I believe django/newforms/fields.py and django/newforms/widgets.py had a reject and some fuzz.

When I did get it applied I was getting some errors when I went into the admin and tried to add data to my model...

    Traceback:
    File ".../django_trunk/django/contrib/admin/views/main.py" in add_stage
      287.         new_data = manipulator.flatten_data()
    Exception Value: 'str' object has no attribute 'items'

comment:19 Changed 6 years ago by Patrick Lauber <patrick.lauber@…>

  • Cc patrick.lauber@… added

comment:20 Changed 6 years ago by guettli

  • Cc hv@… added

Changed 5 years ago by Adys

Updated patch for current trunk, without docs (no idea how much they've changed)

comment:22 Changed 5 years ago by Adys

What's holding DurationField from going upstream (at least in contrib)? I can help fix whatever is necessary.

comment:23 Changed 5 years ago by Gulopine

Well, you'll notice that yours is the first patch to hit this ticket since Jacob said it "needs some UI help." That criticism still stands, since a half-dozen input fields don't make for a very good user experience.

comment:24 Changed 5 years ago by Adys

I had a look through the patch today. Using a DecimalField and playing with commas seems unreasonably hacky. Is using a 64-bit int an option? It shouldn't be a problem under postgres, but I don't know about the other db backends.

comment:25 Changed 5 years ago by Adys

For the interface part, there's two decent choices.
Either go for a parse of the values in a single field, for example (input) "3d 12h 4m 7s", "7d 10m", "150000 ms"; output would be highest value possible first - 60000ms would be "1m", 75000ms would be "1m 15s" (abbreviations are up for change, would go for the standard ones).

Or, go for a single field with a <select> box, with some js shortcuts. [_] [ Unit ... ] (X)

In both cases, the possible units would be years, months, weeks, days, hours, minutes, seconds, milliseconds, microseconds (assuming microsecond precision). Thoughts?

comment:26 Changed 5 years ago by Gulopine

I'm a bit confused as to what you mean by "playing with commas," but thus far, I don't see anything at all hacky with using a DecimalField. Going with a 64-bit int would only be an option if Django supports a standalone cross-database 64-bit integer field. I don't want to invent one just for this. As for being hacky, I find it extremely elegant that the current approach stores seconds in the database, allowing for super-simple conversion: value = datetime.timedelta(seconds=float(value)). Plus, it works with all existing backends and maintains the ability to order records as you would expect (though, admittedly, your 64-bit int would also preserve that feature). What's exactly is hacky about it?

On the interface front, there was (once upon a time) a prevailing notion of using two text boxes: one for days and one for everything else. Basically, the days would be cleaned as a standard integer field, while the other would be cleaned as a time field, so the syntax for stuff less than a day would follow the "hh:mm:ss" format. The only thing that held me back from actually implementing that was that it would also be great to have a JavaScript helper similar to that of TimeField, showing a few useful options to get you started. Something like: 30 seconds, 2 hours, 1 day, 6 weeks (or some such similar options).

I obviously haven't gotten back around to this ticket in quite some time, but I'll see if I can devote some time to it soon to get the original UI proposal up and running.

comment:27 Changed 5 years ago by Adys

It feels wrong to "store" as seconds. Then again, just my opinion.

I was thinking the same thing about a js helper, and a friend came up with the idea of just parsing the field for units. I'll see if I can get a few mins to work on it tonight, I really like the idea. Three big advantages:

  • No need for javascript (6 weeks would be "6w", 30 seconds would be "30s", 2 hours would be "2h", etc)
  • Easy to understand, very user friendly (parsing "3h 10d" is okay, I suppose we'd give validationerror on "1h 10m 2h" or similar tho)
  • No need for multiple input boxes, avoids confusion


On a slightly-related note, I don't know what to do about microseconds. Standard display is µs, but that could cause problems for users with special keyboard layouts etc.

comment:28 Changed 5 years ago by emes

  • Cc michal@… added

I think the year and month units are misleading, as in real life they have no fixed length. This is probably why the timedelta type does accept day as the longest unit possible.

We can assume that year is an astronomical unit, but it's almost never useful in real life. A year has to round to days and it's length is known only when applied to a certain start date. The same (or even worse) happens with a month. In Gregorian calendar there are 4 different lengths possible: 28, 29, 30, 31 days. We could assume that month is astronomical unit again (moon cycle) but it will roughly match only the Muslim calendar.

I'd prefer to leave a week as the longest unit possible.

comment:29 Changed 5 years ago by Adys

  • Cc michal@… removed

Emes, I agree with you. I just wanted to get the implementation in. It's easier to remove than to add-on.

I sent the Dev list a mail a couple of days ago: http://groups.google.com/group/django-developers/browse_thread/thread/612e6c641d109f40?hl=en

Another question I bumped into this morning: Should we allow negative values?

comment:30 Changed 5 years ago by Adys

  • Cc michal@… added

cc michal@…, sorry for the spam, Firefox is horrible.

Changed 5 years ago by Adys

New DurationField implementation (try 3)

comment:31 Changed 5 years ago by Adys

Any feedback on the DurationField patch? I still need to figure out the admin output.

Changed 5 years ago by Adys

Fully working DurationField implementation with changes and fixes from Yuri Baburov

comment:32 Changed 5 years ago by Adys

  • Owner changed from Gulopine to Adys

Yuri did some updates to the patch posted above; I'll post a rebase by tomorrow. This rebase also comments out support for months and years as per list discussion; since it's just two values, might as well leave them commented.

1.1's almost out if I am to trust the targeted bugs. I suppose we could start looking at DurationField implementation for 1.2? This ticket has been open long enough.

Reassigning to myself.

comment:33 Changed 5 years ago by Adys

  • milestone set to 1.2

Ok, 1.1 is out, it's time for an update.
This ticket has been open long enough. I'm decided to get everything closed for 1.2. I'm going to upload a rebased patch today or tomorrow. Right now, there are three outstanding issues with it:

  • In the admin, a durationfield appears as a Decimal's unicode in filters, and when it's equal to zero on the admin detail page.
  • Validation isn't getting handled correctly in the admin - a 500 is raised rather than a ValidationError.
  • Lack of docs/tests

First issue is something I've tried to fix but came up blank. Second one should be easy enough to fix (I'll have a look before rebasing hopefully), third one I'll leave until the former two are fixed.
I'm still *very* itchy about using a DecimalField as backend. But hey...

Changed 4 years ago by Adys

Updated patch using the new BigIntegerField as backend

comment:34 follow-up: Changed 4 years ago by Adys

There, new updated patch.
See: http://groups.google.com/group/django-developers/browse_thread/thread/d32316ce7964dbaf?hl=en

Unless there are bugs, I will not work on this anymore. The core devs seem to hate the idea of having a DurationField... meanwhile I'm perfectly happy keeping my core patched.

comment:35 in reply to: ↑ 34 Changed 4 years ago by russellm

Replying to Adys:

Unless there are bugs, I will not work on this anymore. The core devs seem to hate the idea of having a DurationField...

Yes, the core devs clearly hate the idea of DurationField. That's why this ticket was marked accepted by Jacob, and is on the low prioirty feature list for v1.2.

There is a difference between "don't like an idea" and "haven't had time to apply a patch because we've been busy with one or two other things". If we didn't like the idea, we'd close the ticket.

meanwhile I'm perfectly happy keeping my core patched.

I'll also point out that you don't need to patch core to write your own field definition. You can define fields in your own code base, and import/use them as you wish.

comment:36 Changed 4 years ago by sebzur

Adys, I liked the DurationField idea and I hope You were not seorious saying that You're going to leave it? I've tried it in my my project - there are some issues with it which I'm going to address soon - but all in all it works OK and I will be happy to help maintain it.

It would be great to see it in django itself in the future. Up to this time, I must agree with russellm that it would be easier to use/test it 'externally'. Right now, I had to create another django version (along side with my default system one from Gentoo) and patch it to use DurationField... I want to stay in-sync with Your code - that's why I didn't extract it.

Please, consider such an extraction, I belive it would made the DurationField more attractive (while it's not in django).

comment:37 Changed 4 years ago by Alex

  • milestone changed from 1.2 to 1.3

We are now passed the 1.2 feature freeze.

comment:38 Changed 4 years ago by jpaulett

  • Cc john@… added

I am not sure if this is appropriate, but I took the latest patch by Adys and turned it into a reusable application, django-durationfield. If anyone has an issue with turning the patch into an external app, please let me know.

I am viewing the django-durationfield app as a temporary solution since this patched missed the 1.2 feature deadline. I will try to keep the app in sync with latest patch for this issue (and contribute any fixes, documentation, and tests to this issue).

Assuming that a DurationField is eventually committed to Django core, the switch from the django-durationfield app to the core version should just be a matter of switching a few import statements.

comment:39 Changed 4 years ago by gogna

  • Cc flavio.curella@… added

comment:40 Changed 4 years ago by rasca

  • Keywords feature added

comment:41 Changed 4 years ago by ubernostrum

  • milestone 1.3 deleted

1.3 is feature-frozen now.

comment:42 Changed 3 years ago by slafs

  • Cc slafs added

comment:43 Changed 3 years ago by lrekucki

  • Type changed from enhancement to New feature

comment:44 Changed 3 years ago by lrekucki

  • Severity changed from minor to Normal

comment:45 Changed 3 years ago by fckin.spam@…

  • Cc fckin.spam@… added
  • Easy pickings unset

comment:46 Changed 3 years ago by evgenpioneer@…

  • Cc evgenpioneer@… added

comment:47 Changed 3 years ago by sorin

  • Cc sorin added

comment:48 Changed 3 years ago by Karen Rustad

  • UI/UX unset

This is a vote to get a DurationField (or whatever you want to call it) into Django 1.4. I'm currently running into this problem in that I need to represent an iteration length in some task management software I'm writing, where the length is set by the user and could be anything from a week to a month to a couple months (even more hilarity since months aren't consistent lengths). Of course, timedeltas are super-conveniently built into Python and can handle all of this in a snap...but in Django I have no way to store them. It's driving me nuts.

Like, come on. If Django is Python, timedelta needs a model field. Without, it's incomplete.

comment:49 Changed 3 years ago by anonymous

  • UI/UX set

comment:50 follow-up: Changed 3 years ago by jpaulett

Karen,

While Python does provide the timedelta class, there is no common SQL implementation for storing that object in the database. PostgreSQL does provide the INTERVAL data type, MySQL and SQLite do not have any interval data type (I am not sure about Oracle, MSSQL, etc.).

Luckily Django makes it relatively simple to create custom model fields. If you are willing to be PostgreSQL-specific, you could store timedeltas as INTERVALs (I believe the psycopg driver handles the conversion for you).

You could also convievably pickle the timedelta instance and store that blob in the database. This approach breaks the sorting semantics, potentially corrupts the data across Python versions, and is completely opaque from a pure database perspective.

Another approach is to store the data in the database as a BIGINT. This approach has the advantage of working across databases and retains the the expected sorting semantics. The downside to this approach from a pure database perspective, is that you now must know the function to convert from a BIGINT into an interval. Based upon the patches included in this ticket, myself and several other developers created django-durationfield. One advantage of being a standalone app as opposed to being in core is that we can release new features & bug fixes without having to wait for a major Django point release.

I do not mean to speak for the core developers, but since there is no ideal solution and since using a standalone app is extremely easy, I doubt we will see this feature included in core anytime soon.

comment:51 Changed 3 years ago by carbonXT

  • Cc mike@… added

comment:52 in reply to: ↑ 50 Changed 2 years ago by smcoll

Replying to jpaulett:

While Python does provide the timedelta class, there is no common SQL implementation for storing that object in the database. PostgreSQL does provide the INTERVAL data type, MySQL and SQLite do not have any interval data type (I am not sure about Oracle, MSSQL, etc.).

What about MySQL's TIME type?

"MySQL retrieves and displays TIME values in 'HH:MM:SS' format (or 'HHH:MM:SS' format for large hours values). TIME values may range from '-838:59:59' to '838:59:59'. The hours part may be so large because the TIME type can be used not only to represent a time of day (which must be less than 24 hours), but also elapsed time or a time interval between two events (which may be much greater than 24 hours, or even negative)."

http://dev.mysql.com/doc/refman/5.0/en/time.html

comment:53 Changed 21 months ago by net147

  • Cc net147 added

comment:54 Changed 15 months ago by aaugustin

  • Cc aaugustin added

comment:55 Changed 12 months ago by styne666

  • Cc styne666 added

comment:56 Changed 8 months ago by k_sze

I think this ticket is mostly moot now that there is django-timedeltafield, which works quite nicely. See links below:

https://pypi.python.org/pypi/django-timedeltafield
https://bitbucket.org/schinckel/django-timedelta-field/

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as new
The owner will be changed from Adys to anonymous. Next status will be 'assigned'
as The resolution will be set. Next status will be 'closed'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.