#2443 closed New feature (fixed)
Add IntervalField to database models
Reported by: | Owned by: | Marc Tamlyn | |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | dev |
Severity: | Normal | Keywords: | IntervalField interval duration DurationField feature 1.8-alpha |
Cc: | jm.bugtracking@…, treborhudson@…, patrick.lauber@…, hv@…, michal@…, john@…, flavio.curella@…, Sławek Ehlert, fckin.spam@…, evgenpioneer@…, sorin, mike@…, net147, Aymeric Augustin, styne666 | Triage Stage: | Accepted |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | yes |
Easy pickings: | no | UI/UX: | yes |
Description
Although not all backends support the interval data type, it can be implemented using some other data type. This field is quite useful and it would be nice if it were available.
Attachments (10)
Change History (76)
comment:1 by , 18 years ago
priority: | normal → lowest |
---|---|
Severity: | normal → minor |
comment:2 by , 18 years ago
Resolution: | → wontfix |
---|---|
Status: | new → closed |
comment:3 by , 18 years ago
Resolution: | wontfix |
---|---|
Status: | closed → reopened |
start-interval and end-interval doesn't work for all situations. Suppose you want to track "Amount of time spent driving in the past week" Unless I'm misunderstanding your use of start-interval and end-interval, there isn't an easy way to store this value with date/time fields. The real bonus of intervals comes in when you are able to do computations with them (such as "Amount of time spent driving in May" + "Amount of time spent driving in June")
comment:5 by , 18 years ago
Has patch: | set |
---|---|
Keywords: | DurationField added |
Needs tests: | set |
Patch needs improvement: | set |
This simple patch adds a DurationField, which, in my opinion, better encapsulates the use cases described by anonymous above. Calling it a duration implies that it's not tied to any specific points in time, but rather just the passage of a certain amount of time. This makes more sense for things like the length (duration!) of audio and video content, for instance.
The patch itself is fairly simple, based on a FloatField, whose value represents the number of seconds to be represented. This seems sufficient for database storage, since all supported backends already support the equivalent of a FloatField, while max_digits
and decimal_places
allow for a number large enough to represent timedelta.max
, complete with a full set of microseconds. It doesn't take calendars into account at all, but neither does timedelta
, which is also in keeping with the duration concept as opposed to intervals.
The one problem I have with it so far is that it seems like Django is relying on backend database connection modules to handle coercion into Python types, rather than using to_python
, so the value isn't changed into a timedelta
unless you manually call .validate()
on the model. This also prevents the validation errors from occurring properly when editing in the admin. That may be another issue for another day, however. I've marked the issue as "Patch needs improvement" in case I'm missing something that would make this happen.
P.S. Pardon the WikiFormatting on the attachment comment. That is, unless it gets a wiki page, in which case it'll be fine.
comment:6 by , 18 years ago
I should also add, under the heading of "Patch needs improvement," that typing in decimals as a float value is hardly intuitive, and a better UI should be used. I, unfortunately, haven't thought of a good way to visually represent it, so I don't have any suggestions.
by , 18 years ago
Attachment: | durationfield.2.diff added |
---|
Updated DurationField location to actually work (I hadn't tested the other one)
by , 18 years ago
Attachment: | durationfield.3.diff added |
---|
Complete patch with widget, supporting oldforms and newforms, including newforms-admin
comment:7 by , 18 years ago
Needs documentation: | set |
---|---|
Patch needs improvement: | unset |
This most recent patch covers many more aspects of DurationField, including a newforms field and widget, fully compatible with newforms-admin. This new widget implements a bit better of a UI, separating the value into its components (days, hours, minutes, seconds, microseconds).
This new widget can't reasonably be done with oldforms, and since it's going out anyway, oldforms (and the current admin) will simply use a FloatField describing the number of seconds (including microseconds).
Also, this version of the patch relies on #3982 to enable support of the coerce
method.
comment:8 by , 18 years ago
Cc: | added |
---|
comment:9 by , 18 years ago
Needs documentation: | unset |
---|
Once again, this patch supercedes all others, and once again relies on a new patch from #3982. However, it now uses lazy instantiation of timedelta
objects, which should improve performance. Documentation is also included for trunk, with a few minor changes needed for the
newforms-admin
branch (due to the newforms widget included in the patch). Tests will come shortly, once I have a few other patches ironed out.
by , 17 years ago
Attachment: | durationfield.5.diff added |
---|
Updated to work with latest trunk, and to no longer rely on #3982
comment:10 by , 17 years ago
Keywords: | duration added |
---|---|
Owner: | changed from | to
Patch needs improvement: | set |
Status: | reopened → new |
This latest patch is missing a few things before it can be considered final, but it's getting close. It no longer relies on #3982, but it doesn't yet support the recent error_messages
stuff, and fails with the JSON serializer. It works with the XML serializer though, and should work in newforms-admin as is.
by , 17 years ago
Attachment: | durationfield.6.diff added |
---|
Fixed a problem with creating objects by hand, and removed a couple debug lines
comment:11 by , 17 years ago
I've had some cases (mainly in the admin-interface) where get_db_prep_save(self, value) gets called with a value of type Decimal. I don't know why that happens, but I added this code to fix DurationField in these circumstances:
def get_db_prep_save(self, value): if value is None: return None if (type(value) == decimal.Decimal) or (type(value)==float): return str(value) return str(value.days * 24 * 3600 + value.seconds + float(value.microseconds) / 1000000)
If this is the correct way to fix this, please add it to the next diff you make.
comment:12 by , 17 years ago
I'm more interested to know why it's getting a Decimal, because that would probably break other expectations as well. Do you have any code that reliably shows that to happen? If you don't know exactly where it's going wrong, that's fine, just being able to reproduce it will help me figure it out.
comment:13 by , 17 years ago
OK... I did some testing and was able to reproduce the condition. In a simple Model with a DurationField create a record in Admin and enter 0 as the value. django-admin will throw a AttributeError: 'Decimal' object has no attribute 'days' and will prevent saving the value.
If you enter 0 directly into the database or save a datetime.timedelta(0) from the django shell (which works, unlike in the admin) that attribute will return a Decimal("0") instead of the timedelta object (once you re-fetch it from the database).
So my proposed patch to get_db_prep_save is the wrong place to fix this problem. It seems this condition is rooted where the data is converted from to database to the native python object.
Could this have something to do with the backend driver doing the coercion to python types that was reported in #3982?
I've been testing this with sqlite3 btw.
comment:14 by , 17 years ago
In DurationProxy.__get__
the value that instance.__dict__[self.field_name]
returns is a Decimal
object (if the value in the database is 0). With other values it is a TimeDelta
comment:15 by , 17 years ago
Triage Stage: | Design decision needed → Accepted |
---|
This needs some UI help, but it's otherwise perfect.
comment:17 by , 17 years ago
I have no plans to support interval
, since DecimalField
allows for compatibility with all of Django's backends. The only real reason to support interval
is for inspecting existing databases that use it. If you'd like support for that, feel free to add in whatever code it would take to do it. I'm just too bogged down with other things at the moment, and supporting interval
isn't high on my priority list.
comment:18 by , 17 years ago
Cc: | added |
---|
I tried to apply this patch and it no longer applies cleanly. I believe django/newforms/fields.py and django/newforms/widgets.py had a reject and some fuzz.
When I did get it applied I was getting some errors when I went into the admin and tried to add data to my model...
Traceback: File ".../django_trunk/django/contrib/admin/views/main.py" in add_stage 287. new_data = manipulator.flatten_data() Exception Value: 'str' object has no attribute 'items'
comment:19 by , 16 years ago
Cc: | added |
---|
by , 16 years ago
Attachment: | durationfield.7.diff added |
---|
Updated patch for current trunk, without docs (no idea how much they've changed)
comment:22 by , 16 years ago
What's holding DurationField from going upstream (at least in contrib)? I can help fix whatever is necessary.
comment:23 by , 16 years ago
Well, you'll notice that yours is the first patch to hit this ticket since Jacob said it "needs some UI help." That criticism still stands, since a half-dozen input fields don't make for a very good user experience.
comment:24 by , 16 years ago
I had a look through the patch today. Using a DecimalField and playing with commas seems unreasonably hacky. Is using a 64-bit int an option? It shouldn't be a problem under postgres, but I don't know about the other db backends.
comment:25 by , 16 years ago
For the interface part, there's two decent choices.
Either go for a parse of the values in a single field, for example (input) "3d 12h 4m 7s", "7d 10m", "150000 ms"; output would be highest value possible first - 60000ms would be "1m", 75000ms would be "1m 15s" (abbreviations are up for change, would go for the standard ones).
Or, go for a single field with a <select> box, with some js shortcuts. [_] [ Unit ... ] (X)
In both cases, the possible units would be years, months, weeks, days, hours, minutes, seconds, milliseconds, microseconds (assuming microsecond precision). Thoughts?
comment:26 by , 16 years ago
I'm a bit confused as to what you mean by "playing with commas," but thus far, I don't see anything at all hacky with using a DecimalField. Going with a 64-bit int would only be an option if Django supports a standalone cross-database 64-bit integer field. I don't want to invent one just for this. As for being hacky, I find it extremely elegant that the current approach stores seconds in the database, allowing for super-simple conversion: value = datetime.timedelta(seconds=float(value))
. Plus, it works with all existing backends and maintains the ability to order records as you would expect (though, admittedly, your 64-bit int would also preserve that feature). What's exactly is hacky about it?
On the interface front, there was (once upon a time) a prevailing notion of using two text boxes: one for days and one for everything else. Basically, the days would be cleaned as a standard integer field, while the other would be cleaned as a time field, so the syntax for stuff less than a day would follow the "hh:mm:ss" format. The only thing that held me back from actually implementing that was that it would also be great to have a JavaScript helper similar to that of TimeField, showing a few useful options to get you started. Something like: 30 seconds, 2 hours, 1 day, 6 weeks (or some such similar options).
I obviously haven't gotten back around to this ticket in quite some time, but I'll see if I can devote some time to it soon to get the original UI proposal up and running.
comment:27 by , 16 years ago
It feels wrong to "store" as seconds. Then again, just my opinion.
I was thinking the same thing about a js helper, and a friend came up with the idea of just parsing the field for units. I'll see if I can get a few mins to work on it tonight, I really like the idea. Three big advantages:
- No need for javascript (6 weeks would be "6w", 30 seconds would be "30s", 2 hours would be "2h", etc)
- Easy to understand, very user friendly (parsing "3h 10d" is okay, I suppose we'd give validationerror on "1h 10m 2h" or similar tho)
- No need for multiple input boxes, avoids confusion
On a slightly-related note, I don't know what to do about microseconds. Standard display is µs, but that could cause problems for users with special keyboard layouts etc.
comment:28 by , 16 years ago
Cc: | added |
---|
I think the year and month units are misleading, as in real life they have no fixed length. This is probably why the timedelta type does accept day as the longest unit possible.
We can assume that year is an astronomical unit, but it's almost never useful in real life. A year has to round to days and it's length is known only when applied to a certain start date. The same (or even worse) happens with a month. In Gregorian calendar there are 4 different lengths possible: 28, 29, 30, 31 days. We could assume that month is astronomical unit again (moon cycle) but it will roughly match only the Muslim calendar.
I'd prefer to leave a week as the longest unit possible.
comment:29 by , 16 years ago
Cc: | removed |
---|
Emes, I agree with you. I just wanted to get the implementation in. It's easier to remove than to add-on.
I sent the Dev list a mail a couple of days ago: http://groups.google.com/group/django-developers/browse_thread/thread/612e6c641d109f40?hl=en
Another question I bumped into this morning: Should we allow negative values?
by , 16 years ago
Attachment: | durationfield-new.diff added |
---|
New DurationField implementation (try 3)
comment:31 by , 16 years ago
Any feedback on the DurationField patch? I still need to figure out the admin output.
by , 16 years ago
Attachment: | durationfield-new.2.diff added |
---|
Fully working DurationField implementation with changes and fixes from Yuri Baburov
comment:32 by , 15 years ago
Owner: | changed from | to
---|
Yuri did some updates to the patch posted above; I'll post a rebase by tomorrow. This rebase also comments out support for months and years as per list discussion; since it's just two values, might as well leave them commented.
1.1's almost out if I am to trust the targeted bugs. I suppose we could start looking at DurationField implementation for 1.2? This ticket has been open long enough.
Reassigning to myself.
comment:33 by , 15 years ago
milestone: | → 1.2 |
---|
Ok, 1.1 is out, it's time for an update.
This ticket has been open long enough. I'm decided to get everything closed for 1.2. I'm going to upload a rebased patch today or tomorrow. Right now, there are three outstanding issues with it:
- In the admin, a durationfield appears as a Decimal's unicode in filters, and when it's equal to zero on the admin detail page.
- Validation isn't getting handled correctly in the admin - a 500 is raised rather than a ValidationError.
- Lack of docs/tests
First issue is something I've tried to fix but came up blank. Second one should be easy enough to fix (I'll have a look before rebasing hopefully), third one I'll leave until the former two are fixed.
I'm still *very* itchy about using a DecimalField as backend. But hey...
by , 15 years ago
Attachment: | durationfield.patch added |
---|
Updated patch using the new BigIntegerField as backend
follow-up: 35 comment:34 by , 15 years ago
There, new updated patch.
See: http://groups.google.com/group/django-developers/browse_thread/thread/d32316ce7964dbaf?hl=en
Unless there are bugs, I will not work on this anymore. The core devs seem to hate the idea of having a DurationField... meanwhile I'm perfectly happy keeping my core patched.
comment:35 by , 15 years ago
Replying to Adys:
Unless there are bugs, I will not work on this anymore. The core devs seem to hate the idea of having a DurationField...
Yes, the core devs clearly hate the idea of DurationField. That's why this ticket was marked accepted by Jacob, and is on the low prioirty feature list for v1.2.
There is a difference between "don't like an idea" and "haven't had time to apply a patch because we've been busy with one or two other things". If we didn't like the idea, we'd close the ticket.
meanwhile I'm perfectly happy keeping my core patched.
I'll also point out that you don't need to patch core to write your own field definition. You can define fields in your own code base, and import/use them as you wish.
comment:36 by , 15 years ago
Adys, I liked the DurationField idea and I hope You were not seorious saying that You're going to leave it? I've tried it in my my project - there are some issues with it which I'm going to address soon - but all in all it works OK and I will be happy to help maintain it.
It would be great to see it in django itself in the future. Up to this time, I must agree with russellm that it would be easier to use/test it 'externally'. Right now, I had to create another django version (along side with my default system one from Gentoo) and patch it to use DurationField... I want to stay in-sync with Your code - that's why I didn't extract it.
Please, consider such an extraction, I belive it would made the DurationField more attractive (while it's not in django).
comment:38 by , 15 years ago
Cc: | added |
---|
I am not sure if this is appropriate, but I took the latest patch by Adys and turned it into a reusable application, django-durationfield. If anyone has an issue with turning the patch into an external app, please let me know.
I am viewing the django-durationfield app as a temporary solution since this patched missed the 1.2 feature deadline. I will try to keep the app in sync with latest patch for this issue (and contribute any fixes, documentation, and tests to this issue).
Assuming that a DurationField is eventually committed to Django core, the switch from the django-durationfield app to the core version should just be a matter of switching a few import statements.
comment:39 by , 14 years ago
Cc: | added |
---|
comment:40 by , 14 years ago
Keywords: | feature added |
---|
comment:42 by , 14 years ago
Cc: | added |
---|
comment:43 by , 14 years ago
Type: | enhancement → New feature |
---|
comment:44 by , 14 years ago
Severity: | minor → Normal |
---|
comment:45 by , 14 years ago
Cc: | added |
---|---|
Easy pickings: | unset |
comment:46 by , 14 years ago
Cc: | added |
---|
comment:47 by , 14 years ago
Cc: | added |
---|
comment:48 by , 13 years ago
UI/UX: | unset |
---|
This is a vote to get a DurationField (or whatever you want to call it) into Django 1.4. I'm currently running into this problem in that I need to represent an iteration length in some task management software I'm writing, where the length is set by the user and could be anything from a week to a month to a couple months (even more hilarity since months aren't consistent lengths). Of course, timedeltas are super-conveniently built into Python and can handle all of this in a snap...but in Django I have no way to store them. It's driving me nuts.
Like, come on. If Django is Python, timedelta needs a model field. Without, it's incomplete.
comment:49 by , 13 years ago
UI/UX: | set |
---|
follow-up: 52 comment:50 by , 13 years ago
Karen,
While Python does provide the timedelta class, there is no common SQL implementation for storing that object in the database. PostgreSQL does provide the INTERVAL data type, MySQL and SQLite do not have any interval data type (I am not sure about Oracle, MSSQL, etc.).
Luckily Django makes it relatively simple to create custom model fields. If you are willing to be PostgreSQL-specific, you could store timedeltas as INTERVALs (I believe the psycopg driver handles the conversion for you).
You could also convievably pickle the timedelta instance and store that blob in the database. This approach breaks the sorting semantics, potentially corrupts the data across Python versions, and is completely opaque from a pure database perspective.
Another approach is to store the data in the database as a BIGINT. This approach has the advantage of working across databases and retains the the expected sorting semantics. The downside to this approach from a pure database perspective, is that you now must know the function to convert from a BIGINT into an interval. Based upon the patches included in this ticket, myself and several other developers created django-durationfield. One advantage of being a standalone app as opposed to being in core is that we can release new features & bug fixes without having to wait for a major Django point release.
I do not mean to speak for the core developers, but since there is no ideal solution and since using a standalone app is extremely easy, I doubt we will see this feature included in core anytime soon.
comment:51 by , 13 years ago
Cc: | added |
---|
comment:52 by , 13 years ago
Replying to jpaulett:
While Python does provide the timedelta class, there is no common SQL implementation for storing that object in the database. PostgreSQL does provide the INTERVAL data type, MySQL and SQLite do not have any interval data type (I am not sure about Oracle, MSSQL, etc.).
What about MySQL's TIME type?
"MySQL retrieves and displays TIME values in 'HH:MM:SS' format (or 'HHH:MM:SS' format for large hours values). TIME values may range from '-838:59:59' to '838:59:59'. The hours part may be so large because the TIME type can be used not only to represent a time of day (which must be less than 24 hours), but also elapsed time or a time interval between two events (which may be much greater than 24 hours, or even negative)."
comment:53 by , 12 years ago
Cc: | added |
---|
comment:54 by , 12 years ago
Cc: | added |
---|
comment:55 by , 11 years ago
Cc: | added |
---|
comment:56 by , 11 years ago
I think this ticket is mostly moot now that there is django-timedeltafield, which works quite nicely. See links below:
https://pypi.python.org/pypi/django-timedeltafield
https://bitbucket.org/schinckel/django-timedelta-field/
comment:57 by , 10 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:59 by , 10 years ago
Needs documentation: | unset |
---|---|
Patch needs improvement: | unset |
comment:60 by , 10 years ago
Patch needs improvement: | set |
---|
I think we need support for qs.filter(event_startdate__gte=datetime.now() - F('event_duration))
. I think this works on PostgreSQL, but at least on Oracle it produces wrong results. The event_duration is in microseconds, but in oracle the date arithmetic interprets integers as days. Actually, I am not sure if this query works at all.
In any case usages of F('duration_field') need tests. Raising errors from F() usage is acceptable, but producing wrong results is clearly not OK.
Another issue that keeps nagging me is the use of SubfieldBase to support value conversion (SubfieldBase basically calls to_python() anytime the field is set on the object). Other fields in Django core do not use this, instead they let the backend convert the values. I am not sure if usage of SubfieldBase leads to any problems, but I keep getting the feeling we should be consistent in not using SubfieldBase. One case where usage of backend specific converters differ from usage of to_python + SubfieldBase is that SubfieldBase does value conversions on assignment. So:
>>> obj.duration_field = '12:34:56' >>> obj.duration_field OUT: '12:34:56' if not using SubfieldBase OUT: timedelta(12h, 34m, 56s)
Currently no local field in Django core do value conversions on assignment. In addition there is clear overhead in model __init__
when using SubfieldBase, I'd guess the overhead is somewhere around 20% per field that uses SubfieldBase, though I haven't benchmarked this.
comment:61 by , 10 years ago
Patch needs improvement: | unset |
---|---|
Triage Stage: | Accepted → Ready for checkin |
Changes have been made to:
- remove SubFieldBase (completely!)
- Support arithmetic with DurationField
I believe this is now ready for checkin, I just need a final check from someone for the arithmetic changes.
comment:62 by , 10 years ago
Keywords: | 1.8-alpha added |
---|---|
Patch needs improvement: | set |
Triage Stage: | Ready for checkin → Accepted |
There's a test failure that needs fixing.
comment:63 by , 10 years ago
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
I'm tentatively marking this as wontfix, as surely a combination of a "start-interval" and "end-interval" fields is better normalisation.
Feel free to correct me if I'm wrong though!