Update ISO_INPUT_FORMATS to allow date filtering for ISO8601 dates

[Shawn Campbell]

Currently django_filters and django_rest_framework pull in the ISO_INPUT_FORMATS dictionary from django.utils.formats to try and convert dates/datetimes passed in as strings to datetime objects. There is currently no format strings compatible with datetime.isoformat(). I have updated the dictionary to add compatible format strings and tested django_filters and django_rest_framework, both work correctly with this update. I've ran all of the tests in django and all pass as expected. This is the github pull request: ​https://github.com/django/django/pull/3195

[Preston Timmons]

Left some comments on the PR. It could use documentation updates and some more specific tests for DateTimeField.

[José Padilla]

I decided to step up and finish updating this to get it done. Would we also have to update DATETIME_INPUT_FORMATS on global_settings to include then new formats?

[José Padilla]

Pull Request: ​https://github.com/django/django/pull/3268

Update: If this looks good, what docs/release notes should I work on?

[Tom Christie]

I'm slightly unsure about this - Django still wouldn't be able to handle ISO8601 timezone information after this change, and I assume that eg. 2006-10-25T14:30:59Z and 2006-10-25T14:30:59+01:00 would both still fail. That doesn't necessarily make this change invalid, but it doesn't look like it'd resolve the issue as it is described. (And no, strftime *can't* handle all the ISO8601 cases, see django.utils.dateparse).

Confusingly I also don't see ISO_INPUT_FORMATS referenced in django-filter or django-rest-framework (I know the later doesn't use it for sure) so I'm unclear on *exactly* what downstream issue we're trying to resolve, or able to review if this would be an adequate fix for that.

[Shawn Campbell]

Sorry, I got swallowed by work and family commitments and wasn't able to come back and update the PR based upon feedback. Thanks @jpadilla for updating this.

ISO_INPUT_FORMATS isn't referenced directly in either django-filter nor django-rest-framework. The chain goes like this:

• DRF references DF.filterset (line 191 in my local django-filter)
  • DF.filter.DateTimeFilter which references django.forms.fields.DateTimeFleld (on line 120 in filters.py in django-filters) which calls
    • django.formats.get_format_lazy('DATETIME_INPUT_FORMATS') which returns a list of all the setup datetime formats to try and parse the incoming string against using strptime
      • This is on line 490 in the django.forms.fields.py file. get_format_lazy calls get_format which references ISO_INPUT_FORMATS.

That's why updating the ISO_INPUT_FORMATS dict fixes the DRF issue for most/some cases. I had overlooked the timezone issue with ISO8601 apparently. Maybe we should also update the BaseTemporalField (DateTimeFields super class) to use the django.utils.dateparse instead of strptime?

That seems like a larger change than the original but seems like the correct answer to me.

[Tom Christie]

to use the django.utils.dateparse instead of strptime?

I guess what's awkward about that is that it's then not obvious how to also support the input formats list.

For reference, in REST framework we have a special format string 'ISO8601' that uses dateparse, and any other format strings use strftime I've no idea if that'd also make sense for Django or not.

[José Padilla]

Replying to jaegerpicker:

Since this was originally an issue you were working on, and apparently needs a bit more work after my PR, which I don't fully understand, would you want to pick this up again or should we just close it for now?

Sorry, I got swallowed by work and family commitments and wasn't able to come back and update the PR based upon feedback. Thanks @jpadilla for updating this.

ISO_INPUT_FORMATS isn't referenced directly in either django-filter nor django-rest-framework. The chain goes like this:

• DRF references DF.filterset (line 191 in my local django-filter)
  • DF.filter.DateTimeFilter which references django.forms.fields.DateTimeFleld (on line 120 in filters.py in django-filters) which calls
    • django.formats.get_format_lazy('DATETIME_INPUT_FORMATS') which returns a list of all the setup datetime formats to try and parse the incoming string against using strptime
      • This is on line 490 in the django.forms.fields.py file. get_format_lazy calls get_format which references ISO_INPUT_FORMATS.

That's why updating the ISO_INPUT_FORMATS dict fixes the DRF issue for most/some cases. I had overlooked the timezone issue with ISO8601 apparently. Maybe we should also update the BaseTemporalField (DateTimeFields super class) to use the django.utils.dateparse instead of strptime?

That seems like a larger change than the original but seems like the correct answer to me.

[Shawn Campbell]

I'd be glad to take this on, can't promise how soon I'll get to it though. It's entering our busiest part of the year at work. I'll try to work on it asap.

Replying to jpadilla:

Replying to jaegerpicker:

Since this was originally an issue you were working on, and apparently needs a bit more work after my PR, which I don't fully understand, would you want to pick this up again or should we just close it for now?

Sorry, I got swallowed by work and family commitments and wasn't able to come back and update the PR based upon feedback. Thanks @jpadilla for updating this.

ISO_INPUT_FORMATS isn't referenced directly in either django-filter nor django-rest-framework. The chain goes like this:

• DRF references DF.filterset (line 191 in my local django-filter)
  • DF.filter.DateTimeFilter which references django.forms.fields.DateTimeFleld (on line 120 in filters.py in django-filters) which calls
    • django.formats.get_format_lazy('DATETIME_INPUT_FORMATS') which returns a list of all the setup datetime formats to try and parse the incoming string against using strptime
      • This is on line 490 in the django.forms.fields.py file. get_format_lazy calls get_format which references ISO_INPUT_FORMATS.

That's why updating the ISO_INPUT_FORMATS dict fixes the DRF issue for most/some cases. I had overlooked the timezone issue with ISO8601 apparently. Maybe we should also update the BaseTemporalField (DateTimeFields super class) to use the django.utils.dateparse instead of strptime?

That seems like a larger change than the original but seems like the correct answer to me.

[Tim Graham]

Seems to be closely related to or a duplicate of #11385 and #25189. I think we need a discussion on the DevelopersMailingList if we are to make any changes here.