Support for overriding result of model field values

[Anas Abou Allaban]

We have many models with timestamp fields (ex. created_at, updated_at, etc.) that store DateTimeFields.
We use Django as a REST API for web and mobile clients written using JavaScript where time is stored as milliseconds since the UNIX epoch (i.e. Date.now()).

To minimize the amount of processing we do client side, we convert DateTimeFields to milliseconds using a Django Rest Framework (DRF) serializer that looks like this:

class TimestampDateTimeField(serializers.DateTimeField):
    """Convert Python datetime to/from Unix timestamp in milliseconds.
    For use with JavaScript's Date objects which use ms since epoch, not seconds.
    """

    def to_representation(self, value):
        return round(value.timestamp() * 1000)

    def to_internal_value(self, value):
        try:
            result = datetime.fromtimestamp(value / 1000, timezone.get_default_timezone())
            return super(TimestampDateTimeField, self).to_internal_value(result)
        except TypeError:
            raise serializers.ValidationError("Datetime must be a number, not a string")

Ideally, we could instead do this data manipulation at the database layer using the Extract function but with the same behavior as the serializer where we can override the original name of the field.

Example:

# This will currently throw an error
# ValueError: The annotation 'updated_at' conflicts with a field on the model.
m = MyModel.objects.filter(user=user).annotate(
  updated_at=Extract("updated_at", "epoch") * 1000
)

The way we can do this instead is using the extra() method from the QuerySet API to generate a query that would look like:

SELECT (EXTRACT(EPOCH FROM app_mymodel.updated_at) * 1000) AS updated_at
FROM app_mymodel 
WHERE app_mymodel.user_id = 42
LIMIT 1;

I did my best to look online and in the docs for potential solutions/workarounds, but all point back to using the extra(). Is there a way to do this w/o using extra()?

[David Sanders]

Thanks for the request but this was an intentional change to prevent data loss.

You can either setup a new annotation get DRF to refer to that (it's quite easy by specifying the source kwarg for that field in the model serializer's extra_kwargs) or you'll need to manually manipulate the query underneath the queryset eg: ​https://github.com/shangxiao/stupid-django-tricks/tree/master/annotation_overwrite (Standard disclaimer applies here as that's not an official Django recommendation).

Good luck!

[Natalia Bidart]

Hello!

(EDIT: I was writing this message when David submitted theirs, so while David's answer is more accurate, I'll post mine to share the links and doc pointers).

This report seems like a mix of a new feature request and a support request. The best place to get answers to your final question ("Is there a way to do this w/o using extra()?") is using any of the user support channels from ​this link. For feature requests, the current procedure is to start a new conversation on the ​Django Forum to present the idea and get community consensus, as per ​the documented guidelines for requesting features.

Since the goal of this issue tracker is to track issues about Django itself, I'll be closing this ticket as invalid following the ​Ticket Triaging policy.

Having said that, what I personally would do is to have different field names for the DateTimeField and its "processed" milliseconds since epoch, and I would have the latter be a GeneratedField (or an annotation) of the former.

Thank you!