Add overwrite mode to collectstatic

[Prathik Rajendran M]

We had an issue in the production server where the files in STATIC_ROOT were not being replaced by the ones in the individual apps. As a solution for this we ran collectstatic with the -c option. This worked but what happened was the the entire static root was erased, this static root was configured to point to the static files folder in the container app which resulted in some of the files being permanently deleted because collectstatic first clears the folder then collects the staticfiles. While it is true that it was incorrect to set it up this way, it would be ideal if it wouldn't have deleted the files.

Another problem was that due to some reason the source files had a timestamp lesser than the destination files, this could have been due to a caching mechanism but this resulted in the source files not being collected.

To solve for the above issues we need a overwrite method which would overwrite the source files no matter what the timestamp and it would not perform the copy if the source and the destination are the same.

Pull request: ​https://github.com/django/django/pull/3435

[Prathik Rajendran M]

Pull request: ​https://github.com/django/django/pull/3434

[Tim Graham]

Thank-you for the suggestion, but you admitted that you setup things incorrectly and I don't think we should add features to cater to that. Is there a reason you cannot fix your incorrect setup so that --clear won't have unintended side-effects?

[Prathik Rajendran M]

This feature can come in handy for situations where other apps are also writing into the STATIC_ROOT folder or in scenarios where the files in STATIC_ROOT have changed. This would be a safer way to do things as we are not deleting anything,

Since misconfigurations do happen, especially when dev and ops are different teams, I think Django should be able to stop from unintended effects from happening, for-example the deletion of STATIC_ROOT folder here caused the static files of the app to be deleted (since the location of static folder of the app and STATIC_ROOT are the same) and it couldn't be retrieved until another release is done. I think it would be nice if Django alerts the users in these scenarios or we could have an overwrite method which doesn't delete anything.

It's not necessary that we have an overwrite method but it would be ideal if Django would prevent such scenarios from happening. Please let me know what your thoughts/alternative suggestions on this.

[Tim Graham]

I think the documentation contains ​a clear warning about the mistake you made. We also recently ​clarified the warning for collectstatic --clear and ​fixed an issue where the destination directory wouldn't be displayed. Is this enough? (I wonder if you are using an older version of Django that doesn't include these improvements?)

[Prathik Rajendran M]

Yup it's definitely a mistake but we could be mitigating the damage if Django prevented --clear from happening in this case, at times developers don't handle the deployment or there could be multiple developers, someone might have not read that warning, thus I think Django should at least warn that they are within the app's folder and that what is deleted can't be restored by collectstatic.

Is there a way in which we can prevent collectstatic from deleting the files if they don't belong to the app?

Also if the STATIC_ROOT points to a folder within an app then it shouldn't be deleted or atleast user should be warned about the cost, that is they have to re-deploy to get the files back and if they don't have a repository then the files are lost forever.

Also I think it would be better if --clear only deleted files that belongs to Django, this would allow multiple apps (non-django) to write into the STATIC_ROOT too. These were the reasons that led to the development of the --overwrite mode, which doesn't touch any other file apart from that which belongs to django.

[Tim Graham]

I don't think Django can track which static files it manages. Consider an app that has some of its static files removed from its app directory. Django would then have to consider the copies of those files that remain in STATIC_ROOT as "unmanaged" and they would never be deleted.

I think the warnings I linked to in my previous comment should be enough for people to see that files in their app's static directory will be deleted if they proceed.

I'm going to close this ticket as "won't fix" as the idea of an "overwrite option" goes against one of the design decisions of static files. If you wish to argue for a design decision, could you please start a thread on the DevelopersMailingList? Thanks!

[Prathik Rajendran M]

Tim, out of curiosity, what would happen if a file has been deleted from the static files of an app and the user runs collectstatic with no options? Would it be removed from STATIC_ROOT?

[Tim Graham]

I wouldn't expect it would; you'd have to confirm though.