Opened 3 years ago

Closed 15 months ago

Last modified 14 months ago

#22258 closed New feature (fixed)

Show progress during dumpdata

Reported by: Gwildor Sok Owned by: Karol Duleba
Component: Core (Management commands) Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


In response of ticket #22251, I'm opening this as a separate issue like requested. You can read the need for this option there, but shortly speaking, the commands as they are now are not user friendly at all. The commands can go on for a very long time, depending on your database size, without giving any clue of its state and progress.

If and when #22257 is implemented, the commands could start by collecting amount of rows to progress, and periodically printing the progress to the console. This way the user can make a fair estimate on when the commands are done and whether they are actually still running.

Attachments (1)

22258#test.diff (1.2 KB) - added by ANUBHAV JOSHI 3 years ago.

Download all attachments as: .zip

Change History (11)

comment:1 Changed 3 years ago by Aymeric Augustin

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedAccepted

comment:2 Changed 3 years ago by ANUBHAV JOSHI

We can add the patch given #22257, then as posted there:
I think we can keep default as stdout and and for this, we can do the following:

  • When -o is not used, let the data be printed to stdout, as the data is printing people can see it and I don't think any progress display is required there.
  • When -o will be used, then we can display progress percentage or something like that.

backwards compatibilty is there, also issue here is also addressed.

comment:3 Changed 3 years ago by Russell Keith-Magee

Before we get too carried away here, someone needs to explain how it's going to be implemented. json.dumps and lxml don't provide progress APIs AFAIK.

comment:4 Changed 3 years ago by ANUBHAV JOSHI

Well I am attaching a simple diff just for JSON. It does very simply does the job of displaying progress(random not actual), but we can think along similar lines like we could calculate total no. of objects and then print progress accurately depending upon no of objects serialized.

Changed 3 years ago by ANUBHAV JOSHI

Attachment: 22258#test.diff added

comment:5 Changed 2 years ago by Karol Duleba

Owner: changed from nobody to Karol Duleba
Status: newassigned

I was thinking of adding some kind of utility to allow showing progress from any management command.

As far as I looked (loading data) there is no clean way of showing % of progress because of generators/reading streams from files, so progress like during tests. For data load it might be possible to show proper progress bar.

comment:6 Changed 15 months ago by Claude Paroz

Has patch: set
Needs tests: set

188a2418807f661eedfc675731521f2f46a3c49a (#21652) added progress information for loaddata.

This PR is adding progress report for dumpdata:

comment:7 Changed 15 months ago by Claude Paroz

Needs tests: unset

comment:8 Changed 15 months ago by Tim Graham

Summary: Show progress during dumpdata and loaddataShow progress during dumpdata
Triage Stage: AcceptedReady for checkin

comment:9 Changed 15 months ago by Claude Paroz <claude@…>

Resolution: fixed
Status: assignedclosed

In c296e55:

Fixed #22258 -- Added progress status for dumpdata when outputting to file

Thanks Gwildor Sok for the report and Tim Graham for the review.

comment:10 Changed 14 months ago by Tim Graham <timograham@…>

In eaa3c88:

Refs #22258 -- Fixed an unclosed temporary file in fixtures test.

This prevented the temporary directory from being removed
on Windows.

Note: See TracTickets for help on using tickets.
Back to Top