Opened 5 years ago

Closed 5 months ago

Last modified 5 months ago

#26721 closed Cleanup/optimization (fixed)

Document redirecting dumpdata output may result in incorrect encoding in Windows PowerShell

Reported by: OBu Owned by: David Smith
Component: Documentation Version: 1.9
Severity: Normal Keywords: dumpdata loaddata
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by OBu)

When I "dumpdata" my database on my Windows 8.1 machine and load the dumped file with "loaddata", I get encoding errors.

Opening the dumped file with notepad++, setting the encoding to UTF-8 and saving it again helps.

A django-users thread describes the problem in detail so it seems I'm not alone...

It might not be a Django problem but a windows pipeline-issue (I was using the powershell), but it is a bit embarrassing.

Change History (11)

comment:1 Changed 5 years ago by Tim Graham

Description: modified (diff)

comment:2 Changed 5 years ago by Baptiste Mispelon

Hi,

Are you able to try to do dumpdata with both Python 2 and Python 3? Does it make any difference?

Thanks.

comment:3 Changed 5 years ago by OBu

Sorry, I don't. I was using python 3.5.1 and django 1.9.7

I just tried dir > test.txt in my windows command line, just to check the encoding of the resulting file, and notepad++ tells me it is UTF-8.

When I do the same in Windows Powershell, encoding is UCE-2 BE BOM

I assume this is exactly what happened to me, so it's not really a Django issue but a problem resulting from the character encoding in Windwos Powershell.

I'm not sure whether there is a way to fix this other then using the --output option (which results in a correctly encoded UTF-8 file no matter which shell was used), but maybe this problem should be mentioned in the docs since the --output option is new and most help web sites are using > .

Last edited 5 years ago by OBu (previous) (diff)

comment:4 Changed 5 years ago by OBu

Description: modified (diff)

comment:5 Changed 5 years ago by Tim Graham

Component: Core (Serialization)Documentation
Summary: "loaddata" can not load data generated by "dumpdata" due to encoding issuesDocument redirecting dumpdata output may result in incorrect encoding in Windows PowerShell
Triage Stage: UnreviewedAccepted
Type: BugCleanup/optimization

I'm not knowledgeable about Windows to confirm, but assuming some can confirm it, a documentation note seems okay.

comment:6 Changed 8 months ago by Carlton Gibson

#32439 was closed as a duplicate, with dumpdata failing due to the target file not explicitly setting the encoding.

Likely the solution here is to add to the Windows How-To that the default encoding can be set UTF8 now:

Go to Settings there's a "Use Unicode UTF-8 for worldwide language support", box in "Language" - "Administrative Language Settings" - "Change system locale" - "Region Settings".

comment:7 Changed 6 months ago by David Smith

Owner: changed from nobody to David Smith
Status: newassigned

comment:8 Changed 6 months ago by David Smith

Has patch: set

comment:9 Changed 5 months ago by Carlton Gibson

Triage Stage: AcceptedReady for checkin

comment:10 Changed 5 months ago by GitHub <noreply@…>

Resolution: fixed
Status: assignedclosed

In 0456d3e4:

Fixed #26721 -- Doc'd setting UTF-8 on Windows.

comment:11 Changed 5 months ago by Mariusz Felisiak <felisiak.mariusz@…>

In d6b6eda4:

[3.2.x] Fixed #26721 -- Doc'd setting UTF-8 on Windows.

Backport of 0456d3e42795481a186db05719300691fe2a1029 from main

Note: See TracTickets for help on using tickets.
Back to Top