Opened 8 years ago

Closed 6 years ago

#4924 closed (fixed)

Loading of compressed fixtures

Reported by: lars.yencken@… Owned by: jdunck
Component: Core (Serialization) Version: master
Severity: Keywords: compression, fixtures, feature
Cc: semente@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

Fixtures contain a lot of redundancy (in xml or json format), so they compress well. This patch adds detection and loading of bzip2 or gzip compressed fixtures, since these formats are supported natively in Python. It also contains some test cases.

An example of the utility of this patch: in my project, an initial_data.json fixture was around 4Mb and bzip2'ed down to ~150Kb, a much more reasonable size for storing in a repository. I'm sure there are plenty of projects where this is useful.

The tests pass, except for one which fails in the same way the existing fixture test fails.

Attachments (5)

compression.diff (6.5 KB) - added by Lars Yencken <lars.yencken@…> 8 years ago.
Patch minus binaries
fixture5.json (405 bytes) - added by Lars Yencken <lars.yencken@…> 8 years ago.
Fixture 5 (tests)
fixture4.json (416 bytes) - added by Lars Yencken <lars.yencken@…> 8 years ago.
Fixture 4 (should be bzip2'ed)
compression.2.diff (6.8 KB) - added by Lars Yencken <lars.yencken@…> 8 years ago.
Updated patch
compressed-fixture.tar.gz (3.7 KB) - added by jdunck 7 years ago.
Patch and new files for compressed fixture support.

Download all attachments as: .zip

Change History (19)

comment:1 Changed 8 years ago by lars.yencken@…

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Sorry, can't add patch or tests. I'm being spam filtered =(

comment:2 Changed 8 years ago by anonymous

  • Has patch unset

Changed 8 years ago by Lars Yencken <lars.yencken@…>

Patch minus binaries

Changed 8 years ago by Lars Yencken <lars.yencken@…>

Fixture 5 (tests)

Changed 8 years ago by Lars Yencken <lars.yencken@…>

Fixture 4 (should be bzip2'ed)

comment:3 Changed 8 years ago by Lars Yencken <lars.yencken@…>

  • Has patch set

Ok, got past the spam filter now. There's a trac bug which won't let me attach bzip2'ed files. In the tests/modeltests/fixtures/fixtures folder, we add:

fixture4.json.bz2
fixture5.json
fixture5.json.bz2

Fixture 5 is just a copy of fixture 1, to ensure an error gets raised if there is ambiguity as to which fixture file is meant.

comment:4 Changed 8 years ago by Simon G. <dev@…>

  • Triage Stage changed from Unreviewed to Ready for checkin

I've got the tests ready to go, but can't attach them either (keep hitting this trac bug), I'll email them on if a core wants them.

Changed 8 years ago by Lars Yencken <lars.yencken@…>

Updated patch

comment:5 Changed 8 years ago by Lars Yencken <lars.yencken@…>

I noticed a few small problems, and updated the patch to fix them. The problems were:

  • It still checked for all the different compression types, even if a unique filename was specified. So fixture5.json.bz2 and fixture5.json would clash, even if you specifically asked for fixture5.json.bz2.
  • The test case omitted checking for data which gets loaded from the initial_data fixture.
  • The coding style used alternatingCaps for names, instead of using_underscores.

Have a look at the new one, which should be better.

Thanks,
Lars

comment:6 Changed 8 years ago by russellm

  • Triage Stage changed from Ready for checkin to Accepted
  1. Patch needs to be updated following the management.py refactor.
  2. Patch should handle zip files in addition to bz2 and gz.
  3. More extensive tests are required, including:
    1. Need to test explicit compression scheme naming, not just discovery.
    2. Need to check the edge cases where you name a compression scheme, but not a fixture format.
  4. Documentation of the new capabilities are required.

comment:7 Changed 8 years ago by Guilherme M. Gondim (semente) <semente@…>

  • Cc semente@… added

(only subscribing)

comment:8 Changed 7 years ago by mcroydon

  • Owner changed from nobody to mcroydon

I'll see what I can do about russellm's suggestions.

comment:9 Changed 7 years ago by mcroydon

Need more caffeine, dropping this for the time being.

comment:10 Changed 7 years ago by mcroydon

  • Owner changed from mcroydon to nobody

comment:11 Changed 7 years ago by russellm

  • Keywords feature added
  • Needs documentation set
  • Needs tests set
  • Patch needs improvement set

Changed 7 years ago by jdunck

Patch and new files for compressed fixture support.

comment:12 Changed 7 years ago by jdunck

  • Needs documentation unset
  • Needs tests unset
  • Owner changed from nobody to jdunck
  • Patch needs improvement unset
  • Status changed from new to assigned

I've uploaded a new patch addressing the concerns russellm raised in his 09/13/07 comment.

comment:13 Changed 7 years ago by jdunck

  • Triage Stage changed from Accepted to Ready for checkin

comment:14 Changed 6 years ago by jacob

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [9527]) Fixed #4924: added support for loading compressed fixtures. Thanks to Lars Yencken and Jeremy Dunck.

Note: See TracTickets for help on using tickets.
Back to Top