Opened 9 years ago

Closed 2 years ago

#5419 closed New feature (wontfix)

Add a model fuzz testing method to the test framework

Reported by: Adrian Holovaty Owned by: Kevin Kubasik
Component: Testing framework Version: master
Severity: Normal Keywords: feature
Cc: d1fffuz0r@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Fuzz testing is when random, crazy data is thrown at some code to make sure it can handle crazy input. The Django test system should provide an assertModelPassesFuzzTest() method (or pick a better name), which would take a model class as its argument and do the following:

  • Insert a bunch of random data, including non-ASCII Unicode data, into a model instance and make sure the instance's __unicode__() doesn't throw UnicodeEncodeError or UnicodeDecodeError.
  • Call every custom model method that doesn't have alters_data=True (or maybe include those, too?) and make sure nothing silly happens. (In this case, it will be hard to determine what "silly" means, without some hints from the model method as to its expected output.)

Attachments (3)

fuzz.diff (5.0 KB) - added by Fraser Nevett 9 years ago.
fuzz-docs.diff (2.0 KB) - added by Fraser Nevett 9 years ago.
fuzz.2.patch (8.0 KB) - added by Roman Gladkov 4 years ago.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 9 years ago by Adrian Holovaty

Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedAccepted

comment:2 Changed 9 years ago by anonymous

Owner: changed from nobody to anonymous
Status: newassigned

comment:3 Changed 9 years ago by Fraser Nevett

Owner: changed from anonymous to Fraser Nevett
Status: assignednew

Last change was me, sorry.

Adrian raised this ticket, so I assume it is correct to set to "Accepted".

Changed 9 years ago by Fraser Nevett

Attachment: fuzz.diff added

Changed 9 years ago by Fraser Nevett

Attachment: fuzz-docs.diff added

comment:4 Changed 9 years ago by Fraser Nevett

Has patch: set

The attached patch (plus documentation) introduces two new assertions:

  • assertModelUnicodePassesFuzzTest
  • assertFunctionPassesFuzzTest

The first of these, assertModelUnicodePassesFuzzTest, assigns random unicode data to each of an instance of the model's fields and checks that calling the instance's unicode method doesn't raise a UnicodeError

I couldn't see how it would be possible to have an assertModelPassesFuzzTest (as Adrian suggested) that calls each of a models methods, as it would be very awkward to specify, for each method, valid argument types, response types and exceptions. Instead I have created assertFunctionPassesFuzzTest that allows you to call each method individually specifying valid inputs and outputs.

comment:5 Changed 9 years ago by Philippe Raoult

Keywords: feature added

comment:6 Changed 7 years ago by Kevin Kubasik

Owner: changed from Fraser Nevett to Kevin Kubasik

comment:7 Changed 5 years ago by Gabriel Hurley

Severity: Normal
Type: New feature

comment:8 Changed 5 years ago by Julien Phalip

Easy pickings: unset
Needs tests: set

comment:9 Changed 5 years ago by Aymeric Augustin

UI/UX: unset

Change UI/UX from NULL to False.

comment:10 Changed 4 years ago by Aymeric Augustin

Unicode-related problems and badly written unicode methods are less common than they used to be 5 years ago. I haven't encountered this class of problems frequently.

Given the lack of activity here, I suggest to abandon this ticket.

comment:11 Changed 4 years ago by Roman Gladkov

Cc: d1fffuz0r@… added

Interesting feature. I want try to upgrade this patch for current version and to make tests for it.

Changed 4 years ago by Roman Gladkov

Attachment: fuzz.2.patch added

comment:12 Changed 4 years ago by Roman Gladkov

Needs tests: unset

patch has been updated and tested in python 2.7.3/3.3.0. also added some tests.

comment:13 Changed 2 years ago by Tim Graham

Resolution: wontfix
Status: newclosed

Agreed with Aymeric's comment from 19 months ago, it doesn't seem like this is something that Django needs to provide.

Note: See TracTickets for help on using tickets.
Back to Top