Opened 8 years ago

Closed 15 months ago

#5419 closed New feature (wontfix)

Add a model fuzz testing method to the test framework

Reported by: adrian Owned by: kkubasik
Component: Testing framework Version: master
Severity: Normal Keywords: feature
Cc: d1fffuz0r@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Fuzz testing is when random, crazy data is thrown at some code to make sure it can handle crazy input. The Django test system should provide an assertModelPassesFuzzTest() method (or pick a better name), which would take a model class as its argument and do the following:

  • Insert a bunch of random data, including non-ASCII Unicode data, into a model instance and make sure the instance's __unicode__() doesn't throw UnicodeEncodeError or UnicodeDecodeError.
  • Call every custom model method that doesn't have alters_data=True (or maybe include those, too?) and make sure nothing silly happens. (In this case, it will be hard to determine what "silly" means, without some hints from the model method as to its expected output.)

Attachments (3)

fuzz.diff (5.0 KB) - added by frasern 8 years ago.
fuzz-docs.diff (2.0 KB) - added by frasern 8 years ago.
fuzz.2.patch (8.0 KB) - added by d1ffuz0r 3 years ago.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 8 years ago by adrian

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Accepted

comment:2 Changed 8 years ago by anonymous

  • Owner changed from nobody to anonymous
  • Status changed from new to assigned

comment:3 Changed 8 years ago by frasern

  • Owner changed from anonymous to frasern
  • Status changed from assigned to new

Last change was me, sorry.

Adrian raised this ticket, so I assume it is correct to set to "Accepted".

Changed 8 years ago by frasern

Changed 8 years ago by frasern

comment:4 Changed 8 years ago by frasern

  • Has patch set

The attached patch (plus documentation) introduces two new assertions:

  • assertModelUnicodePassesFuzzTest
  • assertFunctionPassesFuzzTest

The first of these, assertModelUnicodePassesFuzzTest, assigns random unicode data to each of an instance of the model's fields and checks that calling the instance's unicode method doesn't raise a UnicodeError

I couldn't see how it would be possible to have an assertModelPassesFuzzTest (as Adrian suggested) that calls each of a models methods, as it would be very awkward to specify, for each method, valid argument types, response types and exceptions. Instead I have created assertFunctionPassesFuzzTest that allows you to call each method individually specifying valid inputs and outputs.

comment:5 Changed 8 years ago by PhiR

  • Keywords feature added

comment:6 Changed 6 years ago by kkubasik

  • Owner changed from frasern to kkubasik

comment:7 Changed 4 years ago by gabrielhurley

  • Severity set to Normal
  • Type set to New feature

comment:8 Changed 4 years ago by julien

  • Easy pickings unset
  • Needs tests set

comment:9 Changed 4 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:10 Changed 3 years ago by aaugustin

Unicode-related problems and badly written unicode methods are less common than they used to be 5 years ago. I haven't encountered this class of problems frequently.

Given the lack of activity here, I suggest to abandon this ticket.

comment:11 Changed 3 years ago by d1ffuz0r

  • Cc d1fffuz0r@… added

Interesting feature. I want try to upgrade this patch for current version and to make tests for it.

Changed 3 years ago by d1ffuz0r

comment:12 Changed 3 years ago by d1ffuz0r

  • Needs tests unset

patch has been updated and tested in python 2.7.3/3.3.0. also added some tests.

comment:13 Changed 15 months ago by timo

  • Resolution set to wontfix
  • Status changed from new to closed

Agreed with Aymeric's comment from 19 months ago, it doesn't seem like this is something that Django needs to provide.

Note: See TracTickets for help on using tickets.
Back to Top