Opened 8 years ago

Closed 6 years ago

Last modified 5 years ago

#8794 closed (fixed)

Profanity filter suffers from the Scunthorpe problem

Reported by: Daniel Pope <dan@…> Owned by: nobody
Component: contrib.comments Version: master
Severity: Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

The implementation of the profanity filter suffers from the Scunthorpe Problem; ie. that it considers the town of Scunthorpe, amongst other innocuous words, to be profane.

Profanity filtering is A Hard Problem, and naïve solutions like this one cause frustrating problems to end-users.

Checking the current profanities list for false positives in a couple of word lists I had to hand also yields:

gobbledegook
snigger
Brushite
Cushite
Niggerhead
Peshito
Peshitto
Shittah
Shittah tree
Shittim
Shittim wood
Shittle
Shittlecock
Shittleness

Obviously proper names are not in my dictionary, but they cause frequent and often more annoying problems.

I suggest to disable the filter by default so that scope of the problem is limited, and at the very least the filter must be restricted to re.match(r'\b' + word + '\b'). Users who need stricter profanity filters should have the responsibility for doing so, and potentially annoying their users themselves. Django should not be doing it for them.

Attachments (2)

8794.diff (2.6 KB) - added by Thejaswi Puthraya 8 years ago.
git-patch of the latest checkout
t8794.diff (1.5 KB) - added by Luke Plant 6 years ago.
Fix for the actual Scunthorpe bug.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 8 years ago by Thejaswi Puthraya

Triage Stage: UnreviewedDesign decision needed

It's quite easy to toggle the COMMENTS_ALLOW_PROFANITIES setting. I am a -0 on this. Am marking this as a DDN.

comment:2 Changed 8 years ago by Adrian Holovaty

Triage Stage: Design decision neededAccepted

I have come to believe that we shouldn't be distributing a list of profanities with Django. Problem is, we're committed to backwards compatibility -- so we'll have to leave those in until the next major release.

Changed 8 years ago by Thejaswi Puthraya

Attachment: 8794.diff added

git-patch of the latest checkout

comment:3 Changed 6 years ago by Thejaswi Puthraya

milestone: 2.0

Check #6290. This ticket has to be pushed for the next major revision ie 2.0.

comment:4 Changed 6 years ago by Russell Keith-Magee

milestone: 2.01.3

This needs to be handled with our usual deprecation plan rather than just removing it. Bumping to the 1.3 release following a recent discussion on django-dev

Changed 6 years ago by Luke Plant

Attachment: t8794.diff added

Fix for the actual Scunthorpe bug.

comment:5 Changed 6 years ago by Adrian Holovaty

Resolution: fixed
Status: newclosed

(In [13996]) Set default PROFANITIES_LIST setting to an empty tuple. This is technically backwards-compatible if you rely on a Web framework providing you with a woefully incomplete list of naughty words. Fixes #8794

comment:6 Changed 5 years ago by Jacob

milestone: 1.3

Milestone 1.3 deleted

comment:11 Changed 5 years ago by Aymeric Augustin

In [16935]:

Fixed #17012 - Removed references to the 'hasNoProfanities' validator. Refs #8794.

Note: See TracTickets for help on using tickets.
Back to Top