hexed strings in common passwords database are not handled

[Michel Le Bihan]

Hello,

The common passwords database file (​https://github.com/django/django/blob/main/django/contrib/auth/common-passwords.txt.gz) used by CommonPasswordValidator contains hexed strings like $hex[617364666a6b6c3a] on line 1679. That decodes to asdfjkl: which I believe is a common password that was intended to be included in the database. Another example is $hex[2623323333363a] on line 8616 that decodes to &#2336:. I see that ​https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce contains the line 50334:72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b:$HEX[617364666a6b6c3a] and echo -n 'asdfjkl:' | sha1sum produces 72aff1cfd90a90fd4174eb6dfdff5df7bbbe7e5b. CommonPasswordValidator does not handle those hexed strings which I believe is wrong.

I propose to update the database file to decode the hexed values and remove those that obviously can't be entered by a user.

[Sarah Boyce]

Thank you for the ticket
I think you are referring to there passwords ​asdfjkl; and ​ठ (; not :) which are validating correctly, the others are not in the list

[Michel Le Bihan]

Hello,

I'm referring to those with : at the end. Please just search $hex in that file and you will notice the issue.

[Michel Le Bihan]

Please see this line: ​https://gist.github.com/roycewilliams/226886fd01572964e1431ac8afc999ce#file-pwnedpasswords-v6-top20k-txt-L1728

[Sarah Boyce]

Ah thank you!
Folks can de-hex using this tool: ​https://www.rapidtables.com/convert/number/hex-to-ascii.html

Here is a regression test:

tests/auth_tests/test_validators.py

@@ -273 +273 @@
             CommonPasswordValidator().validate("godzilla")
         self.assertEqual(cm.exception.messages, [expected_error])
 
+    def test_common_hexed_codes(self):
+        expected_error = "This password is too common."
+        common_hexed_passwords = ["asdfjkl:", "&#2336:"]
+        for password in common_hexed_passwords:
+            with self.subTest(password=password):
+                with self.assertRaises(ValidationError) as cm:
+                    CommonPasswordValidator().validate(password)
+                self.assertEqual(cm.exception.messages, [expected_error])
+
     def test_validate_custom_list(self):
         path = os.path.join(

[Michel Le Bihan]

I sent a patch that unhexes the values in the database: ​https://github.com/django/django/pull/19155

[Raphael Gaschignard]

While most of the $hex entries are removed, there's still one entry that's just $hex. I _believe_ this is an oversight, and not intentional.

[Raphael Gaschignard]

Per discusison in the PR, seems like this is actually part of the dataset.

[Sarah Boyce <42296566+sarahboyce@…>]

In 727731d:

Fixed #36179 -- Unhexed entries and removed duplicates in auth/common-passwords.txt.gz.

[Sarah Boyce <42296566+sarahboyce@…>]

In cb2ab4e:

[5.2.x] Fixed #36179 -- Unhexed entries and removed duplicates in auth/common-passwords.txt.gz.

Backport of 727731d76d9dfd5304d536478d862778f6dd6d9b from main.