Clarify "memory_cost" of Argon2PasswordHasher differs from command line utility

[Si Feng]

The memory_cost doesn't seems to be consistent with the reference C implementation (​https://github.com/P-H-C/phc-winner-argon2), where the m actually means "Sets the memory usage of 2^{N KiB}

(default 12)", not plain "KiB". I observed the latter definition in argon2-cffi too and thus it could be due to that lib, but this is really confusing, as other libraries (for example, the Ruby binding) are consistent with the reference implementation. The problem of this inconsistency is that the "plain" version is usually much bigger than the "right" version and since there is no way to distinguish between the two, once Django 1.10 is shipped, it would become really hard if it is decided to use the "right" version in the future.

[Tim Graham]

Bas, could you offer your input here?

[Bas Westerbaan]

This is an inconsistency in the reference C implementation itself. The specification and the reference C implementation do not limit themselves to power of two memory_cost's. For some unknown reason, the commandline util which is shipped, does limit itself to power of two. In the source code of the commandline util (src/run.c) we find

m_cost = ARGON2_MIN(UINT64_C(1) << input, UINT32_C(0xFFFFFFFF));

I see two possible solutions.

The first one is to use power-of-two memory_cost's in Argon2. We should then also rename memory_cost to memory_cost_exp to reduce confusion the other way around. A big drawback is that we loose fine-grained control over the running time of Argon2. Every increase in memory_cost_exp will (at the very least) double the running time of the hash. (Recall that it is strongly advised to increase memory_cost instead of time_cost, to strengthen the hash.)

The second solution is to ask the Argon2 developers to change their util and add a second option -M which allows one to specify an arbitrary memory_cost.

[Si Feng]

Replying to bwesterb:

This is an inconsistency in the reference C implementation itself. The specification and the reference C implementation do not limit themselves to power of two memory_cost's. For some unknown reason, the commandline util which is shipped, does limit itself to power of two. In the source code of the commandline util (src/run.c) we find

m_cost = ARGON2_MIN(UINT64_C(1) << input, UINT32_C(0xFFFFFFFF));

I see two possible solutions.

The first one is to use power-of-two memory_cost's in Argon2. We should then also rename memory_cost to memory_cost_exp to reduce confusion the other way around. A big drawback is that we loose fine-grained control over the running time of Argon2. Every increase in memory_cost_exp will (at the very least) double the running time of the hash. (Recall that it is strongly advised to increase memory_cost instead of time_cost, to strengthen the hash.)

The second solution is to ask the Argon2 developers to change their util and add a second option -M which allows one to specify an arbitrary memory_cost.

Thanks for the investigation. Another possible issue is that the hash_len is currently harded-coded to use argon2.DEFAULT_HASH_LENGTH, which is 16 for now. I'm not sure if such value could change / may be changed in the future, and if that happens, would the validation fail. Haven't looked into validation logic thus don't know if the hash value needs to be 100% match.

[Tim Graham]

Is this a release blocker for Django 1.10? I'm not clear what the next steps are.

[Bas Westerbaan]

Replying to timgraham:

Is this a release blocker for Django 1.10? I'm not clear what the next steps are.

In my opinion this shouldn't be a blocker: it only affects those who choose a configuration different from the default. If you want, I could write a PR tomorrow which adds a mention in the docs and code that the memory_cost attribute we use is the same as in the API, but different from the commandline utility.

[Tim Graham]

Sounds good, thanks!

[Bas Westerbaan]

Here is the PR: ​https://github.com/django/django/pull/6655

[Tim Graham <timograham@…>]

In 9407cc9:

Fixed #26635 -- Clarified Argon2PasswordHasher's memory_cost differs from command line utility.

[Tim Graham <timograham@…>]

In c6aa941:

[1.10.x] Fixed #26635 -- Clarified Argon2PasswordHasher's memory_cost differs from command line utility.

Backport of 9407cc966b02e5ef69b7f561a6b972aff5d9c2e0 from master