IPv6 address field support

[mattimustang@…]

Hi the attached patch adds a new field called IP6AddressField that allows the entry of IPv6 addresses. It has a validator too called isValidIPAddress6 and I've also added another validator called isValidIPAddress that checks whether an address is IPv4 or IPv6.

Tested on postgres backend.

I initially did this as a replacement for the existing IPAddressField but it breaks backward compatibility because all the non-postgres backends have to have their fields sizes increased.

It's up to you whether you want to do this or not.

[mattimustang@…]

IP6AddressField patch

[Jacob]

This looks good, but there are a few things missing:

• Naming is inconsistant; some places use "ip6" and some "ipv6". I'd prefer "ipv6" (so "IPv6AddressField" and such).
• There's a bug in the maxlength of IP6AddressField; it should be 39, not 15.

I'm marking wontfix; if you can fix those things with a new patch I'll apply it.

[mattimustang@…]

Updated IPv6 Field patch

[mattimustang@…]

I've cleaned up naming and fixed the bug in the original patch. All fields and validators are clearly marked as being IPv4 or IPv6 now with the exception of the original IPAddressField. Renaming this would break backwards compatibility.

[Seer]

Hi all
im fine, gl all!

[Russell Keith-Magee]

This needs a lot more work before it is ready for checkin. There are no tests for the validators, no documentation of the new field, and it doesn't provide any integration with newforms.

[Simon G. <dev@…>]

Bumping back to wontfix. The current implementation needs a lot of work to get it up to speed, and the fact that this hasn't been touched in over a year suggests that people aren't really crying out for a dedicated field here.

[Johann Queuniet <johann.queuniet@…>]

IPv4/IPv6 field with doc, tests and integration with new/oldforms

[Johann Queuniet <johann.queuniet@…>]

Since I don't want to see this sinking into oblivion, I'm taking on the task.

Here is a patch I've been using for a few months. It has documentation, integration with new/oldforms and a few tests for the newforms field.
IPy is used to validate, since the compressed IPv6 form is near impossible to accurately validate with regexps. I'd be happy if somebody could prove me wrong, of course.

Note that this field can store both IPv4 and IPv6 addresses. Which should be a good thing since I don't think many people are running pure IPv6 stacks out there. But that makes the field name a bit misleading. Any suggestion?

[Matthew Flanagan <mattimustang@…>]

Hi Johann,

I'm very impressed with your patch. Well done! However I don't think IPy is necessary to perform IPv6 address validation, ​http://code.djangoproject.com/attachment/ticket/811/ipv6field.diff has regexps that will do that. Not that I object to using IPy but the Django core developers might object to introducing a third-party package.

I'm +1 for merging IPv4 and IPv6 fields into a single field.

[Johann Queuniet <johann.queuniet@…>]

I considered your patch but it actually fails on some edge cases, like 1::2:3:4:5:6:7:8:9 or 0:0:0:0:0:0:0::ffff:1.2.3.4. To elaborate a little on what I said, I don't see any way to *accurately* validate the compressed form with regexp, short from using a separate expression for each possible position of the dual colon shortcut. Nearly times two with the IPv4 compatibility notation. But I'm certainly not a regexp god so yes, feel free to propose a better way and I'll be glad to learn something new :)

Or instead, an alternate solution: getting ride of the regexps altogether and checking if an address converts cleanly to a binary string form. But wouldn't that equals to incude a fair part of IPy in the framework?

I'm also reluctant to introduce yet another third-party dependency, but there is a choice between accuracy and convenience here. Aside from the root of the problem (invalid data), convenience is problematic with PostgreSQL since the inet field will perform a formal validation and either behave eraticaly or throw a ProgrammingException.

To sum things up, it's either :

1. allowing invalid data to pass through
2. using something like fifteen regexps
3. adding a part of IPy to the framework
4. asking people to install IPy

If I had to choose, I'd be +1 for the third, -1 for the first and +0 for whatever else. This patch implements the last one since it was the easiest way to get something done. But since it doesn't seem like a trivial issue, we should push this issue on the mailing list. I'm okay to rewrite the validation according to whatever consensus is reached.

[Marc Fargas]

Replying to Johann Queuniet <johann.queuniet@gmail.com>:

I considered your patch but it actually fails on some edge cases, like 1::2:3:4:5:6:7:8:9 or 0:0:0:0:0:0:0::ffff:1.2.3.4. To elaborate a little on what I said, I don't see any way to *accurately* validate the compressed form with regexp, short from using a separate expression for each possible position of the dual colon shortcut. Nearly times two with the IPv4 compatibility notation. But I'm certainly not a regexp god so yes, feel free to propose a better way and I'll be glad to learn something new :)

You could always change the "::" to ":0000:" before validation, the address is the same anyway. Or even:

• Change "::" to ":0000:"
• Validate
• If valid, take the original address as the good one

[Johann Queuniet <johann.queuniet@…>]

It's a bit more complicated since a :: can act as a shortcut for any number of 0s. But that's part of what I meant by "adding a part of IPy". You can read that as doing some normalization work before the actual validation.

I was in the mood for a first try, so here it is.

def normalize(addr):
    # Some basic error checking
    if addr.count('::') > 2 or addr.find(':::') > -1:
        raise ValueError

    li = addr.split(':')
    nbfull = len([elem for elem in li if elem != ''])

    if nbfull == 8:
        # No need to bother
        return ':'.join(full)
    if nbfull > 8:
        # Has to be invalid anyway
        raise ValueError

    # Begin normalization of ::
    start, end, index = (None, None, 0)
    for elem in li:
        if elem == '':
            if start is None:
                start = index
                end = index
            else:
                end = index
        index += 1
    pad = 8 - nbfull
    if end != start:
        li[start:end-start+1] = ['0'] * pad
    else:
        li[start] = '0'
        if pad > 1:
            li[start:1] = ['0'] * (pad - 1)
    return ':'.join(li)

I'll try to add support for IPv4 and put a new patch together by the week-end. The old one shouldn't apply cleanly anymore since FilePathField has been added to newforms/fields.py

[Johann Queuniet <johann.queuniet@…>]

Same as the 02/02/08 patch, without IPy

[Johann Queuniet <johann.queuniet@…>]

Well, here we are. I added a normalization function in django.utils.http, since I didn't want to create a new module just for that. It should get ride of tricky things like IPv4 mapped addresses and this :: shortcut, without the IPy dependency this time.

[Marc Fargas]

All the concerns that were raised for this ticket (when marked "Needs improvement") seem to have been solved with the latest patch. Even the inclusion of documentation and tests.
The patch still applies cleanly and passes tests so I hope no triager will feel upset for me moving this to Ready for checkin ;)

[hwaara@…]

There's an error in the patch: the new field added in django/db/backends/mysql/creation.py should be named 'IP6AddressField' and not 'IPAddressField' (which is a duplicate of the existing 'IPAddressField').

[Johann Queuniet]

IPv6 patch for trunk, revision 9781

[Johann Queuniet]

IPv6 patch for trunk, revision 9781

[Johann Queuniet]

Previous patch updated for current trunk, revision 9781. Nothing was changed, I just moved code around and deleted all the oldforms cruft from pre-1.0.

Don't know why but Trac won't show the patch as it usually does. Download as original format seems to work though, so I'll just leave it as it is.

[mludvig@…]

Thanks for the patch! Any chance to have this integrated to the official Django tree? There seem to be no objections from Django core devs and the patch is in a "Ready for checkin" stage. What is holding it back?

[Adam Nelson]

Somehow the latest patches seem to be gone.

[Adam Nelson]

Patch is there - just a bug in Trac. Anyway, it does not apply cleanly.

[Johann Queuniet <johann.queuniet@…>]

patch bump to r13294, updated to 1.2 validators

[Johann Queuniet <johann.queuniet@…>]

Also, a few cosmetic changes to address prior remark on IPv6/IP6 namming inconsistance. All objects and functions should use IPv6 as prefix now.

[Sasha Romijn]

Am I the only one who finds it very confusing that IPv6AddressField supports IPv4 addresses?

I strongly agree that we should have a field that accepts both, but I'm not so sure of the naming here. IPAddressField would be the best name, but obviously we can't use that. I also don't have another suggestion (yet). But, once we use this, we probably can't change the name again, for backwards compatibility.

Any other thoughts?

[Johann Queuniet <johann.queuniet@…>]

I totally agree with you, but since the neutral/logical name is already taken, it's hard to think of anything not utterly confusing.

[Sasha Romijn]

I've been asking around and came up with, in order of preference: IP46AddressField, IPv46AddressField, ComboAddressField, GenericAddressField, InternetAddressField, HostAddressField

I prefer the IPv4-6 specific names, because we might want to support a different type of internet address in 5 years.

[Łukasz Rekucki]

I think this still needs some polish:

1. There's a typo in django/core/validators:

   validate_ipv6_address = IPv6Validator(_(u'Enter a valid IPv4 address.'), 'invalid')  
   

   It probably should read: "Enter a valid IPv4/IPv6 address.". OTOH, the message in the form field is "Enter a valid IP address.". This needs to be consistent.

2. ipv6_normalize() doesn't really fit in django.utils.http; There are utils modules with less code, so creating a one for IP won't hurt.
3. The name is really confusing. Also, you can't create a field that only accepts IPv6 addresses. Maybe instead of a seperate field, a flag could be added to the existing IPAddressField:

     IPAddressField() # accepts only IPv4, 15 chars in DB (the commons case now, backward compatible)
     IPAddressField(allow6=True) # accepts both IPv4 and IPv6, 39 chars 
     IPAddressField(allow6=True, allow4=False) # accepts only IPv6, 39 chars (the future)
   

   When everyone switches to IP6, we can then change the default flag values.

[Sander Steffann]

Replying to erikr:

I prefer the IPv4-6 specific names, because we might want to support a different type of internet address in 5 years.

It will be a bit sooner than 5 years... The IPv4 address pools will run out in 2011. An IPv6 compatible field would be really appreciated before then.

Thanks for working on this!
Sander

[Sander Steffann]

Implementing suggested changes

[Sander Steffann]

Please check if the new patch is acceptable.

Also: it would be great if this could be included in 1.3. What are the chances for that? Please take into account that the global and regional IPv4 address pools will run out in 2011.

Thanks,
Sander

[Łukasz Rekucki]

Django 1.3 is in beta stage right now, so it's feature frozen.

[Preston Timmons]

I tested this patch. It applies cleanly and has docs and tests. I also updated the milestone to 1.3.

Can a core committer please give feedback or mark this RFC?

[Karen Tracey]

This is not ready for checkin. In a quick review I notice:

1. No docs in the latest patch.
2. Latest patch adds flags to an existing model field that determines what length the underlying DB char field has...I'm not sure that is a good idea. How to resolve the naming issue for the new field is probably best discussed on django-developers.
3. It looks like while the model field guards against nonsense spec of neither address form supported, the form field does not and as a result uses no validators?

We're hopefully going to ship 1.3 in hours or days, there is no way this could make 1.3.

[Jannis Leidel]

While I think the patch I just attached is fine for now, I've also just found this: ​http://support.freenas.org/browser/freenas/trunk/gui/contrib/IPAddressField.py

It's neat because it uses Google's ipaddr package (​http://code.google.com/p/ipaddr-py/) that allows all kinds of crazy things with the resulting objects. Anyone an opinion if this makes sense?

[Sasha Romijn]

@Jezdez: good suggestion, but I thought Django was very much against depending on external libraries?

[Jannis Leidel]

Replying to erikr:

@Jezdez: good suggestion, but I thought Django was very much against depending on external libraries?

Yeah, we'd have to bundle it.

[Sasha Romijn]

I will work on this during the djangocon.eu sprints, and hope to get it into a ready to checkin stage.

[Sasha Romijn]

Patch for my IPv46AddressField

[Sasha Romijn]

I have attached a new patch for this. I believe this addresses all issues.

Unlike some of the previous patches, this adds one new field, the IPv46AddressField, that supports both IPv4 and IPv6 addresses. For me, it is very unpleasant to have to use two fields, one for IPv4 and one for IPv6, every time I need to store an IP address. The form field has optional parameters to limit it to only IPv4 or only IPv6. So it is still trivial to create a field for IPv6 addresses only, if you want that. With it's IPv4-only mode, this can therefore replace IPAddressField

For naming, I settled on IPv46Address for the reasons:

• The field will support both IPv4 and IPv6
• Any kind of generic naming may be unclear
• Generic naming may land us in the same issue as with IPAddressField: due to some new invention, the field is incomplete, but we can't change it
• IPv46AddressField is a bit more readable than IP46AddressField, and fits closer with existing convention

The form field, in it's to_python(), normalises the value that it has, so "2001:0::2" will be turned into "2001::2", as recommended by the RFCs. The rationale for this behaviour is that if one user enters "2001:0::2" and another enters "2001::2", in most cases, I want that to be turned into the same record. The behaviour can be disabled with an optional parameter.

I have added docs and pretty extensive tests, so I believe the patch is complete.

[Klaas van Schelven]

Hi Erik,

My 2¢ on your latest patch, let's make sure to get it in today or tomorrow. A great candidate for the bottle of booze that will be handed out for "oldest ticket fixed" on the sprints.

• My eye keeps tripping up on IPv46AddressField; I'd probably go with IPv6AddressField even though it's including IPv4. I don't think we'll have a new protocol in the near future given the huge address space so any version of "Generic" or "NG/New" IpAddressField would be fine with me. (BikeShedding)
• The RegEx is huge and uncommented, other than some tests there is no way to verify that it works.
• Just a wild idea: we're already using 'socket' for conversion, could we not just use that for validation as well? E.g. catch exceptions; I'm impartial in the reuse/bundle discussion.
• skip_ipv6_normalization => no_ipv6_normalization (BikeShedding)
• only_ipv4, doesn't feel very "Django". I think the pattern would be to make the validators (IPv4, IPv6 and the version for both) available and let the user pass any of those 3 as the value of 'validator'. The version for both would be default I suppose.
• _(u'Enter a valid IP address.') => _(u'Enter a valid IPv4 or IPv6 address.')
• Split up the tests, wherever there's a newline, I'd introduce some meaning in the separate test's name.
• I've not run the tests yet, so I can't vouch for that as we speak

Klaas

[Klaas van Schelven]

Correcting myself.

skip_ipv6_normalization=False

should be changed to

normalize_ipv6=True

[Klaas van Schelven]

Erik: one more thing; you're currently catching the gaierror, I think that should be propagated, because at that point it really _is_ an exception since you're trying to normalize and that fails.

You could than use the try/except block in the validator (and return False on catch).

[Sasha Romijn]

GenericIPAddressField patch

[Sasha Romijn]

I have attached a updated patch.

Fixes made in this:

• Name changed to GenericAddressField
• _(u'Enter a valid IP address.') => _(u'Enter a valid IPv4 or IPv6 address.')
• IPv6 normalization configuration removed - enabling this would lead to inconsistent behaviour of uniqueness errors between postgresql and other DB backends
• only_ipv4, etc. have been replaced by a validator option, as suggested
• The regular expression and call to getaddressinfo have been replaced by the new django.utils.ipv6, based mostly on ipaddr-py
• Split up the tests for the form field, as suggested
• I have also added validation to the model - previously the form field would be strict, but the model would have happily excepted "hamster" as a IP address - this would also cause serious issues with the missing normalization
• The model now also has a proper get_prep_value which means a search for an IPv6 address is normalized before it is sent to the database
• There is now an optional parameter to unpack mapped IPv4 addresses - by default disabled, to have the default of least surprise - but I would certainly use it in some cases
• A lot of tests and some documentation were added and updated to reflect all of the above

[Sasha Romijn]

GenericIPAddressField patch

[Klaas van Schelven]

Nice...

Last remark: you're still importing socket even though you're not using it anymore.

I'll run the tests for good measure.

[Sasha Romijn]

removed unneeded import of socket in fields.py

[Klaas van Schelven]

Tests are running, I'm marking this RFC, will doublecheck w/ one of the core developers here.

[Jannis Leidel]

In [16366]:

Fixed #811 -- Added support for IPv6 to forms and model fields. Many thanks to Erik Romijn.

[Dan McGee]

What's the design decision on updating django.contrib.comments.models to utilize the new IPv6-capable address field? It appears to be the only core Django application utilizing this field.

It assumes the value of REMOTE_ADDR can be coerced into the field, as you can see in this line from django/contrib/comments/views/comments.py:

comment.ip_address = request.META.get("REMOTE_ADDR", None)

This currently populates PostgreSQL-backed instances correctly if hooked up to an IPv6-capable webserver, but all other databases will fail or truncate (perhaps even silently!) any non-IPv4 value exceeding 15 characters, such as ::ffff:192.168.100.1. The prudent decision to me seems to be to modify the Comment model to use ip_address = models.IPAddressField(_('IP address'), unpack_ipv4=True, blank=True, null=True).

If I should bring this up in a new bug report, let me know, but I have had zero luck getting any of those ever noticed or looked at.

[Aymeric Augustin]

Yes, please create a new ticket. Posting on a closed ticket will often be ignored, but it's guaranteed that at least one person will take a look at new tickets.

[Dan McGee]

No problem- #16302 "Ensure contrib (namely comments) is IPv6 capable" has been created.

[onelson]

I realize this is a bit beside the point, but in case it's useful I recently put out a simple model field:
​https://bitbucket.org/onelson/django-ipyfield

As for my case, I didn't have a problem delegating the validation of input to IPy and I also needed a single field able to handle the storage of both v4 and v6 addresses. To dodge the potential storage issues on the db side, I leverage IPy to store all addresses as longs.

I guess this isn't good for core, or contrib dependencies because of the dependency on IPy, but I just wanted to let you know!