Too many open files.

[eltonplima@…]

I'm doing a migration between two applications in django, everything was going normal until I got a class that has a field of type FileField. The import function normally to a given time in the system raises the following exception:

Original exception was:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 382, in execute
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 196, in run_from_argv
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 232, in execute
  File "/home/eltonplima/workspace/ci/ci/migracao/management/commands/import_protocolo.py", line 291, in handle
  File "/home/eltonplima/workspace/ci/ci/migracao/management/commands/import_protocolo.py", line 139, in protocolo_relatorio_arquivo
  File "/home/eltonplima/workspace/ci/ci/migracao/protocolo/protocolos/models.py", line 168, in __unicode__
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/files.py", line 46, in _get_file
  File "/usr/local/lib/python2.7/dist-packages/django/core/files/storage.py", line 33, in open
  File "/usr/local/lib/python2.7/dist-packages/django/core/files/storage.py", line 156, in _open
IOError: [Errno 24] Too many open files: u'/home/eltonplima/workspace/ci/ci/media/uploads/estado_a/municipio_a/cliente_a/2010/03_marco/arquivo.pdf'

That is, django is opening each file but is not closing, so I'm forced to close it manually, in theory django should not automatically close the file?

My code:

conversoes = ArquivoProtocolo.objects.using('protocolo').all()
if conversoes.count() > 0:
    for conversao in conversoes:
        ProtocoloRelatorioArquivo.objects.get_or_create(
         id=conversao.id,
         tipo_relatorio=TipoRelatorio.objects.get(id=conversao.tipo_relatorio.id),
         arquivo=conversao.arquivo,
         protocolo_relatorio=ProtocoloRelatorio.objects.get(id=conversao.protocolo_arquivo.id)
        )
        conversao.arquivo.close()# This should really be necessary?

[Aymeric Augustin]

Generally this isn't an issue because you don't open thousands of files in a query. The file objects can be garbage collected and the file descriptors closed as soon as the request is finished.

Django can't close the files automatically because it doesn't know when you're done with the file. Specifically we can't use the context manager.

So I suppose it's just a gotcha we should document.

[Chris Beaven <smileychris@…>]

In [ffa6d95f65363b7f4f9047ab11561880be29049a]:

Fixed #18154 -- Documentation on closing File objects and best practices