id	summary	reporter	owner	description	type	status	component	version	severity	resolution	keywords	cc	stage	has_patch	needs_docs	needs_tests	needs_better_patch	easy	ui_ux
7987	Leaks in Memory when using Iterator and saving the Model - Get Error Code 24 - 'Too Many Files Open'	chris@…	nobody	"I'm currently making an experimental spider, getting information from www.paixie.net.

I am using a Django model as my database connector and have found something really strange going on in the code.

At first, i thought it was because i was opening and saving files. However, after about a day of troubleshooting, I isolated the problem to the following code:

{{{
def download_all_product_images(image_dir):		
			
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.big_image.image) is '':
			print "".......Downloading big image @"", shoe.big_image
			shoe.big_image.image = download_product_image(image_dir, shoe.big_image.image_url, str(shoe.id), 'b')
			shoe.big_image.save()
	
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.medium_image.image) is '':
			print "".......Downloading medium image @"", shoe.medium_image
			shoe.medium_image.image = download_product_image(image_dir, shoe.medium_image.image_url, str(shoe.id), 'm')
			shoe.medium_image.save()


			#count = 1
			#for other_image in shoe.other_images.all().iterator():
				#if str(other_image.image) is '':
			#	print "".......Downloading image "", count, "" @ "", other_image
			#	other_image.image = download_product_image(image_dir, other_image.image_url, str(shoe.id), str(count))
			#	other_image.save()
			#	count = count + 1	
			
		print ""End Function""
}}}

When i run this, after about 1000 files are downloaded, I get an IOError, saying that i had opened too many files

I checked the /proc/<id>/fd directory and found that files were increasing when i save them to the database

---------------------------

However, the strangest thing i can't figure out is that if i swapped the two methods... memory only slightly increases:

{{{
def download_all_product_images(image_dir):		
		
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.medium_image.image) is '':
			print "".......Downloading medium image @"", shoe.medium_image
			shoe.medium_image.image = download_product_image(image_dir, shoe.medium_image.image_url, str(shoe.id), 'm')
			shoe.medium_image.save()
	
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.big_image.image) is '':
			print "".......Downloading big image @"", shoe.big_image
			shoe.big_image.image = download_product_image(image_dir, shoe.big_image.image_url, str(shoe.id), 'b')
			shoe.big_image.save()
	
}}}

Please let me know what is happening.....



"		closed	Database layer (models, ORM)	dev		invalid	Model, Iterator, Memory Leak		Unreviewed	0	0	0	0	0	0