Leaks in Memory when using Iterator and saving the Model - Get Error Code 24 - 'Too Many Files Open'

[chris@…]

I'm currently making an experimental spider, getting information from www.paixie.net.

I am using a Django model as my database connector and have found something really strange going on in the code.

At first, i thought it was because i was opening and saving files. However, after about a day of troubleshooting, I isolated the problem to the following code:

def download_all_product_images(image_dir):		
			
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.big_image.image) is '':
			print ".......Downloading big image @", shoe.big_image
			shoe.big_image.image = download_product_image(image_dir, shoe.big_image.image_url, str(shoe.id), 'b')
			shoe.big_image.save()
	
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.medium_image.image) is '':
			print ".......Downloading medium image @", shoe.medium_image
			shoe.medium_image.image = download_product_image(image_dir, shoe.medium_image.image_url, str(shoe.id), 'm')
			shoe.medium_image.save()


			#count = 1
			#for other_image in shoe.other_images.all().iterator():
				#if str(other_image.image) is '':
			#	print ".......Downloading image ", count, " @ ", other_image
			#	other_image.image = download_product_image(image_dir, other_image.image_url, str(shoe.id), str(count))
			#	other_image.save()
			#	count = count + 1	
			
		print "End Function"

When i run this, after about 1000 files are downloaded, I get an IOError, saying that i had opened too many files

I checked the /proc/<id>/fd directory and found that files were increasing when i save them to the database

---

However, the strangest thing i can't figure out is that if i swapped the two methods... memory only slightly increases:

def download_all_product_images(image_dir):		
		
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.medium_image.image) is '':
			print ".......Downloading medium image @", shoe.medium_image
			shoe.medium_image.image = download_product_image(image_dir, shoe.medium_image.image_url, str(shoe.id), 'm')
			shoe.medium_image.save()
	
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.big_image.image) is '':
			print ".......Downloading big image @", shoe.big_image
			shoe.big_image.image = download_product_image(image_dir, shoe.big_image.image_url, str(shoe.id), 'b')
			shoe.big_image.save()
	

Please let me know what is happening.....

[chris@…]

My Project that contains the error

[chris]

Actually, I'm testing it again and i think it's the way I called mechanize.

Will give update on further test results...

[chris]

Kinda embarrassed now... =P

Djanga ROCKS!!!!

Please close this ticket

Chris.

[Karen Tracey]

Note you could have closed it yourself, you don't need any special privs for that.