Opened 7 years ago

Closed 7 years ago

#7987 closed (invalid)

Leaks in Memory when using Iterator and saving the Model - Get Error Code 24 - 'Too Many Files Open'

Reported by: chris@… Owned by: nobody
Component: Database layer (models, ORM) Version: master
Severity: Keywords: Model, Iterator, Memory Leak
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

I'm currently making an experimental spider, getting information from www.paixie.net.

I am using a Django model as my database connector and have found something really strange going on in the code.

At first, i thought it was because i was opening and saving files. However, after about a day of troubleshooting, I isolated the problem to the following code:

def download_all_product_images(image_dir):		
			
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.big_image.image) is '':
			print ".......Downloading big image @", shoe.big_image
			shoe.big_image.image = download_product_image(image_dir, shoe.big_image.image_url, str(shoe.id), 'b')
			shoe.big_image.save()
	
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.medium_image.image) is '':
			print ".......Downloading medium image @", shoe.medium_image
			shoe.medium_image.image = download_product_image(image_dir, shoe.medium_image.image_url, str(shoe.id), 'm')
			shoe.medium_image.save()


			#count = 1
			#for other_image in shoe.other_images.all().iterator():
				#if str(other_image.image) is '':
			#	print ".......Downloading image ", count, " @ ", other_image
			#	other_image.image = download_product_image(image_dir, other_image.image_url, str(shoe.id), str(count))
			#	other_image.save()
			#	count = count + 1	
			
		print "End Function"

When i run this, after about 1000 files are downloaded, I get an IOError, saying that i had opened too many files

I checked the /proc/<id>/fd directory and found that files were increasing when i save them to the database


However, the strangest thing i can't figure out is that if i swapped the two methods... memory only slightly increases:

def download_all_product_images(image_dir):		
		
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.medium_image.image) is '':
			print ".......Downloading medium image @", shoe.medium_image
			shoe.medium_image.image = download_product_image(image_dir, shoe.medium_image.image_url, str(shoe.id), 'm')
			shoe.medium_image.save()
	
		for shoe in Shoe.objects.all().iterator():
			#if str(shoe.big_image.image) is '':
			print ".......Downloading big image @", shoe.big_image
			shoe.big_image.image = download_product_image(image_dir, shoe.big_image.image_url, str(shoe.id), 'b')
			shoe.big_image.save()
	

Please let me know what is happening.....

Attachments (1)

bbxiespider.zip (13.7 KB) - added by chris@… 7 years ago.
My Project that contains the error

Download all attachments as: .zip

Change History (4)

Changed 7 years ago by chris@…

My Project that contains the error

comment:1 Changed 7 years ago by chris

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Actually, I'm testing it again and i think it's the way I called mechanize.

Will give update on further test results...

comment:2 Changed 7 years ago by chris

Kinda embarrassed now... =P

Djanga ROCKS!!!!

Please close this ticket

Chris.

comment:3 Changed 7 years ago by kmtracey

  • Resolution set to invalid
  • Status changed from new to closed

Note you could have closed it yourself, you don't need any special privs for that.

Note: See TracTickets for help on using tickets.
Back to Top