Opened 6 years ago

Closed 6 years ago

#30383 closed Cleanup/optimization (worksforme)

Performance issue with ThreadPoolExecutor/Multiprocessing.

Reported by: shonejin Owned by: nobody
Component: Core (Other) Version: dev
Severity: Normal Keywords: performance, multiprocessing, pool
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When starting a background task that directly/indirectly use Python's ThreadPoolExecutor or Multiprocessing, I notice each pool worker can only utilize a small portion of CPU, leaving major of the CPU free. However, if I execute the same code outside of Django, the CPU utilization goes up by 5 times and thus performance improves a lot. I also tried "subprocess.call()" in Django to call 'python EXTERNAL_CODE.py' to run the computing but the CPU utilization issue is still there.

No documentations available explaining why this happens, any supplement info is helpful! Thanks!

Change History (1)

comment:1 by Mariusz Felisiak, 6 years ago

Component: Uncategorized → Core (Other)
Resolution: → worksforme
Status: new → closed
Summary: Performance issue with ThreadPoolExecutor/Multiprocessing → Performance issue with ThreadPoolExecutor/Multiprocessing.
Type: Uncategorized → Cleanup/optimization
Version: 2.2 → master

I was not able to reproduce this issue with ThreadPoolExecutor or subprocess.

Note: See TracTickets for help on using tickets.
Back to Top