Opened 5 years ago
Closed 5 years ago
#30383 closed Cleanup/optimization (worksforme)
Performance issue with ThreadPoolExecutor/Multiprocessing.
| Reported by: | shonejin | Owned by: | nobody |
|---|---|---|---|
| Component: | Core (Other) | Version: | dev |
| Severity: | Normal | Keywords: | performance, multiprocessing, pool |
| Cc: | Triage Stage: | Unreviewed | |
| Has patch: | no | Needs documentation: | no |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
When starting a background task that directly/indirectly use Python's ThreadPoolExecutor or Multiprocessing, I notice each pool worker can only utilize a small portion of CPU, leaving major of the CPU free. However, if I execute the same code outside of Django, the CPU utilization goes up by 5 times and thus performance improves a lot. I also tried "subprocess.call()" in Django to call 'python EXTERNAL_CODE.py' to run the computing but the CPU utilization issue is still there.
No documentations available explaining why this happens, any supplement info is helpful! Thanks!
Change History (1)
comment:1 by , 5 years ago
| Component: | Uncategorized → Core (Other) |
|---|---|
| Resolution: | → worksforme |
| Status: | new → closed |
| Summary: | Performance issue with ThreadPoolExecutor/Multiprocessing → Performance issue with ThreadPoolExecutor/Multiprocessing. |
| Type: | Uncategorized → Cleanup/optimization |
| Version: | 2.2 → master |
Note:
See TracTickets
for help on using tickets.
I was not able to reproduce this issue with
ThreadPoolExecutororsubprocess.