Distribute the work more evenly between faster and slower workers

[hyungukangphd]

Looking for a feature to distribute the work unequally to each worker. Often, I have one or two cores that are running slower than others, or get started up and running after others. Once some of the faster cores are done, they sit idle while I wait for the other cores to catch up. Often the faster cores will do its allotment in half the time.

Request: some way to distribute work such that the faster cores either get more work or help out the other thread. I know that dynamic allocation is difficult during the run, so perhaps processor binding, unequal allocation, or spawning more workers than processor cores, etc.

Spawning more workers does speed up things somewhat, but I don't know which core will pick up which work.

[HiCiChlid]

For me, I always deal with pandas dataframe, and before multi-core working, I am used to shuffle the dataframe first.