WIP: Support basic data parallel #366
Conversation
shendiaomo
force-pushed
the
basic_data_parallel
branch
from
cf33bbd
to
2d4a9f4
Compare
Codecov Report
@@ Coverage Diff @@
## develop #366 +/- ##
===========================================
- Coverage 87.77% 87.57% -0.20%
===========================================
Files 33 34 +1
Lines 1505 1513 +8
===========================================
+ Hits 1321 1325 +4
- Misses 121 125 +4
Partials 63 63
|
| // 1. Scatter the input to the given devices, | ||
| // 2. Replicate (deep clone) the model on each device, | ||
| // 3. Evaluate each module with its input on its device, | ||
| // 4. Gather the outputs of each replica into a single output tensor, located on the `outputDevice`. |
There was a problem hiding this comment.
There are two approaches for data parallelism in for multi-GPU training:
- Single-Process Multi-GPU
- Per Process Per GPU
PyTorch DistributedDataParallel has proved that Per Process Per GPU is more efficient.
Single-Process Multi-GPU is not the recommended mode for "
"DDP. In this mode, each DDP instance operates on multiple "
"devices and creates multiple module replicas within one "
"process. The overhead of scatter/gather and GIL contention "
"in every forward pass can slow down training. "
"Please consider using one DDP instance per device or per "
"module replica by explicitly setting device_ids or "
"CUDA_VISIBLE_DEVICES.
There was a problem hiding this comment.
So, scatter--> parallel apply --> gather is not suggested. Instead, we launch a training process for each device. Each training process does dataloading/forward/backward/allreduce/update individually.
shendiaomo
force-pushed
the
basic_data_parallel
branch
from
cb02186
to
784a3b7
Compare
shendiaomo
force-pushed
the
basic_data_parallel
branch
from
1b2b2fa
to
3007a5a
Compare
No description provided.