This is a tensorboard plugin by Trainy to supplement the existing PyTorch Profiler. This provides additional visualizations to effectively characterize traces for runs involving multiple GPUs. The plugin expects all traces to be collected using torch.profile and to be located in the same folder.
Install tensorboard and the plugin.
pip install tensorboard
pip install nodify-plugin
Generate PyTorch profiler traces as shown here and bring up the tensorboard where your traces are living. A set of example logs are provided in this repo under log/resnet18
tensorboard --logdir log/resnet18/
To view the plugin for development, create a virtual environment, install the requirements, and install the plugin.
python -m venv venv
. venv/bin/activate
pip install -e .
A lot of the features on the roadmap use Meta's Dynolog, kineto, and holistic trace analyzer.
- On-demand tracing and metrics through dynologger
- Recommendations for fixing multinode bottlenecks
- reading from logs stored on cloud object-stores (e.g. Amazon S3, Azure Blob)
For feature requests or bug reports/fixes, feel free to open a Github issue or make a pull request. We'd love to connect with any interested developers and we check our Discord to discuss the direction of our projects everyday. Connect with us either throught the text channels or through DM's.