Notes

All source code samples were taken from the Horovod examples repository here (last checked: April 26, 2021). The samples have been slightly modified. Our changes are limited to,

• The data loading mechanism
• A bit of code cleanup
• A few additional comments pertaining to our custom data loading mechanism

Note: All newly added statements follow a comment beginning with [HPCNS]. All statements that demonstrate the use of Horovod follow a comment beginning with [Horovod] (as added by Horovod developers).

The following samples are included:

1. keras_mnist.py: A simple training program for an MNIST classifier that uses the Keras API with Horovod.
2. mnist.py: Also a training program for an MNIST classifier, this sample demonstrates using Horovod's DistributedGradientTape with a custom training loop.