The source project of this merge request has been removed.
Addition of Choco sharing
This merge request adds implementation of ChocoSGD1 in form of a sharing module decentralizepy.sharing.Choco
. As a compression of choice the implementation uses topk sparsification over all elements of a model.
This sharing module adds two new parameters:
- step_size - Step size parameter from the formulation of ChocoSGD in the original paper1
- alpha - Percentage of model parameters to keep during topk sparsification of a model
1 Koloskova, Anastasia, Sebastian Stich, and Martin Jaggi. "Decentralized stochastic optimization and gossip algorithms with compressed communication." International Conference on Machine Learning. PMLR, 2019.
Edited by Milos Vujasinovic