The source project of this merge request has been removed.
Addition of Choco sharing
This merge request adds implementation of ChocoSGD1 in form of a sharing module decentralizepy.sharing.Choco
. As a compression of choice the implementation uses topk sparsification over all elements of a model.
This sharing module adds two new parameters:
- step_size - Step size parameter from the formulation of ChocoSGD in the original paper1
- alpha - Percentage of model parameters to keep during topk sparsification of a model
1 Koloskova, Anastasia, Sebastian Stich, and Martin Jaggi. "Decentralized stochastic optimization and gossip algorithms with compressed communication." International Conference on Machine Learning. PMLR, 2019.
Edited by Milos Vujasinovic
Merge request reports
Activity
Please register or sign in to reply