Skip to content

Addition of Choco sharing

Milos Vujasinovic requested to merge (removed):main into main

This merge request adds implementation of ChocoSGD1 in form of a sharing module decentralizepy.sharing.Choco. As a compression of choice the implementation uses topk sparsification over all elements of a model.

This sharing module adds two new parameters:

  • step_size - Step size parameter from the formulation of ChocoSGD in the original paper1
  • alpha - Percentage of model parameters to keep during topk sparsification of a model

1 Koloskova, Anastasia, Sebastian Stich, and Martin Jaggi. "Decentralized stochastic optimization and gossip algorithms with compressed communication." International Conference on Machine Learning. PMLR, 2019.

Edited by Milos Vujasinovic

Merge request reports