diff --git a/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png b/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png
index 1e25a25bb1869801caaa9e03301377a4a3799809..b336b456cfd1e46b644c9f5b228840c8d62ac341 100644
Binary files a/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png and b/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png differ
diff --git a/mlsys2022style/d-cliques.tex b/mlsys2022style/d-cliques.tex
index 8e91e3a687a71156ba2f55fd127c1b00f7882df1..008dde73225d3e512cb6411fd7b3829579d69063 100644
--- a/mlsys2022style/d-cliques.tex
+++ b/mlsys2022style/d-cliques.tex
@@ -345,7 +345,8 @@ of the local models across nodes.
inter-clique connections (see main text for details).}
\end{figure}
-\paragraph{Clique Averaging.} We address this problem by adding \emph{Clique
+\paragraph{\label{section:clique-averaging} Clique Averaging.}
+We address this problem by adding \emph{Clique
Averaging} to D-SGD
(Algorithm~\ref{Algorithm:Clique-Unbiased-D-PSGD}), which essentially
decouples gradient averaging from model averaging. The idea is to use only the
diff --git a/mlsys2022style/exp.tex b/mlsys2022style/exp.tex
index 7fcb30d64fc2ce662940e8a251e3032910acd887..3a10b2f314b44a521b811431279e13d83cada8f7 100644
--- a/mlsys2022style/exp.tex
+++ b/mlsys2022style/exp.tex
@@ -3,13 +3,13 @@
\section{Evaluation}
\label{section:evaluation}
-%In this section, we first compare D-Cliques to alternative topologies to
-%confirm the relevance of our main design choices. Then,
-%we evaluate some extensions of D-Cliques to further reduce the number of
-%inter-clique connections so as to gracefully scale with the number of
-%nodes.
-
- \todo{EL: Revise intro to section}
+In this section, we first compare D-Cliques to alternative topologies to
+show their benefits and the relevance of our main design choices. Then,
+we evaluate different inter-clique topologies to further to further reduce the number of
+inter-clique connections so as to gracefully scale with the number of
+nodes. We finally show D-Cliques to be resilient to some intra-clique
+connectivity failures and that Greedy Swap (Alg.~\ref{Algorithm:greedy-swap})
+constructs cliques with low skew efficiently.
\subsection{Experimental setup.}
\label{section:experimental-settings}
@@ -21,15 +21,16 @@ can remove much of the effect of label distribution skew.
We experiment with two datasets: MNIST~\cite{mnistWebsite} and
CIFAR10~\cite{krizhevsky2009learning}, which both have $L=10$ classes.
-For MNIST, we use 45k and 10k examples from the original 60k
-training set for training and validation respectively. The remaining 5k
-training examples were randomly removed to ensure all 10 classes are balanced
-while ensuring that the dataset is evenly divisible across 100 and 1000 nodes.
-We use all 10k examples of
-the test set to measure prediction accuracy. For CIFAR10, classes are evenly
-balanced: we use 45k/50k images of the original training set for training,
-5k/50k for validation, and all 10k examples of the test set for measuring
-prediction accuracy.
+For MNIST, we use 50k and 10k examples from the original 60k training
+set for training and validation respectively. We use all 10k examples of
+the test set to measure prediction accuracy. The validation set preserves the
+original unbalanced ratio of the classes in the test set.
+For CIFAR10, classes are evenly balanced: we initially used 45k/50k images
+of the original training set for training, 5k/50k for validation, and all 10k examples
+of the test set for measuring prediction accuracy. After tuning hyper-parameters
+on initial experiments, we then used all 50k images of the original training set
+for training for all experiments, as the 45k did not split evenly with 1000 nodes
+in the partitioning scheme explained in the next paragraph.
We use the non-IID partitioning scheme proposed by ~\cite{mcmahan2016communication}
in their seminal Federated Learning paper for MNIST on both MNIST and CIFAR10:
@@ -45,7 +46,7 @@ We
use a logistic regression classifier for MNIST, which
provides up to 92.5\% accuracy in the centralized setting.
For CIFAR10, we use a Group-Normalized variant of LeNet~\cite{quagmire}, a
-deep convolutional network which achieves an accuracy of $72.3\%$ in the
+deep convolutional network which achieves an accuracy of $74.15\%$ in the
centralized setting.
These models are thus reasonably accurate (which is sufficient to
study the effect of the topology) while being sufficiently fast to train in a
@@ -56,11 +57,13 @@ validation set for 100 nodes, obtaining respectively $0.1$ and $128$ for
MNIST and $0.002$ and $20$ for CIFAR10.
For CIFAR10, we additionally use a momentum of $0.9$.
-We evaluate 100- and 1000-node networks by creating multiple models in memory and simulating the exchange of messages between nodes.
+We evaluate 100- and 1000-node networks by creating multiple models
+in memory and simulating the exchange of messages between nodes.
To ignore the impact of distributed execution strategies and system
optimization techniques, we report the test accuracy of all nodes (min, max,
average) as a function of the number of times each example of the dataset has
-been sampled by a node, i.e. an \textit{epoch}. This is equivalent to the classic case of a single node sampling the full distribution.
+been sampled by a node, i.e. an \textit{epoch}. This is equivalent to the classic
+case of a single node sampling the full distribution.
To further make results comparable across different number of nodes, we lower
the batch size proportionally to the number of nodes added, and inversely,
e.g. on MNIST, 128 with 100 nodes vs. 13 with 1000 nodes. This
@@ -72,18 +75,36 @@ resulting communication overhead is impractical.}
Finally, we compare our results against an ideal baseline: either a
fully-connected network topology with the same number of nodes or a single IID
node. In both cases, the topology has no effect on
-the optimization. For a certain choice of number of nodes and
-mini-batch size, both approaches are equivalent.
+the optimization. Both approaches effectively optimize a single model and sample
+uniformly from the global distribution, yet we have observed a fully-connected network
+ to convergence slightly faster and obtain slightly
+better final accuracy than a single node sampling from the global distribution in the presence
+of data heterogeneity\footnote{We
+conjecture that an heterogeneous data partition in a fully-connected network may force
+more balanced representation of all classes in the union of all mini-batches, leading to better convergence.}.
+We therefore compare against a fully-connected network unless the simulation time
+prevented us to obtain results in time for submission, in which case we use a single node IID.
\subsection{D-Cliques match the Convergence Speed of Fully-Connected with a Fraction of the Edges}
+\label{section:d-cliques-vs-fully-connected}
+
+In this first experiment, we show that D-Cliques with Clique Averaging (and Momentum mentioned) converges
+almost as fast as a fully-connected network on both MNIST and CIFAR10. Figure~\ref{fig:convergence-speed-dc-vs-fc-2-shards-per-node}
+illustrates the convergence speed of D-Cliques with $n=100$ nodes on MNIST (with Clique Averaging)
+and CIFAR10 (with Clique Averaging and Momentum). Observe that the convergence speed is
+very close to that of a fully-connected topology, and significantly better than with
+a ring or a grid (see Figure~\ref{fig:iid-vs-non-iid-problem}).
+It also has less variance than both the ring and grid. With
+100 nodes, it offers a reduction of $\approx90\%$ in the number of edges
+compared to a fully-connected topology.
% From directory 'results-v2':
% MNIST
% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name fully-connected d-cliques/greedy-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-09-28-23:16:47-CEST-labostrex117 all/2021-09-28-23:18:49-CEST-labostrex119 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 80 --ymax 92.5 --yaxis test-accuracy --labels 'fully-connected' 'd-cliques (fc) w/ cliq-avg' --save-figure ../mlsys2022style/figures/convergence-speed-mnist-dc-fc-vs-fc-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-09-28-23:16:47-CEST-labostrex117 all/2021-09-28-23:18:49-CEST-labostrex119 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 80 --ymax 92.5 --yaxis test-accuracy --labels 'fully-connected' 'd-cliques (fc) w/ cliq-avg' --save-figure ../mlsys2022style/figures/convergence-speed-mnist-dc-fc-vs-fc-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3
% CIFAR10
% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name fully-connected d-cliques/greedy-swap --nodes:name 2-shards-eq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-02-18:58:22-CEST-labostrex114 all/2021-10-03-19:53:21-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'fully-connected w/ mom.' 'd-cliques (fc) w/ c-avg, mom.' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node.png --linestyles 'solid' 'dashed' --legend 'lower right' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-02-18:58:22-CEST-labostrex114 all/2021-10-03-19:53:21-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'fully-connected w/ mom.' 'd-cliques (fc) w/ c-avg, mom.' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node.png --linestyles 'solid' 'dashed' --legend 'lower right' --font-size 18 --linewidth 3
\begin{figure}[htbp]
\centering
@@ -98,45 +119,42 @@ mini-batch size, both approaches are equivalent.
\includegraphics[width=\textwidth]{figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node}
\caption{\label{fig:convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node} CIFAR10}
\end{subfigure}
-\caption{\label{fig:convergence-speed-dc-vs-fc-2-shards-per-node} Convergence Speed of D-Cliques constructed with Greedy Swap Compared to Fully-Connected on 100 Nodes (2 shards/node).}
+\caption{\label{fig:convergence-speed-dc-vs-fc-2-shards-per-node} Convergence Speed of D-Cliques constructed with Greedy Swap Compared to Fully-Connected on 100 Nodes (2 shards/node). Bold line is the average accuracy over all nodes. Thinner upper and lower lines and maximum and minimum accuracy over all nodes.}
\end{figure}
+\subsection{Clique Averaging and Momentum are Beneficial and Sometimes Necessary}
-Figure~\ref{fig:convergence-speed-dc-vs-fc-2-shards-per-node} illustrates the
-convergence speed of D-Cliques with $n=100$ nodes on MNIST (with Clique Averaging) and CIFAR10 (with Clique Averaging and Momentum). Observe that the
-convergence speed is
-very close
-to that of a fully-connected topology, and significantly better than with
-a ring or a grid (see Figure~\ref{fig:iid-vs-non-iid-problem}). With
-100 nodes, it offers a reduction of $\approx90\%$ in the number of edges
-compared to a fully-connected topology.
-
-\subsection{Clique Averaging and Momentum are Necessary}
+Figure~\ref{fig:d-clique-mnist-clique-avg} shows that Clique Averaging (Alg.~\autoref{Algorithm:Clique-Unbiased-D-PSGD})
+ reduces the variance of models across nodes and accelerates
+convergence on MNIST. Note that Clique Averaging induces a small
+additional cost, as gradients
+and models need to be sent in two separate rounds of messages.
+Nonetheless, compared to fully connecting all nodes, the total number
+of messages is reduced by $\approx 80\%$.
% From directory 'results-v2':
% MNIST
% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name d-cliques/greedy-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-03:53:42-CEST-labostrex119 all/2021-09-28-23:18:49-CEST-labostrex119 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 89 --ymax 92.5 --yaxis test-accuracy --labels 'd-cliques w/o c-avg.' 'd-cliques w/ c-avg.' --save-figure ../mlsys2022style/figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-03:53:42-CEST-labostrex119 all/2021-09-28-23:18:49-CEST-labostrex119 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 89 --ymax 92.5 --yaxis test-accuracy --labels 'd-cliques (fc) w/o c-avg.' 'd-cliques (fc) w/ c-avg.' --save-figure ../mlsys2022style/figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 2.5
\begin{figure}[htbp]
\centering
- \includegraphics[width=0.35\textwidth]{figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node}
+ \includegraphics[width=0.23\textwidth]{figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node}
\caption{\label{fig:d-clique-mnist-clique-avg} Effect of Clique Averaging on MNIST. Y axis starts at 89.}
\end{figure}
-As illustrated in Figure~\ref{fig:d-clique-mnist-clique-avg}, Clique Averaging
-significantly reduces the variance of models across nodes and accelerates
-convergence to reach the same level as the one obtained with a
-fully-connected topology. Note that Clique Averaging induces a small
-additional cost, as gradients
-and models need to be sent in two separate rounds of messages. Nonetheless, compared to fully connecting all nodes, the total number of messages is reduced by $\approx 80\%$.
-
+Figure~\ref{fig:cifar10-c-avg-momentum} shows the interaction between
+Clique Averaging and Momentum on CIFAR10. Without Clique Averaging,
+the use of momentum is actually detrimental. With Clique Averaging, the
+situation reverses and momentum is again beneficial. The combination
+of both has the fastest convergence speed and the lowest variance among all
+four possibilities.
% CIFAR10
% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name d-cliques/greedy-swap --nodes:name 2-shards-eq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
% w/o Clique Averaging
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-23:37:42-CEST-labostrex117 all/2021-10-04-03:13:46-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques w/o momentum' 'd-cliques w/ momentum' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-wo-c-avg-no-mom-vs-mom-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-23:37:42-CEST-labostrex117 all/2021-10-04-03:13:46-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques (fc) w/o momentum' 'd-cliques (fc) w/ momentum' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-wo-c-avg-no-mom-vs-mom-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3
% w/ Clique Averaging
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-16:10:34-CEST-labostrex117 all/2021-10-03-19:53:21-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques w/o momentum' 'd-cliques w/ momentum' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-w-c-avg-no-mom-vs-mom-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-16:10:34-CEST-labostrex117 all/2021-10-03-19:53:21-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques (fc) w/o momentum' 'd-cliques (fc) w/ momentum' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-w-c-avg-no-mom-vs-mom-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3 --legend 'lower right'
\begin{figure}[htbp]
\centering
@@ -154,23 +172,20 @@ and models need to be sent in two separate rounds of messages. Nonetheless, comp
\caption{\label{fig:cifar10-c-avg-momentum} Effect of Clique Averaging and Momentum on CIFAR10 with LeNet.}
\end{figure}
-As shown in
-Figure~\ref{fig:cifar10-c-avg-momentum},
-the use of Clique Averaging restores the benefits of momentum and closes the gap
-with the centralized setting.
-
-
-
\subsection{D-Cliques Converge Faster than Random Graphs}
+\autoref{fig:d-cliques-comparison-to-non-clustered-topologies} shows that D-Cliques, even
+without Clique Averaging or Momentum converge faster and with lower variance than a
+random graph with a similar number of edges (10) per node, and therefore that a careful
+design of the topology is indeed necessary.
% From directory 'results-v2':
% MNIST
% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name random-graph d-cliques/greedy-swap greedy-neighbourhood-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-03:53:42-CEST-labostrex119 all/2021-09-29-22:17:08-CEST-labostrex118 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 80 --ymax 92.5 --yaxis test-accuracy --labels 'd-cliques (fc) w/o cliq-avg' 'random 10' --save-figure ../mlsys2022style/figures/convergence-mnist-random-vs-d-cliques-2-shards.png --linestyles 'solid' 'dashed' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-03:53:42-CEST-labostrex119 all/2021-09-29-22:17:08-CEST-labostrex118 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 80 --ymax 92.5 --yaxis test-accuracy --labels 'd-cliques (fc) w/o cliq-avg' 'random 10' --save-figure ../mlsys2022style/figures/convergence-mnist-random-vs-d-cliques-2-shards.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3
% CIFAR10
% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name d-cliques/greedy-swap random-graph --nodes:nb-nodes 100 --algorithm:learning-momentum 0.9 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-23:37:42-CEST-labostrex117 all/2021-10-05-18:38:30-CEST-labostrex115 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques (fc) w/o cliq-avg w/o mom.' 'random 10 w/o mom.' --save-figure ../mlsys2022style/figures/convergence-cifar10-random-vs-d-cliques-2-shards.png --linestyles 'solid' 'dashed' --font-size 18
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-23:37:42-CEST-labostrex117 all/2021-10-05-18:38:30-CEST-labostrex115 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques (fc) w/o cliq-avg w/o mom.' 'random 10 w/o mom.' --save-figure ../mlsys2022style/figures/convergence-cifar10-random-vs-d-cliques-2-shards.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3
\begin{figure}[htbp]
\centering
@@ -188,23 +203,47 @@ with the centralized setting.
\caption{\label{fig:convergence-random-vs-d-cliques-2-shards} Comparison to Random Graph with 10 edges per node \textit{without} Clique Averaging or Momentum (see main text for justification).}
\end{figure}
+In comparison to a random graph however, D-Cliques provide additional benefits: they ensure
+a diverse representation of all classes in the immediate neighbourhood of all nodes; they enable
+ Clique Averaging to debias gradients; and they provide a high-level of clustering, i.e. neighbors
+ of a node are neighbors themselves, which tends to lower variance.
+In order, to distinguish the effect of the first two from the last, we compare D-Cliques to other variations
+of random graphs: (1) with the additional constraint that all classes should be represented in the immediate neighborhood of all nodes
+(i.e. 'all classes repr.'), and (2) in combination with unbiased gradients computed using
+the average of the gradients of all neighbors for all nodes. Satisfying the first constraint while obtaining
+the same skew as D-Cliques built with Greedy Swap was challenging with the current partitioning scheme, so
+we performed the experiments in a more heterogeneous setting in which each node has examples of only 1 class.
+In this setting, it is easy to construct cliques and a random graph such that the neighborhood of each node in both
+cases has a skew of 0.
+
+
+
\begin{figure}[htbp]
\centering
\begin{subfigure}[b]{0.23\textwidth}
\centering
- \includegraphics[width=\textwidth]{../figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies}
+ % From directory results/mnist:
+% python ../../../../Software/non-iid-topology-simulator/tools/v1/plot_convergence.py fully-connected-cliques/all/2021-03-10-10:19:44-CET no-init-no-clique-avg/fully-connected-cliques/all/2021-03-12-11:12:49-CET random-10/all/2021-07-23-11:59:56-CEST random-10-diverse/all/2021-03-17-20:28:35-CET --labels 'd-clique (fcc)' 'd-clique (fcc) no clique avg.' '10 random edges' '10 random edges (all classes represented)' --add-min-max --legend 'lower right' --ymin 80 --ymax 92.5 --yaxis test-accuracy --save-figure ../../mlsys2022style/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png --font-size 13 --linestyles 'solid' 'dashed' 'dotted' 'dashdot' --linewidth 3
+ \includegraphics[width=\textwidth]{figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies}
\caption{MNIST}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.23\textwidth}
\centering
- \includegraphics[width=\textwidth]{../figures/d-cliques-cifar10-linear-comparison-to-non-clustered-topologies}
+ % To regenerate the figure, from directory results/cifar10
+% python ../../../../Software/non-iid-topology-simulator/tools/v1/plot_convergence.py no-init/fully-connected-cliques/all/2021-03-13-18:32:55-CET no-init-no-clique-avg/fully-connected-cliques/all/2021-03-13-18:34:35-CET random-10/all/2021-07-23-14:33:48-CEST random-10-diverse/all/2021-03-17-20:30:41-CET random-10-diverse-unbiased-gradient/all/2021-03-17-20:31:14-CET --labels 'd-clique (fcc) clique avg.' 'd-clique (fcc) no clique avg.' '10 random edges' '10 random edges (all classes repr.)' '10 random (all classes repr.) with unbiased grad.' --add-min-max --legend 'upper left' --yaxis test-accuracy --save-figure ../../mlsys2022style/figures/d-cliques-cifar10-linear-comparison-to-non-clustered-topologies.png --ymax 119 --font-size 13 --linestyles 'solid' 'dashed' 'dotted' 'dashdot' 'solid' --markers '' '' '' '' 'o' --linewidth 3
+ \includegraphics[width=\textwidth]{figures/d-cliques-cifar10-linear-comparison-to-non-clustered-topologies}
\caption{CIFAR10}
\end{subfigure}
\caption{\label{fig:d-cliques-comparison-to-non-clustered-topologies} Comparison to Random Graph with 10 edges per node, \textit{with} Clique Averaging (and Momentum) as well as analogous Neighbourhood Averaging for Random Graph \textit{in a more stringent partitioning of 1 class/node}\textsuperscript{*}.}
-\footnotesize\textsuperscript{*}\textit{These results were obtained with a previous version of the simulator but should be consistent with the latest. They will be updated for the final version.}
+\footnotesize\textsuperscript{*}\textit{These results were obtained with a previous version of the simulator but should be consistent with the latest. They will be updated for the final version of the paper.}
\end{figure}
+\autoref{fig:d-cliques-comparison-to-non-clustered-topologies} shows the results for MNIST and CIFAR10. In the case of MNIST,
+D-Cliques converge faster than all other options. In the case of CIFAR10, the clustering appears to be critical
+for good convergence speed: even a random graph with diverse neighborhoods and unbiased gradients
+converges significantly slower.
+
%We demonstrate the advantages of D-Cliques over alternative sparse topologies
%that have a similar number of edges. First, we consider topologies in which
%the neighbors of each node are selected at random (hence without any clique
@@ -275,92 +314,15 @@ with the centralized setting.
%data with sparse topologies requires a very careful design, as we have
%proposed with D-Cliques.
-\subsection{Cliques built with Greedy Swap Converge Faster than Random Cliques}
-
-% From directory 'results-v2':
-% MNIST
-% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name d-cliques/random-cliques d-cliques/greedy-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-22:12:59-CEST-labostrex114 all/2021-09-28-23:18:49-CEST-labostrex119 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 80 --ymax 92.5 --yaxis test-accuracy --labels 'd-cliques random' 'd-cliques greedy-swap' --save-figure ../mlsys2022style/figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18
-% CIFAR10
-% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name d-cliques/random-cliques d-cliques/greedy-swap --nodes:name 2-shards-eq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-04-21:18:33-CEST-labostrex117 all/2021-10-03-19:53:21-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques random' 'd-cliques greedy-swap' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18
-
-\begin{figure}[htbp]
- \centering
- \begin{subfigure}[b]{0.23\textwidth}
- \centering
- \includegraphics[width=\textwidth]{figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node}
- \caption{\label{fig:convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node} MNIST}
- \end{subfigure}
- \hfill
- \begin{subfigure}[b]{0.23\textwidth}
- \centering
- \includegraphics[width=\textwidth]{figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node}
- \caption{\label{fig:convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node} CIFAR10}
- \end{subfigure}
-\caption{\label{fig:convergence-speed-dc-random-vs-dc-gs-2-shards-per-node} Convergence Speed of D-Cliques constructed Randomly vs Greedy Swap on 100 Nodes (2 shards/node).}
-\end{figure}
-
-\subsection{D-Cliques May Tolerate Some Intra-Connectivity Failures}
-% From directory 'results-v2':
-% MNIST
-% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name d-cliques/greedy-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% w/o Clique Gradient
-% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-03:53:42-CEST-labostrex119 all/2021-10-01-21:44:14-CEST-labostrex113 all/2021-10-02-06:53:40-CEST-labostrex113 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 89 --ymax 92.5 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18
-% w/ Clique Gradient
-% python $TOOLS/analyze/diff.py --rundirs all/2021-09-28-23:18:49-CEST-labostrex119 all/2021-10-01-17:08:42-CEST-labostrex113 all/2021-10-02-02:17:43-CEST-labostrex113 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 89 --ymax 92.5 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18
-\begin{figure}[htbp]
- \centering
-
-\begin{subfigure}[htbp]{0.23\textwidth}
- \centering
- \includegraphics[width=\textwidth]{figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal}
-\caption{\label{fig:d-cliques-mnist-wo-clique-avg-impact-of-edge-removal} Without Clique Averaging }
-\end{subfigure}
-\hfill
-\begin{subfigure}[htbp]{0.23\textwidth}
- \centering
- \includegraphics[width=\textwidth]{figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal}
-\caption{\label{fig:d-cliques-mnist-w-clique-avg-impact-of-edge-removal} With Clique Averaging}
-\end{subfigure}
-\caption{\label{fig:d-cliques-mnist-intra-connectivity} MNIST: Impact of Intra-clique Connectivity Failures. Y axis starts at 89.}
-\end{figure}
-
-% CIFAR10
-% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name d-cliques/greedy-swap --nodes:name 2-shards-eq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
-% w/o Clique Gradient
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-04-03:13:46-CEST-labostrex117 all/2021-10-06-17:58:49-CEST-labostrex112 all/2021-10-06-17:45:22-CEST-labostrex115 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 80 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18
-% w/ Clique Gradient
-% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-19:53:21-CEST-labostrex117 all/2021-10-06-12:46:49-CEST-labostrex112 all/2021-10-06-12:49:51-CEST-labostrex115 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 80 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18
-
-\begin{figure}[htbp]
- \centering
-\begin{subfigure}[htbp]{0.23\textwidth}
- \centering
- \includegraphics[width=\textwidth]{figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal}
-\caption{\label{fig:d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal} Without Clique Averaging }
-\end{subfigure}
-\hfill
-\begin{subfigure}[htbp]{0.23\textwidth}
- \centering
- \includegraphics[width=\textwidth]{figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal}
-\caption{\label{fig:d-cliques-cifar10-w-clique-avg-impact-of-edge-removal} With Clique Averaging}
-\end{subfigure}
-\caption{\label{fig:d-cliques-cifar10-intra-connectivity} CIFAR: Impact of Intra-clique Connectivity Failures (with Momentum).}
-\end{figure}
\subsection{D-Cliques Scale with Sparser Inter-Clique Topologies}
-In this last series of experiments, we evaluate the effect of choosing sparser
-inter-clique topologies on the convergence speed for a larger network of 1000
-nodes. We compare the scalability and convergence speed of the several
-D-Cliques variants introduced in Section~\ref{section:interclique-topologies}.
-
-Figure~\ref{fig:d-cliques-cifar10-convolutional} shows the convergence
-speed of all sparse inter-clique topologies on MNIST and CIFAR10, compared to the ideal
-baseline
-of a
+\autoref{fig:d-cliques-scaling-mnist-1000} and \autoref{fig:d-cliques-scaling-cifar10-1000}
+show the convergence
+speed of all sparse inter-clique topologies of Section~\ref{section:interclique-topologies}
+ respectively on MNIST and CIFAR10 on a larger network of 1000 nodes, compared to the ideal
+baseline of a
single IID node performing the same number of updates per epoch (representing
the fastest convergence speed achievable if topology had no impact). Among the linear schemes, the ring
topology converges but is much slower than our fractal scheme. Among the super-linear schemes, the small-world
@@ -373,10 +335,9 @@ fully-connected topology still offers
significant benefits with 1000 nodes, as it represents a 98\% reduction in the
number of edges compared to fully connecting individual nodes (18.9 edges on
average instead of 999) and a 96\% reduction in the number of messages (37.8
-messages per round per node on average instead of 999). We refer to
-Appendix~\ref{app:scaling} for additional results comparing the convergence
-speed across different number of nodes. Overall, these results
-show that D-Cliques can nicely scale with the number of nodes.
+messages per round per node on average instead of 999).
+%We refer to Appendix~\ref{app:scaling} for additional results comparing the convergence speed across different number of nodes.
+Overall, these results show that D-Cliques can nicely scale with the number of nodes.
% From directory 'results-v2':
% MNIST
@@ -445,9 +406,83 @@ show that D-Cliques can nicely scale with the number of nodes.
%\caption{\label{fig:d-cliques-cifar10-convolutional} D-Cliques Convergence Speed with 1000 nodes, non-IID, Constant Updates per Epoch, with Different Inter-Clique Topologies.}
%\end{figure}
-\subsection{Cliques with Low Skew can be Constructed Efficiently with Greedy Swap}
+\subsection{D-Cliques Can Tolerate Some Intra-Connectivity Failures}
+
+We measured the impact of randomly removing 1 and 5 intra-clique edges per
+ clique to assess how critical full connectivity is within cliques.
+
+\autoref{fig:d-cliques-mnist-intra-connectivity} shows that for MNIST, when not using Clique Averaging,
+removing edges decreases slightly the convergence speed and increases
+the variance between nodes. However, when using Clique Averaging, even removing 5 edges affects very little
+the convergence speed.
+
+% From directory 'results-v2':
+% MNIST
+% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name d-cliques/greedy-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
+% w/o Clique Gradient
+% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-03:53:42-CEST-labostrex119 all/2021-10-01-21:44:14-CEST-labostrex113 all/2021-10-02-06:53:40-CEST-labostrex113 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 89 --ymax 92.5 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18 --linewidth 3
+% w/ Clique Gradient
+% python $TOOLS/analyze/diff.py --rundirs all/2021-09-28-23:18:49-CEST-labostrex119 all/2021-10-01-17:08:42-CEST-labostrex113 all/2021-10-02-02:17:43-CEST-labostrex113 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 89 --ymax 92.5 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18 --linewidth 3
+\begin{figure}[htbp]
+ \centering
+
+\begin{subfigure}[htbp]{0.23\textwidth}
+ \centering
+ \includegraphics[width=\textwidth]{figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal}
+\caption{\label{fig:d-cliques-mnist-wo-clique-avg-impact-of-edge-removal} Without Clique Averaging }
+\end{subfigure}
+\hfill
+\begin{subfigure}[htbp]{0.23\textwidth}
+ \centering
+ \includegraphics[width=\textwidth]{figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal}
+\caption{\label{fig:d-cliques-mnist-w-clique-avg-impact-of-edge-removal} With Clique Averaging}
+\end{subfigure}
+\caption{\label{fig:d-cliques-mnist-intra-connectivity} MNIST: Impact of Intra-clique Connectivity Failures. Y axis starts at 89.}
+\end{figure}
+
+\autoref{fig:d-cliques-cifar10-intra-connectivity} shows that for CIFAR10, the impact is stronger. We show the results with and without Clique Averaging
+with momentum in both cases, as momentum is critical for obtaining the best convergence speed. Without Clique Averaging,
+removing edges has a small effect on convergence speed and variance, but the convergence speed is too slow to be practical.
+With Clique Averaging, the removing edges has initially a small effect, when only removing one edge per clique. However, It becomes
+significant when removing 5 edges per clique. D-Cliques can therefore tolerate some connectivity failures between clique members
+but the number of those failures depends on the dataset and model trained.
+% CIFAR10
+% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name d-cliques/greedy-swap --nodes:name 2-shards-eq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
+% w/o Clique Gradient
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-04-03:13:46-CEST-labostrex117 all/2021-10-06-17:58:49-CEST-labostrex112 all/2021-10-06-17:45:22-CEST-labostrex115 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 80 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18 --linewidth 3
+% w/ Clique Gradient
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-03-19:53:21-CEST-labostrex117 all/2021-10-06-12:46:49-CEST-labostrex112 all/2021-10-06-12:49:51-CEST-labostrex115 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 80 --yaxis test-accuracy --labels 'full intra-connectivity' '-1 edge/clique' '-5 edges/clique' --save-figure ../mlsys2022style/figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal.png --linestyles 'solid' 'dashed' 'dotted' --font-size 18 --linewidth 3
+
+\begin{figure}[htbp]
+ \centering
+\begin{subfigure}[htbp]{0.23\textwidth}
+ \centering
+ \includegraphics[width=\textwidth]{figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal}
+\caption{\label{fig:d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal} Without Clique Averaging }
+\end{subfigure}
+\hfill
+\begin{subfigure}[htbp]{0.23\textwidth}
+ \centering
+ \includegraphics[width=\textwidth]{figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal}
+\caption{\label{fig:d-cliques-cifar10-w-clique-avg-impact-of-edge-removal} With Clique Averaging}
+\end{subfigure}
+\caption{\label{fig:d-cliques-cifar10-intra-connectivity} CIFAR: Impact of Intra-clique Connectivity Failures (with Momentum).}
+\end{figure}
+
+\subsection{Greedy Swap Improves Random Cliques at an Affordable Cost}
+\label{section:greedy-swap-vs-random-cliques}
+
+In the next two sub-sections, we compare cliques built with Greedy Swap (Alg.~\ref{Algorithm:greedy-swap})
+to Random Cliques, on their quality (skew), the cost
+of their construction, and their convergence speed.
+
+\subsubsection{Cliques with Low Skew can be Constructed Efficiently with Greedy Swap}
\label{section:cost-cliques}
+We compared the final average skew of 10 cliques created either randomly or with Greedy Swap,
+over 100 experiments after 1000 steps. \autoref{fig:skew-convergence-speed-2-shards}, in the form of an histogram,
+ shows that Greedy Swap generate cliques of significantly lower skew, close to 0 in a majority of cases for both MNIST and CIFAR10.
+
% MNIST
% python $TOOLS/plot/skew/final-distribution.py --rundirs skews-mnist/* --save-figure ../mlsys2022style/figures/final-skew-distribution-mnist.png --labels 'Greedy Swap' 'Random Cliques' --linewidth 2.5 --font-size 18 --linestyles 'solid' 'dashed'
% CIFAR10
@@ -469,9 +504,48 @@ show that D-Cliques can nicely scale with the number of nodes.
\caption{\label{fig:final-skew-distribution} Final Quality of Cliques (Skew) with a Maximum Size of 10 over 100 Experiments.}
\end{figure}
+\autoref{fig:skew-convergence-speed-2-shards} shows such a low skew can be achieved
+in less than 400 steps for both MNIST and CIFAR, which in practice takes less than 6 seconds in Python 3.7 on a
+Macbook Pro 2020 for a network of 100 nodes and cliques of size 10. Greedy Swap
+is therefore fast and efficient. Moreover, it illustrates that an unbalanced number of examples
+between classes makes the construction of cliques with low skew harder and slower.
+
%python $TOOLS/analyze/filter.py skews --topology:name d-cliques/greedy-swap | python $TOOLS/plot/skew/convergence.py --max-steps 400 --labels 'MNIST (unbalanced classes)' 'CIFAR10 (balanced classes)' --linewidth 2.5 --save-figure ../mlsys2022style/figures/skew-convergence-speed-2-shards.png
\begin{figure}[htbp]
\centering
\includegraphics[width=0.3\textwidth]{figures/skew-convergence-speed-2-shards}
\caption{\label{fig:skew-convergence-speed-2-shards} Speed of Skew Decrease during Clique Construction. Bold line is the average over 100 experiments and 10 cliques/experiments. Thin lines are respectively the minimum and maximum over all experiments. In wall-clock time, 1000 steps take less than 6 seconds in Python 3.7 on a MacBook Pro 2020.}
-\end{figure}
\ No newline at end of file
+\end{figure}
+
+\subsubsection{Cliques built with Greedy Swap Converge Faster than Random Cliques}
+
+\autoref{fig:convergence-speed-dc-random-vs-dc-gs-2-shards-per-node} compares
+the convergence speed of cliques optimized with Greedy Swap for 1000 steps with cliques built randomly
+(equivalent to Greedy Swap with 0 steps). For both MNIST and CIFAR10, convergence speed
+increases significantly and variance between nodes decreases dramatically. Decreasing the skew of cliques
+is therefore critical to convergence speed.
+
+% From directory 'results-v2':
+% MNIST
+% python $TOOLS/analyze/filter.py all --dataset:name mnist --topology:name d-cliques/random-cliques d-cliques/greedy-swap --nodes:name 2-shards-uneq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
+% python $TOOLS/analyze/diff.py --rundirs all/2021-09-29-22:12:59-CEST-labostrex114 all/2021-09-28-23:18:49-CEST-labostrex119 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 80 --ymax 92.5 --yaxis test-accuracy --labels 'd-cliques random' 'd-cliques greedy-swap' --save-figure ../mlsys2022style/figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3
+% CIFAR10
+% python $TOOLS/analyze/filter.py all --dataset:name cifar10 --topology:name d-cliques/random-cliques d-cliques/greedy-swap --nodes:name 2-shards-eq-classes --meta:seed 1 --nodes:nb-nodes 100 | python $TOOLS/analyze/diff.py
+% python $TOOLS/analyze/diff.py --rundirs all/2021-10-04-21:18:33-CEST-labostrex117 all/2021-10-03-19:53:21-CEST-labostrex117 --pass-through | python $TOOLS/plot/convergence.py --add-min-max --ymin 0 --ymax 100 --yaxis test-accuracy --labels 'd-cliques random' 'd-cliques greedy-swap' --save-figure ../mlsys2022style/figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node.png --linestyles 'solid' 'dashed' --font-size 18 --linewidth 3
+
+\begin{figure}[htbp]
+ \centering
+ \begin{subfigure}[b]{0.23\textwidth}
+ \centering
+ \includegraphics[width=\textwidth]{figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node}
+ \caption{\label{fig:convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node} MNIST}
+ \end{subfigure}
+ \hfill
+ \begin{subfigure}[b]{0.23\textwidth}
+ \centering
+ \includegraphics[width=\textwidth]{figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node}
+ \caption{\label{fig:convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node} CIFAR10}
+ \end{subfigure}
+\caption{\label{fig:convergence-speed-dc-random-vs-dc-gs-2-shards-per-node} Convergence Speed of D-Cliques constructed Randomly vs Greedy Swap on 100 Nodes (2 shards/node).}
+\end{figure}
+
diff --git a/mlsys2022style/figures/convergence-cifar10-random-vs-d-cliques-2-shards.png b/mlsys2022style/figures/convergence-cifar10-random-vs-d-cliques-2-shards.png
index c0c4192ba79bb0252f6168de68774a657e0d3647..2ccdf6e6d0cc661e89f413b3fcb27b37b1aa72b6 100644
Binary files a/mlsys2022style/figures/convergence-cifar10-random-vs-d-cliques-2-shards.png and b/mlsys2022style/figures/convergence-cifar10-random-vs-d-cliques-2-shards.png differ
diff --git a/mlsys2022style/figures/convergence-mnist-random-vs-d-cliques-2-shards.png b/mlsys2022style/figures/convergence-mnist-random-vs-d-cliques-2-shards.png
index 9cce7f1da4b953d64863e0074c9c71a589cfc445..afd990e527036e902809043a01dbcdd14804b20a 100644
Binary files a/mlsys2022style/figures/convergence-mnist-random-vs-d-cliques-2-shards.png and b/mlsys2022style/figures/convergence-mnist-random-vs-d-cliques-2-shards.png differ
diff --git a/mlsys2022style/figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node.png
index d952518d4db139adb158e7319b765eadffee3604..171cd76fc83a120528a9ad9b3e9bc8274bf989fb 100644
Binary files a/mlsys2022style/figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-cifar10-dc-fc-vs-fc-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node.png
index 1bdf7fc40c52e81a26af648d0dcc0bf220eb92eb..4e6998e023d760def7e9633c8a732a0adcca8798 100644
Binary files a/mlsys2022style/figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-cifar10-dc-random-vs-dc-gs-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/convergence-speed-cifar10-w-c-avg-no-mom-vs-mom-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-cifar10-w-c-avg-no-mom-vs-mom-2-shards-per-node.png
index af4490a544e0f8bd11f587487d36e99bd570a2f6..bf56cb5ed5f3b9cfc9bd07e1eaa68366f6bb334e 100644
Binary files a/mlsys2022style/figures/convergence-speed-cifar10-w-c-avg-no-mom-vs-mom-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-cifar10-w-c-avg-no-mom-vs-mom-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/convergence-speed-cifar10-wo-c-avg-no-mom-vs-mom-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-cifar10-wo-c-avg-no-mom-vs-mom-2-shards-per-node.png
index 512f1d8a88c1d94ae7cfde6c23482ef724b7d4fc..6c09ccba79c7911d6bddf906dc7e423a9d71f9fe 100644
Binary files a/mlsys2022style/figures/convergence-speed-cifar10-wo-c-avg-no-mom-vs-mom-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-cifar10-wo-c-avg-no-mom-vs-mom-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/convergence-speed-mnist-dc-fc-vs-fc-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-mnist-dc-fc-vs-fc-2-shards-per-node.png
index 6a2a0cff0055474e62e8c0f7fe453c1120d38809..388911cdb5cf854f14616fa84d93861383c6fca1 100644
Binary files a/mlsys2022style/figures/convergence-speed-mnist-dc-fc-vs-fc-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-mnist-dc-fc-vs-fc-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node.png
index dc12fa66d2f0ab0ad8aafc5f54c0fa92098e9fd8..e5e1da286815013ceead2a64dcdad12f1bbb4aa9 100644
Binary files a/mlsys2022style/figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-mnist-dc-no-c-avg-vs-c-avg-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node.png b/mlsys2022style/figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node.png
index 11b44ef1ff37e8131cd1bc53a4fe163d0953bf7e..32b99663928b475bcf78baa3f20d4b093990aaea 100644
Binary files a/mlsys2022style/figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node.png and b/mlsys2022style/figures/convergence-speed-mnist-dc-random-vs-dc-gs-2-shards-per-node.png differ
diff --git a/mlsys2022style/figures/d-cliques-cifar10-linear-comparison-to-non-clustered-topologies.png b/mlsys2022style/figures/d-cliques-cifar10-linear-comparison-to-non-clustered-topologies.png
new file mode 100644
index 0000000000000000000000000000000000000000..50dbdc5c3547fb7f8fd290f8ff67dfe46dda91aa
Binary files /dev/null and b/mlsys2022style/figures/d-cliques-cifar10-linear-comparison-to-non-clustered-topologies.png differ
diff --git a/mlsys2022style/figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal.png b/mlsys2022style/figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal.png
index 6730130b49def3b1c1c5ce7edd7642b33d3b4725..610a707ddf0a64639f77ea241b62268cd724b0bb 100644
Binary files a/mlsys2022style/figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal.png and b/mlsys2022style/figures/d-cliques-cifar10-w-clique-avg-impact-of-edge-removal.png differ
diff --git a/mlsys2022style/figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal.png b/mlsys2022style/figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal.png
index a57823e16683902713ec9d4134abbfab98c8ea62..d2ca6bf43deeb703eee2e7c79a8ac724cb6d0e33 100644
Binary files a/mlsys2022style/figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal.png and b/mlsys2022style/figures/d-cliques-cifar10-wo-clique-avg-impact-of-edge-removal.png differ
diff --git a/mlsys2022style/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png b/mlsys2022style/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png
new file mode 100644
index 0000000000000000000000000000000000000000..b336b456cfd1e46b644c9f5b228840c8d62ac341
Binary files /dev/null and b/mlsys2022style/figures/d-cliques-mnist-linear-comparison-to-non-clustered-topologies.png differ
diff --git a/mlsys2022style/figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal.png b/mlsys2022style/figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal.png
index 46b8e770d7c9f84aeb5ffe9b480612dec1ee9741..c2c6879c298483f8665fb0f08c32f57c19a2410b 100644
Binary files a/mlsys2022style/figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal.png and b/mlsys2022style/figures/d-cliques-mnist-w-clique-avg-impact-of-edge-removal.png differ
diff --git a/mlsys2022style/figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal.png b/mlsys2022style/figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal.png
index 0bfcf339fb3bb2a6bcb606e9673b38466cf827dd..744ce0199fc8d6f0e075f48fb9de3f49257bf37d 100644
Binary files a/mlsys2022style/figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal.png and b/mlsys2022style/figures/d-cliques-mnist-wo-clique-avg-impact-of-edge-removal.png differ