The Dat Protocol\footnote{\url{https://www.datprotocol.com/}} is a peer-to-peer protocol that allows sharing files in a distributed network. It is based on append-only logs like Secure-Scuttlebutt (SSB) [ref] but it provides more mature core libraries.
The Dat Protocol\footnote{\url{https://www.datprotocol.com/}}~\cite{Robinson2018} is a peer-to-peer protocol that allows sharing files in a distributed network. It is based on append-only logs like Secure-Scuttlebutt (SSB)~\cite{10.1145/3357150.3357396} but it provides more mature core libraries.
As a distributed file sharing protocol, it differs from BitTorrent[ref] in that contents can be updated by their author.
As a distributed file sharing protocol, it differs from BitTorrent\footnote{\url{https://www.bittorrent.com/}} in that contents can be updated by their author.
This report aims at showing that Dat can be used with a Smallworld setup, to easily share files that could be often updated among peers, similarly to a Dropbox instance. We will explain how to configure Dat to replicate files, and we will cover the case where we replicate files between two clients A and B with the Raspberry Pi acting as an intermediary in case only one of the clients is online at a given time.
This report aims at showing that Dat can be used with a Smallworld setup, to easily share files that could be often updated among peers, similarly to a Dropbox instance. We will explain how to configure Dat to replicate files, and we will cover a scenario where we replicate files between two clients A and B with the Raspberry Pi acting as an intermediary when only one of the clients is online at a given time.
\subsection{Real time synchronization with dat-store}
\subsection{Real time synchronization with dat-store}
Dat-store is a command line tool that provides commands for downloading and syncing Dat archives and a service that can be run in background, and can be remotely controlled. \footnote{People usually recommend using a command line tool called \emph{dat-cli} for downloading and syncing Dat archives, but it is not able to run in background, and cannot be remotely controlled and is thus not practical for this application.}\\
Dat-store is a command line tool that provides commands for downloading and syncing Dat archives, as well as a service that can be run in background, notably for remote control. \footnote{People usually recommend using a command line tool called \emph{dat-cli} for downloading and syncing Dat archives, but it is not able to run in background, and cannot be remotely controlled and is thus not practical for this application.}\\
Dat-store introduces the concept of provider. Providers are instances of dat-store running either locally or remotely. The default provider is your local instance, but you can also configure other providers, like the Raspberry Pi.
Dat-store introduces the concept of provider. Providers are instances of dat-store running either locally or remotely. The default provider is your local instance, but you can also configure other providers, like the Raspberry Pi.
This notion is particularly useful for remote control of other providers. \\
This notion is particularly useful for remote control of other providers. \\
Dat-store should be installed on both the client and the Raspberry Pi using
Dat-store should be installed on both the client and the Raspberry Pi using the following command
\begin{lstlisting}
\begin{lstlisting}
npm install -g dat-store
npm install -g dat-store
\end{lstlisting}
\end{lstlisting}
\textbf{Note:} In case you have permission errors, you may need to fix the permissions of your NPM installations. See the manual guide\footnote{\url{https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally}}. Using sudo instead won't fix the problem. \\
\textbf{Note:} In case you have permission errors, you may need to fix the permissions of your NPM installations. See the manual guide\footnote{\url{https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally}}. Using sudo instead won't fix the problem. \\
You should then configure a systemd service to run the store in background. You have to configure both your device and the Raspberry Pi. For the user device you can follow Configuration \ref{cmd:dat_store_systemd_user}below. For the Raspberry Pi you can follow Configuration \ref{cmd:dat_store_systemd_rasp}.
You should then configure a systemd service to run the store in background. You have to configure both your device and the Raspberry Pi. For the user device you can copy paste the commands from Configuration \ref{cmd:dat_store_systemd_user}in a bash terminal. For the Raspberry Pi you can do the same with the commands from Configuration \ref{cmd:dat_store_systemd_rasp}.
\begin{lstlisting}[label=cmd:dat_store_systemd_user, caption=Configure dat-store systemd service on the user device]
\begin{lstlisting}[label=cmd:dat_store_systemd_user, caption=Sequence of bash commands to configure dat-store as a systemd service on the user device. Copy paste in a bash terminal.]
# This will create the service file.
# This will create the service file.
sudo cat << EOF | sudo tee /etc/systemd/system/dat-store.service > /dev/null
sudo cat << EOF | sudo tee /etc/systemd/system/dat-store.service > /dev/null
...
@@ -97,7 +97,7 @@ sudo systemctl start dat-store
...
@@ -97,7 +97,7 @@ sudo systemctl start dat-store
sudo systemctl status dat-store
sudo systemctl status dat-store
\end{lstlisting}
\end{lstlisting}
\begin{lstlisting}[label=cmd:dat_store_systemd_rasp, caption=Configure dat-store systemd service on the raspberry pi]
\begin{lstlisting}[label=cmd:dat_store_systemd_rasp, caption=Sequence of bash commands to configure dat-store as a systemd service on the Raspberry Pi. Copy paste in a bash terminal.]
# This will create the service file.
# This will create the service file.
sudo cat << EOF | sudo tee /etc/systemd/system/dat-store.service > /dev/null
sudo cat << EOF | sudo tee /etc/systemd/system/dat-store.service > /dev/null
...
@@ -139,7 +139,7 @@ We can then imagine the case where we have two clients A and B and one Raspberry
...
@@ -139,7 +139,7 @@ We can then imagine the case where we have two clients A and B and one Raspberry
First, A executes the commands from \autoref{cmd:dat_store_example_A}.
First, A executes the commands from \autoref{cmd:dat_store_example_A}.
Then, client B executes the commands from \autoref{cmd:dat_store_example_B}.
Then, client B executes the commands from \autoref{cmd:dat_store_example_B}.
\begin{lstlisting}[label=cmd:dat_store_example_A, caption=Commands executed by client A]
\begin{lstlisting}[label=cmd:dat_store_example_A, caption=Bash commands executed by client A]
@@ -161,37 +161,31 @@ Changes from A to the folder \emph{mydat} will be replicated by B, even if A is
...
@@ -161,37 +161,31 @@ Changes from A to the folder \emph{mydat} will be replicated by B, even if A is
\section{Beyond Dat}
\section{Beyond Dat}
We also experimented with Pushpin\footnote{\url{https://github.com/automerge/pushpin}}, which is a collaborative coarkboard app. We wanted to showcase a peer-to-peer application with a real-time interface. It is based on hypermerge\footnote{\url{https://github.com/automerge/hypermerge}}, a library providing a JSON-like structure that can be edited concurrently, without worrying about conflicts. And hypermerge is itself based on the Dat protocol.
We also experimented with Pushpin\footnote{\url{https://github.com/automerge/pushpin}}, which is a collaborative coarkboard app. We wanted to showcase a peer-to-peer application with a real-time interface. Pushpin is based on hypermerge\footnote{\url{https://github.com/automerge/hypermerge}}, a library providing a JSON-like structure that can be edited concurrently, without worrying about conflicts. And hypermerge is itself based on the Dat protocol.
However, Pushpin was quite unstable, and Electron was crashing from time to time, and the synchronization also stopped until the window was reloaded. \\
However, Pushpin was quite unstable, and Electron was crashing from time to time, and the synchronization also stopped until the window was reloaded. \\
On top of that, we were unable to install Pushpin on the Raspberry Pi to have a replicating node as Pushpin requires compiling Node 14 and the Raspberry Pi ran out of memory while doing the compilation.
On top of that, we were unable to install Pushpin on the Raspberry Pi to have a replicating node as Pushpin requires compiling Node 14 and the Raspberry Pi ran out of memory while doing the compilation.
\section{Conclusion and future work}
\section{Conclusion and future work}
We've shown how to use dat store to replicate files between devices.
In the previous sections, we have shown how to use dat store to replicate files between devices and a simple usage scenario. Many extensions are possible.
In implementing this application, we've noticed a major need shared between SSB and Dat, that is to provide a way to determine when two devices ended their synchronization.
SSB provides a Node.js client\footnote{\url{https://github.com/ssbc/ssb-client}} that can be used to determine when new posts are received. A proof of concept is showcased at \url{https://github.com/GuilhemN/ssb-copy-follows/tree/POSTS}. This script could be adapted to make a LED blink for instance to notify the user that the devices have synchronized.
First, while implementing this use case, we have noticed a major need shared between SSB and Dat, that is to provide a way to determine when two devices ended their synchronization.
One possible implementation for SSB could be to use the Node.js client\footnote{\url{https://github.com/ssbc/ssb-client}} as it can be used to determine when new posts are received. A proof of concept is showcased at \url{https://github.com/GuilhemN/ssb-copy-follows/tree/POSTS}. This script could be adapted to make a LED blink for instance to notify the user that the devices have synchronized.
Another possible improvement for SSB is the refactoring of the AutoFollow with \emph{ssb-client} to copy the follows of its owner. Similarly a proof of concept is available at \url{https://github.com/GuilhemN/ssb-copy-follows/blob/master/index.js}. It could be integrated to a web page exposed at \verb|http://raspberrypi.local/|.
Second, it appears that much of the activity in the Dat community is migrating toward the Hypercore protocol. In the next section, we detail how to install it on the Raspberry Pi so that future students could extend our work using these libraries instead.
\newpage
\subsection{Dat migration to Hypercore protocol}
\appendix
\section{Dat vs Hypercore protocol}
The Hypercore protocol\footnote{\url{https://hypercore-protocol.org/}} was created in 2016 as a way to provide to Dat more abstract foundations that could be reused among several applications. From then, the Hyper project grew and it was decided in 2020 to separate Dat from Hypercore and to provide both with their own governance\footnote{An explanation video is available at \url{https://www.youtube.com/watch?v=mx52uO5SP7A}}.
The Hypercore protocol\footnote{\url{https://hypercore-protocol.org/}} was created in 2016 as a way to provide Dat more abstract foundations that could be reused among several applications. From then, the Hyper project grew and it was decided in 2020 to separate Dat from Hypercore and to provide both their own governance\footnote{An explanation video is available at \url{https://www.youtube.com/watch?v=mx52uO5SP7A}}.
The Hypercore protocol is now incompatible with the Dat protocol and provides its own CLI tools and libraries.
The Hypercore protocol is now incompatible with the Dat protocol and provides its own CLI tools and libraries.
The main entry of the Hyperspace is the Hyperspace CLI\footnote{\url{https://github.com/hypercore-protocol/cli}}. It allows to create Hyperdrives, folders distributed on the Hyper network, and Bees, which are distributed key-value tables.
The main entry of the Hyperspace is the Hyperspace CLI\footnote{\url{https://github.com/hypercore-protocol/cli}}. It enables the creation of Hyperdrives, i.e.\ folders distributed on the Hyper network, and Bees, which are distributed key-value tables.
The Hypercore provides an actually decentralized implementation for remote peer-to-peer connexion based on DHT (Distributed Hash Tables), while the Dat protocol only supports a rendez-vous approach with a centralized server. \\
While we did not do anything new with Hyper compared to what we did with Dat, we wanted to explain its installation procedure on Raspberry Pis as it is quite complex, and as Hyper will likely be preferred over Dat in the future. \\
Hyper libraries require Node 14 but Node 14 is not officially supporting the Raspberry Pi Zero as it chips an ARMv6 CPU.
In addition, Hypercore is the underlying library of Hyperspace CLI. It provides an actually decentralized implementation for remote peer-to-peer connexion based on DHT (Distributed Hash Tables), while the Dat protocol only supports a rendez-vous approach with a centralized server. \\
The NodeJs team is actually still providing unofficial (and "experimental") builds under \url{https://github.com/nodejs/unofficial-builds/} and those are usable on the Raspberry Pi. You may install them using the scripts provided by \url{https://github.com/sdesalas/node-pi-zero}. You may install the latest version of Node 14 to have a version of node compatible with Hyper. \\
In order to run the Hypercore libraries, the documentation states that Node version 14 or more recent is required. However, Node 14 is not officially supported on the Raspberry Pi Zero as it is based on an ARMv6 CPU.
Thankfully, the NodeJs team is actually still providing unofficial (and "experimental") builds under \url{https://github.com/nodejs/unofficial-builds/} and those are usable on the Raspberry Pi. You may install them using the scripts provided by \url{https://github.com/sdesalas/node-pi-zero}. We tested successfully Node version 14.17.1. \\
You will then be able to install the Hyperspace CLI using \verb|npm install -g @hyperspace/cli|. \\
You will then be able to install the Hyperspace CLI using \verb|npm install -g @hyperspace/cli|. \\
And now you should be able to interact with Hyperdrives using the \emph{hyp} command. You can check its documentation at \url{https://hypercore-protocol.org/guides/hyp/}.
And now you should be able to interact with Hyperdrives using the \emph{hyp} command. You can check its documentation at \url{https://hypercore-protocol.org/guides/hyp/}.
