The Dat Protocol\footnote{\url{https://www.datprotocol.com/}} is a peer-to-peer protocol that allows sharing files in a distributed network. It is based on append-only logs Secure-Scuttlebutt (SSB) but its community is more active, and it provides more mature libraries.
The Dat Protocol\footnote{\url{https://www.datprotocol.com/}} is a peer-to-peer protocol that allows sharing files in a distributed network. It is based on append-only logs like Secure-Scuttlebutt (SSB) [ref] but it provides more mature core libraries.
As a distributed file sharing protocol, it differs from BitTorrent[ref] in that contents can be updated by their author.
As a distributed file sharing protocol, it differs from BitTorrent in that contents can be updated by their author.
This report aims at showing that Dat can be used with a Smallworld setup, to easily share files that could be often updated among peers, similarly to a Dropbox instance. We will explain how to configure Dat to replicate files, and we will cover the case where we replicate files between two clients A and B with the Raspberry Pi acting as an intermediary in case only one of the clients is online at a given time.
This report aims at showing that Dat can be used with a Smallworld setup, to continuously replicate files among peers, similarly to a Dropbox instance. We will explain how to configure Dat to continously replicate files, and we will cover the case where we replicate files between two clients A and B with the Raspberry Pi acting as an intermediary in case only one of the clients is online at a given time.
% \subsection{Installation}
% The installation of Dat is quite straightforward. The dat-cli command utility allows to simply interact with Dat files. The commands in Figure \ref{cmd:dat_install} are used to install dat-cli.
% % TODO: try replicating remotely
% It must be installed on both the emitter and the receiver devices.
\subsection{Real time synchronization with dat-store}
\subsection{Real time synchronization with dat-store}
Dat provides a command tool called \emph{dat-cli} that is advised for downloading and syncing Dat archives, but it is not able to run in background, and cannot be remotely controlled and is thus not practical for this application. \\
A more suited tool is \emph{dat-store}. It implements the same commands as \emph{dat-cli} and on top of that it provides a service that can be run in background, and can be remotely controlled. \\
Dat-store is a command line tool that provides commands for downloading and syncing Dat archives and a service that can be run in background, and can be remotely controlled. \footnote{People usually recommend using a command line tool called \emph{dat-cli} for downloading and syncing Dat archives, but it is not able to run in background, and cannot be remotely controlled and is thus not practical for this application.}\\
Dat-store introduces the concept of provider. Providers are instances of dat-store running either locally or remotely. The default provider is your local instance, but you can also configure other providers, like the Raspberry Pi.
Dat-store introduces the concept of provider. Providers are instances of dat-store running either locally or remotely. The default provider is your local instance, but you can also configure other providers, like the Raspberry Pi.
This notion is particularly useful for remote control of other providers. \\
This notion is particularly useful for remote control of other providers. \\
% It is possible to create a shared folder using \emph{dat share} in the folder to share. This command will give a dat url starting with \emph{dat://} that can be used to retrieve the files on other clients.
% A set of useful commands:
% \begin{itemize}
% \item \emph{dat share}, to share a folder (starts a daemon that waits for incoming connexions)
% \item \emph{dat sync}, tries to retrieve newer version of the files, and share the files for incoming connexions
% \item \emph{dat clone <url> <folder>}, retrieves files from a dat url
% \item \emph{dat pull}, tries to retrieve a newer version of the files
% \end{itemize}
% \subsubsection{Real time synchronization with dat-store}
% TODO: what's dat-store: reverse the order of the following
% A tool to do just that is .
% On the opposite, Dat does not directly provide a daemon to run in background and automatically synchronize folders for you. \\
Dat-store should be installed on both the client and the Raspberry Pi using
Dat-store should be installed on both the client and the Raspberry Pi using
\begin{lstlisting}
\begin{lstlisting}
npm install -g dat-store
npm install -g dat-store
\end{lstlisting}
\end{lstlisting}
\textbf{Note:} In case you have permission errors, you may need to fix the permissions of your NPM installations. See the \href{https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally#manually-change-npms-default-directory}{manual guide}. Using sudo instead won't fix the problem. \\
\textbf{Note:} In case you have permission errors, you may need to fix the permissions of your NPM installations. See the manual guide\footnote{\url{https://docs.npmjs.com/resolving-eacces-permissions-errors-when-installing-packages-globally}}. Using sudo instead won't fix the problem. \\
You should then configure a systemd service to run the store in background. You have to configure both your device and the Raspberry Pi. For the user device you can follow Configuration \ref{cmd:dat_store_systemd_user}. For the Raspberry Pi you can follow Configuration \ref{cmd:dat_store_systemd_rasp}.
You should then configure a systemd service to run the store in background. You have to configure both your device and the Raspberry Pi. For the user device you can follow Configuration \ref{cmd:dat_store_systemd_user} below. For the Raspberry Pi you can follow Configuration \ref{cmd:dat_store_systemd_rasp}.
\begin{figure}
\begin{lstlisting}[label=cmd:dat_store_systemd_user, caption=Configure dat-store systemd service on the user device]
\begin{lstlisting}[label=cmd:dat_store_systemd_user, caption=Configure dat-store systemd service on the user device]
# This will create the service file.
# This will create the service file.
sudo cat << EOF | sudo tee /etc/systemd/system/dat-store.service > /dev/null
sudo cat << EOF | sudo tee /etc/systemd/system/dat-store.service > /dev/null
Dat-store provides interesting commands that we will use in our demo\footnote{See the documentation at \url{https://github.com/datproject/dat-store}.}
Dat-store provides interesting commands that we will use in our demo\footnote{See the documentation at \url{https://github.com/datproject/dat-store}.}:
\begin{itemize}
\begin{itemize}
\item\emph{dat-store add <url|path> [provider]}: Adds a folder or a dat url to the dat-store of the specified provider.
\item\emph{dat-store add <url|path> [provider]}: Adds a folder or a dat url to the dat-store of the specified provider.
\item\emph{dat-store set-provider <url> [provider]}: Sets the url of the specified provider.
\item\emph{dat-store set-provider <url> [provider]}: Sets the url of the specified provider.
...
@@ -177,7 +139,6 @@ We can then imagine the case where we have two clients A and B and one Raspberry
...
@@ -177,7 +139,6 @@ We can then imagine the case where we have two clients A and B and one Raspberry
First, A executes the commands from \autoref{cmd:dat_store_example_A}.
First, A executes the commands from \autoref{cmd:dat_store_example_A}.
Then, client B executes the commands from \autoref{cmd:dat_store_example_B}.
Then, client B executes the commands from \autoref{cmd:dat_store_example_B}.
\begin{figure}
\begin{lstlisting}[label=cmd:dat_store_example_A, caption=Commands executed by client A]
\begin{lstlisting}[label=cmd:dat_store_example_A, caption=Commands executed by client A]
dat-store clone ./mydat <url obtained from the previous command>
dat-store clone ./mydat <url obtained from the previous command>
\end{lstlisting}
\end{lstlisting}
\end{figure}
Changes from A to the folder \emph{mydat} will be replicated by B, even if A is not connected at the same time.
Changes from A to the folder \emph{mydat} will be replicated by B, even if A is not connected at the same time.
...
@@ -208,9 +166,12 @@ We also experimented with Pushpin\footnote{\url{https://github.com/automerge/pus
...
@@ -208,9 +166,12 @@ We also experimented with Pushpin\footnote{\url{https://github.com/automerge/pus
However, Pushpin was quite unstable, and Electron was crashing from time to time, and the synchronization also stopped until the window was reloaded. \\
However, Pushpin was quite unstable, and Electron was crashing from time to time, and the synchronization also stopped until the window was reloaded. \\
On top of that, we were unable to install Pushpin on the Raspberry Pi to have a replicating node as Pushpin requires compiling Node 14 and the Raspberry Pi ran out of memory while doing the compilation.
On top of that, we were unable to install Pushpin on the Raspberry Pi to have a replicating node as Pushpin requires compiling Node 14 and the Raspberry Pi ran out of memory while doing the compilation.
\section{Possible improvements and future work}
\section{Conclusion and future work}
A common need we noticed between SSB and Dat is to provide a way to determine when two devices ended their synchronization. SSB provides a Node.js client\footnote{\url{https://github.com/ssbc/ssb-client}} that can be used to determine when new posts are received. A proof of concept is showcased at \url{https://github.com/GuilhemN/ssb-copy-follows/tree/POSTS}. This script could be adapted to make a LED blink for instance to notify the user that the devices have synchronized.
We've shown how to use dat store to replicate files between devices.
In implementing this application, we've noticed a major need shared between SSB and Dat, that is to provide a way to determine when two devices ended their synchronization.
SSB provides a Node.js client\footnote{\url{https://github.com/ssbc/ssb-client}} that can be used to determine when new posts are received. A proof of concept is showcased at \url{https://github.com/GuilhemN/ssb-copy-follows/tree/POSTS}. This script could be adapted to make a LED blink for instance to notify the user that the devices have synchronized.
Another possible improvement for SSB is the refactoring of the AutoFollow with \emph{ssb-client} to copy the follows of its owner. Similarly a proof of concept is available at \url{https://github.com/GuilhemN/ssb-copy-follows/blob/master/index.js}. It could be integrated to a web page exposed at \verb|http://raspberrypi.local/|.
Another possible improvement for SSB is the refactoring of the AutoFollow with \emph{ssb-client} to copy the follows of its owner. Similarly a proof of concept is available at \url{https://github.com/GuilhemN/ssb-copy-follows/blob/master/index.js}. It could be integrated to a web page exposed at \verb|http://raspberrypi.local/|.
...
@@ -220,9 +181,4 @@ Another possible improvement for SSB is the refactoring of the AutoFollow with \
...
@@ -220,9 +181,4 @@ Another possible improvement for SSB is the refactoring of the AutoFollow with \
\addcontentsline{toc}{chapter}{Bibliography}
\addcontentsline{toc}{chapter}{Bibliography}
\printbibliography
\printbibliography
\end{document}
\end{document}
\ No newline at end of file
% TODO: talk about pushpin + reference and what we've observed: that it didn't sync well + npm issue on rasp
% TODO: need for knowing when the replication is done, state + motivate the problem, sketch the solutions