\item Processes are what execute your program. They have resources
allocated specifically to them such as memories.
\item Threads are an execution unit for your program. This is typically
what executes instructions. There can be many per processes and they
share the resources of the process.
\end{itemize}
\pause
\vfill
\item Shared Memory:
\begin{description}
\item In a shared memory model all data (memory) are visible to all threads (workers) in your application
\item This means that they can get required data easily but also means that one has to be careful that when updating memory only one thread is writing to a particular address
\end{description}
\item Distributed Memory:
\begin{description}
\item Here the workers only see a small part of the overall memory and if they need data that are visible to another worker they have to explicitly ask for it
\item The advantage is that, if the algorithm scales, we can use more nodes to increase the performance and/or problem size
\item All the largest HPC systems use such a distributed memory model.
\end{description}
\end{itemize}
\begin{itemize}
\item In a shared memory model, all data is visible to all threads
\item All the threads can read and write to the same memory space, which
can be problematic if it happens at the same time!
\item Limited to one CPU
\item Multi-threading is often implemented using OpenMP (in HPC codes)
\end{itemize}
\pause
\vfill
\item Distributed Memory:
\begin{itemize}
\item Many processes with distinct memory spaces. The processes need to
explicitly exchange data.
\item We can use many CPUs to increase the performance and/or problem size
\item All the largest HPC systems use such a distributed memory model.
\item The \textit{de facto} standard technology is MPI
\end{itemize}
\end{itemize}
\end{frame}
\subsection{MPI standard and implementations}
\label{sec:implementations}
\begin{frame}
\frametitle{MPI}
\framesubtitle{MPI implementations}
MPI is a standard with many implementations
\frametitle{MPI implementations}
\framesubtitle{}
\begin{itemize}
\item MPICH2
\item MVAPICH2
\item Intel MPI
\item OpenMPI
\item Platform MPI
\item MPI is a standard defined by the MPI Forum committee\\
\url{https://www.mpi-forum.org/}
\item Currently, MPI is at version 4.0
\item\textit{De facto} standard for distributed memory
\item The MPI Forum defines the standard and MPI vendors implement it
\item There are many different implementations
\begin{itemize}
\item MPICH
\item MVAPICH (based on MPICH)
\item Intel MPI
\item OpenMPI
\item Platform MPI (IBM)
\item Etc.
\end{itemize}
\end{itemize}
\end{frame}
\subsection{Compiling an MPI program}
\label{sec:compiling}
\begin{frame}
\frametitle{MPI}
\framesubtitle{MPI code compiler}
\cmd{mpicc/mpif90/...} is really a wrapper for the standard (GCC and Intel) compilers
\frametitle{Compiling MPI code}
\framesubtitle{}
\begin{itemize}
\item MPI is just another library! You need to include and link it to your executable.
\item Hopefully, vendors provide wrappers to make our life easier
\item A few examples: \cmd{mpicc}, \cmd{mpiicc}, \cmd{mpicxx},
\cmd{mpiifort}, \cmd{mpif90}, \cmd{mpifort}, etc.
\item Which one to use? It depends...
\begin{itemize}
\item In general, \cmd{cc} corresponds to C, \cmd{cxx} or \cmd{c++} to C++, and
\cmd{fort} or \cmd{fXY} to Fortran
\item Then, it depends on the vendor, \eg{}:
\begin{itemize}
\item For Intel MPI, it is better to use \cmd{mpiicc}, \cmd{mpiicxx},
and \cmd{mpiifort}
\item OpenMPI advises you to use \cmd{mpicc}, \cmd{mpicxx},
and \cmd{mpifort}
\end{itemize}
\item While you may have many wrappers available, look at the vendor's
documentation!
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}[fragile]
\frametitle{MPI wrappers}
\framesubtitle{Example with Intel MPI}
\begin{itemize}
\item I would like to compile my C code with Intel compiler and MPI
