archiving.tex

\section{Recording and Archiving of the System State \& Technical Data}  \label{sec:metadata}

\subsection{Operations Readiness Requirement}

The Rubin Project Team shall demonstrate that relevant technical data about the system state and surrounding conditions during which the survey data are being collected are recorded and archived.

\subsection{Objectives}

The objective of this requirement is to ensure that the technical state of the hardware/software systems and the surrounding environment are recorded during the time of survey data collection with sufficient fidelity to be used in support of subsequent processing to produce the LSST science products. This is of particular importance for the determination and correction of systematics in the science data as the survey progresses and statistics improve.  Additionally, this includes the technical data record required to ensure efficient operation and maintenance of the observing facility.   The primary repository of this technical data is the Engineering Facility Database (EFD) - it has two components:
1) a searchable database that captures the time-dependent ``housekeeping'' data and
2) the Large File Annex for non-telemetry records (e.g., configuration files, images, other binary files outside the science pixel data etc.).

Technical data at the time of each observation ({\it e.g.} visit) includes but is not limited to:

\begin{itemize}
	\item Technical "housekeeping" data, which includes telemetry, events and commands from each subsystem component as published to the EFD;

	\item Software version including the history of the
	%new_hd
	\begin{enumerate}
		\item low-level, hardware-related software,
		\item the Engineering User Interfaces, and
		\item Comandable SAL Components
	\end{enumerate}
	$\rightarrow$ Software written by or modified at the Observatory is documented, reviewed and version-controlled on GitHub.
	\item The configurations of all subsystems, including their history
	%new_hd
	$\rightarrow$ Configurations are handled through the following workflows:
	\begin{itemize}
		\item For DM, it is \href{https://developer.lsst.io/work/flow.html}{here}
		\item For T\&S, it is \href{https://tssw-developer.lsst.io/work_management/development_workflow.html#development-workflow}{here}
	\end{itemize}

	\item Meteorological and the environmental state on the Summit\newline
	%new_hd
	$\rightarrow$ The observatory has a weather station and uses satellite images from Meteoblue to monitor and predict the meteorological conditions at and around the Observatory.

	\item Environmental conditions in the dome interior\newline
	%new_hd
	$\rightarrow$ The Environmental awareness system foresees a large number of sensors to monitor the environmental conditions in the dome.

	\item State of atmospheric turbulences -- {\it e.g.} seeing \newline
	%new_hd
	$\rightarrow$ A dedicated DIMM is part of the observatory.

	\item State of sky transparency.\newline
	%new_hd
	$\rightarrow$ A dedicated all-sky camera and the DREAM camera are part of the observatory monitoring cloud coverage.
	The AuxTel is equipped with a spectrograph to make detailed characterization of the sky at the same time and direction as the Simonyi telescope is observing.

\end{itemize}

\subsection{Criteria for Completeness}

Satisfying these criteria includes, at a minimum:
\begin{itemize}
	\item Demonstrate the technical data (see above) are being recorded at the Summit Facility by the EFD at >99\% (TBC) reliability level for a period of at least 30 days - e.g., no significant dropouts in the live database at the Summit Facility;
	\item Demonstrate the Summit Facility database is being mirrored to an EFD at the \sout{Base Facility} US Data Facility with a lag time of no more than 35 seconds, ({\it e.g.} one nominal visit); The Base Facility will only hold a backup copy of the EFD that is not instantly queriable.
	\item Demonstrate the recorded data are being archived for long-term access - a copy at the Base Facility in Chile and a copy at SLAC;
	\item Access to the technical data is achievable through standard monitoring dashboards from all support centers, including the Summit Facility, Base Facility, Headquarters for Operations in Tucson and US Data Center;
	\item Access to the technical data through the use of customizable GUI interface(s) and dashboards; and
	\item Technical data are queryable through Rubin Science Platform tools - e.g., Jupyter Lab notebooks and WEB interface.
\end{itemize}

\subsection{Pre--Operations Interactions}

\sout{Transfer and archive the EFD from the Base Facility to the US Data Center. The US Data Center is located at SLAC for the purpose of construction completeness evaluation. The US Data Center is required for external queries from users outside the immediate Rubin Observatory Project.}

\subsection{Artifacts for Completion}
\begin{itemize}
	\item A report documenting minimum criteria as defined in the criteria section above;
	\item \sout{An SDK and example code for custom dashboards and dashboard templates available through a software repository(s) - {\it e.g.} GitHub or similar} This is now done in Chronograf and does not need code to be written; and
	\item Example code for Rubin Science Platform queries to the EFD available through a software repository - {\it e.g.} GitHub or similar.
\end{itemize}