started tft part in background

thesis/sections/background.tex

@@ -1,10 +1,10 @@
 %! Author = alex
 %! Date = 3/7/25
 
-
 \section{Background}\label{sec:background}
 
 \subsection{Physiological Background}\label{subsec:physiological_background}
+
 \subsubsection{Menstrual Cycle}\label{subsec:menstrual_cycle}
 The menstrual cycle describes the physiological changes in the female body that prepare it for pregnancy.
 It is divided into two phases: the \textbf{follicular phase} and the \textbf{luteal phase}.
@@ -98,6 +98,7 @@ Advances in wearable technology have further enabled continuous and automated te
 improving accessibility and usability\cite{alexander_fertilitatsmonitoring_2014, luo_detection_2020, yu_tracking_2022}.
 
 \subsection{Technical Background}\label{subsec:technological_background}
+
 \subsubsection{Time Series Analysis}\label{subsubsec:time_series_analysis}
 Time series analysis is a fundamental tool for studying sequential data that evolves over time.
 Unlike other data types, time series data has an inherent temporal order, where each data point is associated
@@ -107,4 +108,60 @@ understanding the underlying mechanisms that lead to the observed data and predi
 historical information and potentially external factors\cite{cryer_time_2008}
 \\
 Time series analysis encompasses various methods, ranging from simple statistical models to complex deep learning architectures.
-Classical methods
+Classical methods
+
+\subsubsection{Transformer Models}\label{subsubsec:transformer_models}
+
+\subsubsection{Temporal-Fusion-Transformer Models}\label{subsubsec:temporal_fusion_transformer_models}
+The Temporal Fusion Transformer (TFT) is a deep learning model designed for time series forecasting.
+It's a novel approach at time series analysis and forecasting presented by Google in 2020\cite{lim_temporal_2020}.
+The main focus of this architecture is to both achieve high prediction accuracy while maintaining interpretability.
+It differs from a traditional implementation of a transformer architecture by incorporating the support for:
+\begin{itemize}
+    \item Multiple inputs (static, past and future variables)
+    \item Long sequences
+    \item Uncertainty
+\end{itemize}
+Figure~\ref{fig:background_tft_architecture} provides an overview of the TFT architecture.
+
+\paragraph{1. Gating Mechanisms}
+As the precise relationship between input variables is hard to anticipate in advance, the authors introduce a mechanisms,
+that allows the model to learn this relationship, giving it the ability to choose whether to apply a non-linear
+transformation to the inputs.
+Sometimes, the relationship between variables is linear, and applying a non-linear transformation would only add noise.
+They call this mechanism Gated Residual Network (GRN).
+GRNs take the primary input vector $a$ and an optional context vector $c$ and compute the output as follows:
+\begin{align}
+\text{GRN}_{\omega}(a, c) &= \text{LayerNorm}(a + \text{GLU}_{\omega}(\eta_1)) \label{eq:grn} \\
+\eta_1 &= W_{1,\omega} \eta_2 + b_{1,\omega} \label{eq:eta1} \\
+\eta_2 &= \text{ELU}(W_{2,\omega} a + W_{3,\omega} c + b_{2,\omega}) \label{eq:eta2}
+\end{align}
+
+ELU refers to the Exponential Linear Unit activation function, which is a variant of the ReLU function,
+that speeds up training and alleviates the vanishing gradient problem\cite{clevert_fast_2016}.
+LayerNorm refers to a more robust normalization technique compared to batch normalization, which is used to stabilize
+and speed up training in deep neural networks\cite{ba_layer_2016}.
+Here, $\omega$ denotes weight sharing.
+
+For large $W_{2,\omega}\ a + W_{3\omega}\   c + b_{2,\omega} \gg 0$ the output of the ELU is approximately $a$, acting
+as identity function.
+For negative values, it generates a constant output, resulting in a linear transformation of the input.
+Gating layers based on the Gated Linear Unit (GLU)\cite{dauphin_language_2017} provide the model with the ability to
+suppress any parts of the architecture not needed for the prediction in a dataset.
+
+
+\paragraph{2. Variable Selection Networks (VSNs)}
+Variable Selection Networks (VSNs) are a key component of the TFT architecture, allowing the model to both
+select input variables most relevant to the prediction and removing unnecessary, noisy inputs, that could negatively
+impact prediction performance.
+Each component, i.e., static, past, and future variables, use separate VSNs to select the most relevant features.
+Inputs to the VSN use entity embeddings for categorical features or a linear transformation for continuous features.
+This transforms the inputs into the ($d_{model}$)-dimensional space, where $d_{model}$ is the model's hidden dimension.
+
+
+\begin{figure}
+    \centering
+    \includegraphics[width=0.8\textwidth]{background_tft_architecture}
+    \caption{TFT architecture overview\cite{lim_temporal_2020}.}
+    \label{fig:background_tft_architecture}
+\end{figure}