further work on thesis background section

thesis/main.tex

@@ -12,6 +12,7 @@
 \graphicspath{{resources/figures/}}
 
 \usepackage[style=ieee, backend=biber]{biblatex}
+\usepackage{blindtext}
 \addbibresource{../main.bib}
 
 % Document

thesis/sections/background.tex

@@ -1,6 +1,7 @@
 %! Author = alex
 %! Date = 3/7/25
 
+
 \section{Background}\label{sec:background}
 
 \subsection{Physiological Background}\label{subsec:physiological_background}
@@ -13,13 +14,13 @@ During the follicular phase, the ovarian follicles mature, and the endometrium (
 in preparation for a potential implantation of a fertilized egg.
 Around day 14 of a typical cycle, ovulation occurs, marking the transition to the luteal phase.
 Ovulation refers to the rupture of the mature ovarian follicle and the release of an egg cell into the fallopian tube.
-Figure\ref{fig:background_basic_female_reproductive_system} illustrates the female reproductive system,
+Figure~\ref{fig:background_basic_female_reproductive_system} illustrates the female reproductive system,
 including the ovaries and the fallopian tubes.
 \\
 \begin{figure}
     \centering
     \includegraphics[width=0.4\textwidth]{background_female_reproductive_organs}
-    \caption{The basic female reproductive system\cite{wikimedia_commons_basic_2019}.}
+    \caption{The basic female reproductive system~\cite{wikimedia_commons_basic_2019}.}
     \label{fig:background_basic_female_reproductive_system}
 \end{figure}
 Ovulation is triggered by a surge in \textbf{luteinizing hormone (LH)}
@@ -35,19 +36,19 @@ with the unfertilized egg.
 This process, known as menstruation, marks the beginning of a new cycle.
 \\
 The menstrual cycle typically lasts around 28 days, with ovulation occurring near the midpoint.
-However, variations, particularly in the follicular phase length, are common and can be influenced by factors such as stress, diet, exercise and age\cite{silberstein_physiology_2000}.
+However, variations, particularly in the follicular phase length, are common and can be influenced by factors such as stress, diet, exercise and age~\cite{silberstein_physiology_2000}.
 Figure~\ref{fig:background_menstrual_cycle_physiology} provides a detailed overview of the hormonal and physiological changes
 throughout the menstrual cycle.
 
 \begin{figure}
     \centering
     \includegraphics[width=0.6\textwidth]{background_menstrual_cycle_physiology}
-    \caption{Physiological changes during the menstrual cycle\cite{pedroso_menstrual_2022}.}
+    \caption{Physiological changes during the menstrual cycle~\cite{pedroso_menstrual_2022}.}
     \label{fig:background_menstrual_cycle_physiology}
 \end{figure}
 
 Not every cycle results in ovulation—a phenomenon known as anovulation—which leads to a monophasic temperature pattern.
-Anovulation can have various causes, including hormonal imbalances, stress, or underlying health conditions\cite{rosenfield_adolescent_2013}.
+Anovulation can have various causes, including hormonal imbalances, stress, or underlying health conditions~\cite{rosenfield_adolescent_2013}.
 
 Anovulation is reflected in temperature data as either an absence of a clear temperature rise or a rise
 that is insufficient in magnitude or duration to be considered a reliable indicator of ovulation.
@@ -63,15 +64,15 @@ Even ultrasound imaging can only confirm that an egg was released from its folli
 Throughout the menstrual cycle, the chance of fertilization varies significantly.
 An egg cell released from the ovary during ovulation, can be fertilized for up to 24 hours.
 However, since male sperm cells can survive up to 6 days inside the female reproductive tract,
-the fertile window is typically defined as the five days before ovulation until one day after ovulation\cite{dunson_day-specific_1999}.
-Research by \citeauthor{dunson_day-specific_1999} has shown that the highest chance of fertilization is around one day before ovulation,
+the fertile window is typically defined as the five days before ovulation until one day after ovulation~\cite{dunson_day-specific_1999}.
+Research by~\citeauthor{dunson_day-specific_1999} has shown that the highest chance of fertilization is around one day before ovulation,
 as illustrated in Figure~\ref{fig:background_pregnancy_chance}.
 
 \begin{figure}
     \centering
     \includegraphics[width=0.7\textwidth]{background_pregnancy_chance_over_time}
     \caption{Chance of fertilization depending on the day of the menstrual cycle.
-    The highest chance is around one day before ovulation\cite{dunson_day-specific_1999}.}
+    The highest chance is around one day before ovulation~\cite{dunson_day-specific_1999}.}
     \label{fig:background_pregnancy_chance}
 \end{figure}
 
@@ -86,7 +87,7 @@ Several physiological signs correlate with ovulation and can be used for predict
 As shown in Figure~\ref{fig:background_menstrual_cycle_physiology}, these include hormonal fluctuations (LH and FSH surges) and
 changes in body temperature.
 Additionally, variations in cervical mucus consistency, salivary ferning patterns,
-and electrical resistance of the skin and vaginal mucosa have been observed\cite{silberstein_physiology_2000}.
+and electrical resistance of the skin and vaginal mucosa have been observed~\cite{silberstein_physiology_2000}.
 
 Among these, ultrasonography provides the most accurate confirmation of ovulation by detecting follicular changes,
 but it is costly and requires specialized equipment.
@@ -95,7 +96,7 @@ but they require frequent testing.
 Temperature-based methods, particularly body temperature tracking, offer a non-invasive alternative by
 detecting the slight temperature rise that follows ovulation.
 Advances in wearable technology have further enabled continuous and automated temperature monitoring,
-improving accessibility and usability\cite{alexander_fertilitatsmonitoring_2014, luo_detection_2020, yu_tracking_2022}.
+improving accessibility and usability~\cite{alexander_fertilitatsmonitoring_2014, luo_detection_2020, yu_tracking_2022}.
 
 \subsection{Technical Background}\label{subsec:technological_background}
 
@@ -103,65 +104,111 @@ improving accessibility and usability\cite{alexander_fertilitatsmonitoring_2014,
 Time series analysis is a fundamental tool for studying sequential data that evolves over time.
 Unlike other data types, time series data has an inherent temporal order, where each data point is associated
 with a timestamp, capturing its dependence on past values.
-There are usually two main goals:
-understanding the underlying mechanisms that lead to the observed data and predicting future data points based on the
-historical information and potentially external factors\cite{cryer_time_2008}
+Time series analysis typically serves two main goals:
+Understanding the underlying mechanisms that lead to the observed data and predicting future data points based on the
+historical information and potentially external factors~\cite{cryer_time_2008}
 \\
 Time series analysis encompasses various methods, ranging from simple statistical models to complex deep learning architectures.
 Classical methods
 
-\subsubsection{Transformer Models}\label{subsubsec:transformer_models}
 
-\subsubsection{Temporal-Fusion-Transformer Models}\label{subsubsec:temporal_fusion_transformer_models}
-The Temporal Fusion Transformer (TFT) is a deep learning model designed for time series forecasting.
-It's a novel approach at time series analysis and forecasting presented by Google in 2020\cite{lim_temporal_2020}.
-The main focus of this architecture is to both achieve high prediction accuracy while maintaining interpretability.
-It differs from a traditional implementation of a transformer architecture by incorporating the support for:
+In the following, we will introduce the two most common approaches used for machine learning on time series data,
+LSTMs and transformers.
+
+\subsubsection{RNN and LSTM Networks}\label{subsubsec:lstm_networks}
+Recurrent Neural Networks (RNNs) are extensions of classical neural networks that incorporate cyclic connections between neurons.
+These recurrent connections allow the network to retain information from previous inputs by feeding the hidden state
+from a prior time step into the current one, enabling a form of temporal memory.
+
+In practice, this means that input data is processed sequentially, one step at a time.
+At each step \(t\), the input \(x_t\) is combined with the previous hidden state \(h_{t-1}\) to produce a new
+hidden state \(h_t\), which contributes to the output \(o_t\).
+This process allows the network to learn temporal dependencies and model sequential data effectively~\cite{medsker_recurrent_1999}.
+
+However, this approach has the downside that the model cannot explicitly control how it remembers or forgets information at each step,
+limiting its ability to manage long-term dependencies.
+During training via backpropagation, the weights of a neural network are updated based on the partial derivatives of the loss function.
+The more propagations required (e.g., in deeper networks), the more multiplications are needed to compute gradients for earlier weights.
+
+RNNs are typically trained using \textit{backpropagation through time} (BPTT), in which gradients are propagated through many time steps,
+leading to numerical instability.
+If the gradients shrink exponentially, the model suffers from the \emph{vanishing gradient} problem;
+if they grow exponentially, it results in \emph{exploding gradients}~\cite{hochreiter_vanishing_1998}.
+In both cases, learning is significantly impaired.
+
+Exploding gradients can often be mitigated using techniques such as \emph{gradient clipping},
+where the magnitude of the gradient is capped---typically within a range of \([-1, 1]\)---to stabilize training.
+
+Figure~\ref{fig:rnn_unfolded} illustrates the unfolded structure of an RNN across three time steps.
+This technique, known as \emph{unfolding}, clarifies how sequential inputs update the hidden state and generate
+outputs at each time step.
+
+\begin{figure}
+    \centering
+    \includegraphics[width=0.7\textwidth]{recurrent_neural_network_unfold}
+    \caption{Schematic diagram of the unfolded structure of a recurrent neural network~\cite{fdeloche_english_2017}}
+    \label{fig:rnn_unfolded}
+\end{figure}
+
+To address the vanishing gradient problem and enable better long-term memory,
+\emph{Long Short-Term Memory} (LSTM) networks were introduced by \citeauthor{hochreiter_long_1997}~\cite{hochreiter_long_1997}.
+LSTMs extend the RNN architecture by incorporating a memory cell and a series of gates that regulate the flow of
+information: the \emph{forget gate}, the \emph{input gate}, and the \emph{output gate}.
+
 \begin{itemize}
-    \item Multiple inputs (static, past and future variables)
-    \item Long sequences
-    \item Uncertainty
+    \item The \emph{forget gate} determines which information from the previous cell state should be discarded.
+    \item The \emph{input gate} controls what new information is added to the cell state.
+    \item The \emph{output gate} selects relevant parts of the current cell state to produce the output and the next hidden state.
 \end{itemize}
-Figure~\ref{fig:background_tft_architecture} provides an overview of the TFT architecture.
 
-\paragraph{1. Gating Mechanisms}
-As the precise relationship between input variables is hard to anticipate in advance, the authors introduce a mechanisms,
-that allows the model to learn this relationship, giving it the ability to choose whether to apply a non-linear
-transformation to the inputs.
-Sometimes, the relationship between variables is linear, and applying a non-linear transformation would only add noise.
-They call this mechanism Gated Residual Network (GRN).
-GRNs take the primary input vector $a$ and an optional context vector $c$ and compute the output as follows:
-\begin{align}
-\text{GRN}_{\omega}(a, c) &= \text{LayerNorm}(a + \text{GLU}_{\omega}(\eta_1)) \label{eq:grn} \\
-\eta_1 &= W_{1,\omega} \eta_2 + b_{1,\omega} \label{eq:eta1} \\
-\eta_2 &= \text{ELU}(W_{2,\omega} a + W_{3,\omega} c + b_{2,\omega}) \label{eq:eta2}
-\end{align}
-
-ELU refers to the Exponential Linear Unit activation function, which is a variant of the ReLU function,
-that speeds up training and alleviates the vanishing gradient problem\cite{clevert_fast_2016}.
-LayerNorm refers to a more robust normalization technique compared to batch normalization, which is used to stabilize
-and speed up training in deep neural networks\cite{ba_layer_2016}.
-Here, $\omega$ denotes weight sharing.
-
-For large $W_{2,\omega}\ a + W_{3\omega}\   c + b_{2,\omega} \gg 0$ the output of the ELU is approximately $a$, acting
-as identity function.
-For negative values, it generates a constant output, resulting in a linear transformation of the input.
-Gating layers based on the Gated Linear Unit (GLU)\cite{dauphin_language_2017} provide the model with the ability to
-suppress any parts of the architecture not needed for the prediction in a dataset.
-
-
-\paragraph{2. Variable Selection Networks (VSNs)}
-Variable Selection Networks (VSNs) are a key component of the TFT architecture, allowing the model to both
-select input variables most relevant to the prediction and removing unnecessary, noisy inputs, that could negatively
-impact prediction performance.
-Each component, i.e., static, past, and future variables, use separate VSNs to select the most relevant features.
-Inputs to the VSN use entity embeddings for categorical features or a linear transformation for continuous features.
-This transforms the inputs into the ($d_{model}$)-dimensional space, where $d_{model}$ is the model's hidden dimension.
+Each gate employs a sigmoid activation function to regulate the flow of information,
+allowing LSTMs to preserve and update memory over long sequences.
 
 
 \begin{figure}
     \centering
-    \includegraphics[width=0.8\textwidth]{background_tft_architecture}
-    \caption{TFT architecture overview\cite{lim_temporal_2020}.}
-    \label{fig:background_tft_architecture}
+    \includegraphics[width=0.6\textwidth]{lstm_cell_diagram}
+    \caption{Diagram of a LSTM cell showing the flow of information \cite{chevalier_english_2018}}
+    \label{fig:lstm_architecture}
 \end{figure}
+
+Figure~\ref{fig:lstm_architecture} illustrates the architecture of an LSTM cell.
+On the left, the previous cell state \(C_{t-1}\) represents the long-term memory,
+while the hidden state \(h_{t-1}\) encodes the short-term memory from the preceding time step.
+The current input \(x_t\) is processed together with these states to update the cell.
+The resulting new cell state \(C_t\) and hidden state \(h_t\) are passed forward to the next time step or used to produce the model’s output.
+Within the cell, the forget gate determines how much of the previous cell state \(C_{t-1}\) is retained.
+The input gate updates the cell state with new information derived from the current input and previous hidden state.
+Finally, the output gate controls how much of the updated cell state contributes to the hidden state \(h_t\),
+which is passed on to the next time step or used for prediction.
+
+Mathematically, the core LSTM operations are given by:
+
+\[
+    \begin{aligned}
+        f_t &= \sigma(W_f [h_{t-1}, x_t] + b_f) \\
+        i_t &= \sigma(W_i [h_{t-1}, x_t] + b_i) \\
+        \tilde{c}_t &= \tanh(W_c [h_{t-1}, x_t] + b_c) \\
+        c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\
+        o_t &= \sigma(W_o [h_{t-1}, x_t] + b_o) \\
+        h_t &= o_t \odot \tanh(c_t)
+    \end{aligned}
+\]
+
+Here, \( \odot \) denotes element-wise multiplication, and \( \sigma \) is the sigmoid function.
+These equations allow for more stable training and long-range temporal modeling.
+\\
+
+LSTMs are widely used in biomedical applications due to their capacity to handle sequences of variable length and complexity.
+In the context of ovulation prediction, where hormonal patterns exhibit periodicity but also irregularity,
+LSTMs are well-suited to learn relevant time-dependent signals from sequential physiological measurements.
+
+While powerful, LSTMs can be computationally intensive and sensitive to hyperparameter tuning.
+Therefore, they are often compared with alternative architectures,
+including simpler feedforward networks and more recent attention-based models,
+to evaluate trade-offs in performance, interpretability, and computational cost.
+
+The next section introduce the \emph{Transformer} architecture, a more recent alternative that forgoes
+recurrence in favor of attention mechanisms
+
+\subsubsection{Transformer Models}\label{subsubsec:transformer_models}