Merge remote-tracking branch 'origin/main'

# Conflicts:
#	thesis/sections/background.tex

thesis/sections/background.tex

@@ -182,23 +182,6 @@ The input gate updates the cell state with new information derived from the curr
 Finally, the output gate controls how much of the updated cell state contributes to the hidden state \(h_t\),
 which is passed on to the next time step or used for prediction.
 
-Mathematically, the core LSTM operations are given by:
-
-\[
-    \begin{aligned}
-        f_t &= \sigma(W_f [h_{t-1}, x_t] + b_f) \\
-        i_t &= \sigma(W_i [h_{t-1}, x_t] + b_i) \\
-        \tilde{c}_t &= \tanh(W_c [h_{t-1}, x_t] + b_c) \\
-        c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\
-        o_t &= \sigma(W_o [h_{t-1}, x_t] + b_o) \\
-        h_t &= o_t \odot \tanh(c_t)
-    \end{aligned}
-\]
-
-Here, \( \odot \) denotes element-wise multiplication, and \( \sigma \) is the sigmoid function.
-These equations allow for more stable training and long-range temporal modeling.
-\\
-
 LSTMs are widely used in biomedical applications due to their capacity to handle sequences of variable length and complexity.
 In the context of ovulation prediction, where hormonal patterns exhibit periodicity but also irregularity,
 LSTMs are well-suited to learn relevant time-dependent signals from sequential physiological measurements.
@@ -213,4 +196,117 @@ recurrence in favor of attention mechanisms.
 
 \subsubsection{Transformer Models}\label{subsubsec:transformer_models}
 
+Transformer models are a class of neural architectures that use \emph{self-attention}
+to model dependencies in sequential data without relying on recurrence~\cite{vaswani_attention_2017}.
+Unlike recurrent neural networks (RNNs), Transformers process input sequences in parallel,
+allowing them to model relationships between any pair of input tokens or timesteps directly.
+This mitigates the limitations of recurrent models, such as long-term memory constraints
+and vanishing gradients.
+
+Originally introduced for machine translation, Transformers have proven broadly applicable to
+various sequence modeling tasks due to their flexibility, scalability, and strong performance
+on complex temporal patterns.
+
+At the core of the Transformer is the attention mechanism, which enables the model to compute
+context-aware representations by weighing the importance of different input positions for each output.
+This is achieved through \emph{scaled dot-product attention}, where queries, keys, and values are
+linearly projected from the input and used to compute attention scores.
+
+\begin{figure}
+    \centering
+    \includegraphics[width=0.4\textwidth]{background_transformer_architecture}
+    \caption{The Transformer - architecture for an encoder-decoder model~\cite{vaswani_attention_2017}}
+    \label{fig:background_transformer_architecture}
+\end{figure}
+
+
+Figure~\ref{fig:background_transformer_architecture} illustrates the original encoder-decoder model introduced by~\citeyear{vaswani_attention_2017}.
+
+The Transformer architecture consists of two components: an \emph{Encoder} and a \emph{Decoder}.
+
+\paragraph{Encoder:}
+
+The encoder is responsible for encoding the input into a contextualized representation.
+In the case of machine translation, this input would be a sentence in the source language.
+
+
+The input tokens are first mapped to dense continuous vector representations (embeddings).
+Since the attention mechanism permutation-invariant---that is, it does not inherently encode the order of tokens in the sequence---
+\emph{positional encodings} are added to the token embeddings to provide information about the token positions in the sequence.
+
+Without positional encoding, repeated tokens such as `The` would be indistinguishable
+to the model regardless of their location, even if they play different syntactic or semantic roles.
+Positional encodings, often based on sinusoidal functions, inject a unique position-dependent signal
+into each token, enabling the model to distinguish between identical tokens in different positions.
+
+Inside each encoder block, \emph{Multi-Head-Attention} is applied to the inputs.
+Multi-Head-Attention extends the regular attention mechanism, by adding multiple attention heads that focus on different parts of the embeddings.
+Each head does the scaled dot-product attention independently on its slice of the data.
+The outputs of all heads are then concatenated and combined via a linear projection.
+
+The outputs of the attention mechanism are then processed in a feed forward network to allow for a non-linear projection.
+Residual connections for both the attention and the feed forward allow for better gradient flow and model stability.
+
+\paragraph{Decoder:}
+
+In a sequence-to-sequence Transformer, the decoder generates the output sequence autoregressively,
+using the contextualized representation produced by the encoder.
+
+At inference time, generation begins with a special \emph{start-of-sequence} token.
+Like the encoder, the decoder embeds its inputs and augments them with positional encodings
+to retain information about token order.
+
+To ensure that the model does not access future tokens during training,
+a \emph{look-ahead mask} is applied within the decoder’s self-attention mechanism.
+This masking ensures that each position can only attend to earlier positions in the sequence,
+preventing information leakage.
+This component is referred to as \emph{masked multi-head self-attention}.
+
+Following the masked self-attention, the decoder incorporates information from the encoder
+via a \emph{cross-attention} layer.
+Here, the decoder’s hidden states act as queries, while the encoder’s outputs serve as keys and values.
+This allows the decoder to condition its predictions on the entire encoded input sequence.
+
+The output of the cross-attention layer is passed through a position-wise feed-forward network
+and further normalization and residual connections, analogous to the encoder blocks.
+
+For tasks such as machine translation, the final decoder outputs are linearly projected
+to the target vocabulary size, and a softmax function is applied to produce a probability distribution
+over possible next tokens.
+
+During inference, tokens are sampled sequentially from this distribution and fed back into the decoder for the next prediction step.
+This process continues until a special \emph{end-of-sequence} token is generated, indicating that the model has completed the output sequence.
+
+Depending on the use case and data complexity, multiple encoder and decoder layers can be stacked
+to increase model capacity and improve predictive performance.
+
+While originally developed for machine translation, the Transformer architecture has since been applied
+successfully to a range of tasks, including time-series forecasting and biomedical data analysis(\cite{wu_deep_nodate,zeng_are_2022}).
+Its ability to model long-range dependencies without recurrence makes it particularly suited for biomedical time-series,
+where signals may be irregular, noisy, or span varying temporal scales.
+
+\subsubsection{Convolutional Layers as Temporal Feature Extractors}
+For high-resolution time-series data, the input dimensionality can become large,
+especially in models like Transformers that process the entire sequence in parallel.
+This can lead to increased memory consumption and slower training.
+To mitigate this and retain as much information as possible, convolutional layers can be used
+to reduce the sequence length while preserving important local patterns.
+
+In this context, one-dimensional convolutions act as learnable filters that slide over the input sequence to extract temporal features.
+Each filter is parameterized to respond to specific local structures in the data, such as peaks, slopes, or short-term motifs.
+By adjusting the \emph{stride}—the step size of the convolution—the model can control the degree of downsampling,
+effectively reducing the number of time steps passed to subsequent layers.
+
+Additional dimensionality reduction can be achieved using pooling operations, such as \emph{max pooling}, which retains only the maximum value within a given window.
+These techniques reduce the computational load while maintaining salient information for downstream processing.
+
+Figure~\ref{fig:background_convolution_example} illustrates a simple one-dimensional convolution applied to a sequence using a filter of size 3.
+The stride determines how far the filter moves at each step, affecting both the resolution and length of the resulting feature map.
+
+\begin{figure}
+    \centering
+    \includegraphics[width=0.6\textwidth]{background_convolution_example}
+    \caption{Example of a simple 1-D convolution on an input sequence.}
+    \label{fig:background_convolution_example}
+\end{figure}