fixes

thesis/main.tex

@@ -17,6 +17,7 @@
 \usepackage[style=ieee, backend=biber]{biblatex}
 \addbibresource{../main.bib}
 \usepackage{booktabs}
+\usepackage{amsfonts}
 
 % Document
 \begin{document}

thesis/sections/background.tex

@@ -14,15 +14,6 @@ During the follicular phase, the ovarian follicles mature, and the endometrium (
 in preparation for a potential implantation of a fertilized egg.
 Around day 14 of a typical cycle, ovulation occurs, marking the transition to the luteal phase.
 Ovulation refers to the rupture of the mature ovarian follicle and the release of an egg cell into the fallopian tube.
-Figure~\ref{fig:background_basic_female_reproductive_system} illustrates the female reproductive system,
-including the ovaries and the fallopian tubes.
-
-\begin{figure}[htb]
-    \centering
-    \includegraphics[width=0.4\textwidth]{background_female_reproductive_organs}
-    \caption{The basic female reproductive system~\cite{wikimedia_commons_basic_2019}.}
-    \label{fig:background_basic_female_reproductive_system}
-\end{figure}
 
 Ovulation is triggered by a surge in \textbf{luteinizing hormone (LH)}
 and \textbf{follicle-stimulating hormone (FSH)}, following a peak in estradiol levels.
@@ -36,10 +27,12 @@ If fertilization does not occur, progesterone levels drop, leading to the sheddi
 with the unfertilized egg.
 This process, known as menstruation, marks the beginning of a new cycle.
 
-\begin{figure}[htb]
+\begin{figure}[htbp]
     \centering
     \includegraphics[width=0.6\textwidth]{background_menstrual_cycle_physiology}
-    \caption{Physiological changes during the menstrual cycle~\cite{pedroso_menstrual_2022}.}
+    \caption{Physiological changes during the menstrual cycle~\cite{pedroso_menstrual_2022}.
+    The figure shows the relationship between hormonal fluctuations, physiological changes, and body temperature, with a clear temperature rise around ovulation.}
+
     \label{fig:background_menstrual_cycle_physiology}
 \end{figure}
 
@@ -47,10 +40,11 @@ The menstrual cycle typically lasts around 28 days, with ovulation occurring nea
 However, variations, particularly in the follicular phase length, are common and can be influenced by factors such as stress, diet, exercise and age~\cite{silberstein_physiology_2000}.
 Figure~\ref{fig:background_menstrual_cycle_physiology} provides a detailed overview of the hormonal and physiological changes throughout the menstrual cycle.
 
-\begin{figure}[htb]
+\begin{figure}[htbp]
     \centering
     \includegraphics[width=0.9\textwidth]{background_labeled_cycle}
-    \caption{A cycles temperature curve with its phases and ovulation}
+    \caption{Body core temperature curve across a menstrual cycle. The red line represents a locally smoothed temperature trend.
+    Menstruation, ovulation, and the fertile phase are indicated in red, blue, and green, respectively.}
     \label{fig:background_labeled_cycle}
 \end{figure}
 
@@ -63,7 +57,7 @@ The luteal phase begins at the ovulation and continues until the next menstruati
 Not every cycle results in ovulation—a phenomenon known as anovulation—which leads to a monophasic temperature pattern.
 Anovulation can have various causes, including hormonal imbalances, stress, or underlying health conditions~\cite{rosenfield_adolescent_2013}.
 
-\begin{figure}[htb]
+\begin{figure}[htbp]
     \centering
     \includegraphics[width=0.9\textwidth]{background_anovulatory_cycle}
     \caption{Example of a cycle without an ovulation and the resulting absence of a temperature rise}
@@ -312,23 +306,43 @@ information: the \emph{forget gate}, the \emph{input gate}, and the \emph{output
 Each gate employs a sigmoid activation function to regulate the flow of information,
 allowing LSTMs to preserve and update memory over long sequences.
 
-
 \begin{figure}[htbp]
     \centering
-    \includegraphics[width=0.6\textwidth]{lstm_cell_diagram}
-    \caption{Diagram of a LSTM cell showing the flow of information \cite{chevalier_english_2018}}
+    \includegraphics[width=0.6\textwidth]{background_lstm_structure}
+    \caption{Architecture of a memory cell \( c_j \) and its gate units \( \text{in}_j \) and \( \text{out}_j \).
+    At the center, the self-recurrent linear unit (with weight 1.0) maintains the internal state~\cite{hochreiter_long_1997}.
+    See text for more details.}
     \label{fig:lstm_architecture}
 \end{figure}
 
-Figure~\ref{fig:lstm_architecture} illustrates the architecture of an LSTM cell.
-On the left, the previous cell state \(C_{t-1}\) represents the long-term memory,
-while the hidden state \(h_{t-1}\) encodes the short-term memory from the preceding time step.
-The current input \(x_t\) is processed together with these states to update the cell.
-The resulting new cell state \(C_t\) and hidden state \(h_t\) are passed forward to the next time step or used to produce the model’s output.
-Within the cell, the forget gate determines how much of the previous cell state \(C_{t-1}\) is retained.
-The input gate updates the cell state with new information derived from the current input and previous hidden state.
-Finally, the output gate controls how much of the updated cell state contributes to the hidden state \(h_t\),
-which is passed on to the next time step or used for prediction.
+The LSTM unit maintains an internal state \( s_{c_j}(t) \) for each cell $c_j$ that evolves over time to capture long-term dependencies.
+Figure~\ref{fig:lstm_architecture} shows the structure of a memory cell along with its input and output gates.
+The \emph{input gate} activation \( y_j^{\text{in}}(t) \) controls the influence of the new candidate input \( g(\text{net}_{c_j}(t)) \) on the state update (Eq.~\eqref{eq:state_update}):
+
+\begin{equation}
+    s_{c_j}(t) = s_{c_j}(t - 1) + y_j^{\text{in}}(t) \cdot g(\text{net}_{c_j}(t))
+    \label{eq:state_update}
+\end{equation}
+
+The \emph{output gate} \( y_j^{\text{out}}(t) \), computed from \( \text{net}_{\text{out}_j}(t) \), determines how much of the transformed internal state is passed on as visible output (Eq.~\eqref{eq:cell_output}):
+
+\begin{equation}
+    y_j^{c}(t) = y_j^{\text{out}}(t) \cdot h(s_{c_j}(t))
+    \label{eq:cell_output}
+\end{equation}
+
+\(g\) and \(h\) are differentiable functions, here \emph{tanh}.
+The gate activations themselves are derived from net inputs formed as weighted sums of previous outputs \( y^u(t-1) \), as given by:
+
+\begin{align}
+    \text{net}_{\text{out}_j}(t) &= \sum_u w_{\text{out}_j,u} \, y^u(t - 1) \label{eq:net_out} \\
+    \text{net}_{\text{in}_j}(t) &= \sum_u w_{\text{in}_j,u} \, y^u(t - 1) \label{eq:net_in} \\
+    \text{net}_{c_j}(t) &= \sum_u w_{c_j,u} \, y^u(t - 1) \label{eq:net_c}
+\end{align}
+
+During training, the weights associated with the inputs \( x_t \) and previous outputs \( h_{t-1} \),
+as well as the biases for each gate in a cell, are learned through backpropagation.
+To build more expressive models, memory cells can be stacked in multiple layers, and their outputs concatenated or passed sequentially to higher layers.
 
 LSTMs are widely used in biomedical applications due to their capacity to handle sequences of variable length and complexity.
 In the context of ovulation prediction, where hormonal patterns exhibit periodicity but also irregularity,
@@ -382,56 +396,142 @@ The input tokens are first mapped to dense continuous vector representations (em
 Since the attention mechanism permutation-invariant---that is, it does not inherently encode the order of tokens in the sequence---
 \emph{positional encodings} are added to the token embeddings to provide information about the token positions in the sequence.
 
+
 Without positional encoding, repeated tokens such as `The` would be indistinguishable
 to the model regardless of their location, even if they play different syntactic or semantic roles.
 Positional encodings, often based on sinusoidal functions, inject a unique position-dependent signal
 into each token, enabling the model to distinguish between identical tokens in different positions.
 
-Inside each encoder block, \emph{Multi-Head-Attention} is applied to the inputs.
-Multi-Head-Attention extends the regular attention mechanism, by adding multiple attention heads that focus on different parts of the embeddings.
-Each head does the scaled dot-product attention independently on its slice of the data.
-The outputs of all heads are then concatenated and combined via a linear projection.
+In this work, we use since and cosine functions of different frequencies:
+\begin{align}
+    PE_{\text{pos}, 2i} &= \sin\left(\frac{\text{pos}}{10000^{\frac{2i}{d_{\text{model}}}}}\right), \\
+    PE_{\text{pos}, 2i+1} &= \cos\left(\frac{\text{pos}}{10000^{\frac{2i}{d_{\text{model}}}}}\right)
+\end{align}
+where \(i\) is the dimension of the input and \(pos\) is the position in the sequence.
+This was done according to the original paper~\cite{vaswani_attention_2017}.
 
-The outputs of the attention mechanism are then processed in a feed forward network to allow for a non-linear projection.
-Residual connections for both the attention and the feed forward allow for better gradient flow and model stability.
+\begin{figure}[htbp]
+    \centering
+    \includegraphics[width=0.8\textwidth]{background_attention_mechanism}
+    \caption{(left) Scaled Dot-Product Attention.
+        (right) Multi-Head Attention consisting of several attention layers running in parallel~\cite{vaswani_attention_2017}.}
+    \label{fig:background_attention_mechanism}
+\end{figure}
+
+Inside each encoder block, \emph{Multi-Head Attention} is applied to the input sequence.
+This mechanism allows the model to jointly attend to information from different representation subspaces at different positions.
+The process begins by linearly projecting the input sequence $X \in \mathbb{R}^{n \times d_{\text{model}}}$ into three distinct matrices:
+
+\begin{align}
+    Q &= XW^Q \\
+    K &= XW^K \\
+    V &= XW^V
+\end{align}
+
+where $W^Q$, $W^K$, and $W^V$ are learned projection matrices.
+Each row of $X$ corresponds to the embedding of one token, and the projections produce the queries ($Q$), keys ($K$), and values ($V$) for each token.
+
+The core operation of attention is then computed as:
+
+\begin{equation}
+    \text{Attention}(Q, K, V) = \text{softmax}\left( \frac{QK^\top}{\sqrt{d_k}} \right) V
+\end{equation}
+
+The dot product $QK^\top$ measures the pairwise similarity between queries and keys, scaled by the dimensionality $d_k$ to control the magnitude.
+The softmax normalizes the scores across each row, producing an attention distribution over all input positions.
+
+Multi-head attention runs this mechanism in parallel across $h$ different heads.
+Each head uses different learned projections:
+
+\begin{equation}
+    \text{head}_i = \text{Attention}(XW_i^Q, XW_i^K, XW_i^V)
+\end{equation}
+
+The resulting vectors from all heads are concatenated and projected:
+
+\begin{equation}
+    \text{MultiHead}(X) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O
+\end{equation}
+
+This output is then passed through a position-wise feed-forward network (FFN), applied independently to each token:
+
+\begin{equation}
+    \text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2
+\end{equation}
+
+To stabilize training, the Transformer uses residual connections and layer normalization:
+
+\begin{align}
+    x_1 &= \text{LayerNorm}(X + \text{MultiHead}(X)) \\
+    x_2 &= \text{LayerNorm}(x_1 + \text{FFN}(x_1))
+\end{align}
+
+In summary, each encoder block consists of:
+\begin{itemize}
+    \item A multi-head attention mechanism that projects the input into queries, keys, and values.
+    \item A feed-forward network that processes the attention output.
+    \item Residual connections and normalization applied after both components.
+\end{itemize}
 
 \paragraph{Decoder:}
 
-In a sequence-to-sequence Transformer, the decoder generates the output sequence autoregressively,
-using the contextualized representation produced by the encoder.
+In a sequence-to-sequence Transformer, the decoder generates the target sequence autoregressively, one token at a time, conditioned on the input sequence processed by the encoder.
 
-At inference time, generation begins with a special \emph{start-of-sequence} token.
-Like the encoder, the decoder embeds its inputs and augments them with positional encodings
-to retain information about token order.
+At inference time, decoding begins with a special \emph{start-of-sequence} token. Each token is embedded and augmented with positional encodings to incorporate information about token order, just as in the encoder.
 
-To ensure that the model does not access future tokens during training,
-a \emph{look-ahead mask} is applied within the decoder’s self-attention mechanism.
-This masking ensures that each position can only attend to earlier positions in the sequence,
-preventing information leakage.
-This component is referred to as \emph{masked multi-head self-attention}.
+Each decoder block consists of three main sublayers:
 
-Following the masked self-attention, the decoder incorporates information from the encoder
-via a \emph{cross-attention} layer.
-Here, the decoder’s hidden states act as queries, while the encoder’s outputs serve as keys and values.
-This allows the decoder to condition its predictions on the entire encoded input sequence.
+\begin{enumerate}
+    \item \textbf{Masked Multi-Head Self-Attention}:
+    The decoder first applies multi-head self-attention over its own output sequence.
+    To prevent positions from attending to future tokens (which would leak information during training),
+    a causal \emph{look-ahead mask} is applied to the attention weights.
+    Formally, for input $Y$ to the decoder:
 
-The output of the cross-attention layer is passed through a position-wise feed-forward network
-and further normalization and residual connections, analogous to the encoder blocks.
+    \begin{equation}
+        \text{MaskedAttention}(Q, K, V) = \text{softmax}\left( \frac{QK^\top}{\sqrt{d_k}} + M \right)V
+    \end{equation}
 
-For tasks such as machine translation, the final decoder outputs are linearly projected
-to the target vocabulary size, and a softmax function is applied to produce a probability distribution
-over possible next tokens.
+    where $M$ is a mask matrix with $-\infty$ in positions that should be masked (future positions), and $0$ elsewhere.
 
-During inference, tokens are sampled sequentially from this distribution and fed back into the decoder for the next prediction step.
-This process continues until a special \emph{end-of-sequence} token is generated, indicating that the model has completed the output sequence.
+    \item \textbf{Cross-Attention (Encoder-Decoder Attention)}:
+    The output from the masked self-attention is used to attend to the encoder’s representations.
+    This allows the decoder to incorporate context from the entire source sequence.
+    The queries come from the decoder hidden states, while the keys and values come from the encoder outputs:
 
-Depending on the use case and data complexity, multiple encoder and decoder layers can be stacked
-to increase model capacity and improve predictive performance.
+    \begin{equation}
+        \text{CrossAttention}(Q, K, V) = \text{Attention}(YW^Q, XW^K, XW^V)
+    \end{equation}
+
+    where $X$ is the encoder output, and $Y$ is the decoder’s self-attention output.
+
+    \item \textbf{Feed-Forward Network and Normalization}:
+    The result of the cross-attention layer is passed through a feed-forward network (FFN),
+    followed by residual connections and layer normalization, mirroring the encoder architecture:
+
+    \begin{align}
+        y_1 &= \text{LayerNorm}(Y + \text{MaskedMultiHead}(Y)) \\
+        y_2 &= \text{LayerNorm}(y_1 + \text{CrossAttention}(y_1, X)) \\
+        y_3 &= \text{LayerNorm}(y_2 + \text{FFN}(y_2))
+    \end{align}
+\end{enumerate}
+
+Finally, the output of the last decoder block is projected into the vocabulary space using a learned weight matrix $W_{\text{vocab}}$
+and passed through a softmax function to produce a distribution over the next possible token:
+
+\begin{equation}
+    P(\text{next token} \mid \text{previous tokens}) = \text{softmax}(y_3 W_{\text{vocab}}^\top + b)
+\end{equation}
+
+During inference, the model generates tokens one at a time, feeding each predicted token back into the decoder to generate the next.
+This continues until a special \emph{end-of-sequence} token is produced.
+
+Multiple decoder blocks can be stacked to increase model depth and expressivity.
+While originally developed for machine translation, the Transformer decoder has since been applied to a wide range of tasks,
+including time-series forecasting and biomedical modeling~\cite{wu_deep_nodate,zeng_are_2022}.
+Its ability to model complex, long-range dependencies without recurrence makes it well-suited to domains like biomedical time-series,
+where signals are often irregular and span diverse temporal resolutions.
 
-While originally developed for machine translation, the Transformer architecture has since been applied
-successfully to a range of tasks, including time-series forecasting and biomedical data analysis(\cite{wu_deep_nodate,zeng_are_2022}).
-Its ability to model long-range dependencies without recurrence makes it particularly suited for biomedical time-series,
-where signals may be irregular, noisy, or span varying temporal scales.
 
 \subsubsection{Convolutional Layers as Temporal Feature Extractors}
 For high-resolution time-series data, the input dimensionality can become large,

thesis/sections/introduction.tex

@@ -47,7 +47,4 @@ where false predictions can have severe consequences.
 This work aims to develop an ovulation predictor that is both accurate and generalizable while maintaining interpretability,
 allowing for insights into key variables and patterns influencing the prediction.
 
-
-
-
-% explain focus: improve explainability
+% research questions

thesis/sections/methodology.tex

@@ -347,6 +347,8 @@ Model-specific architectural parameters are:
     \item \textbf{Number of Layers} — Depth of the LSTM stack.
 \end{itemize}
 The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}.
+A fixed dropout rate of \(p = 0.5\) has been selected for each LSTM-layer expect the last,
+as was shown to be an effective value~\cite{pham_dropout_2014}
 
 \subsubsection{Transformer Architecture}\label{subsubsec:transformer_architecture}
 
@@ -375,6 +377,7 @@ Since the task does not require sequence-to-sequence modeling, only the encoder
 Its output—one vector per input token—is aggregated via 1D adaptive average pooling, resulting in a single vector representation per sequence.
 This vector is then passed through a linear projection layer to produce the two target outputs:
 fertility probability and ovulation-over indicator.
+Note, that in contrast to the original use case of machine-translation, not special tokens are necessary here, as we do not perform sequence-to-sequence prediction.
 
 Figure~\ref{fig:methodology_transformer_architecture} shows the overall architecture.
 The stacked inputs and outputs indicate batch processing.
@@ -386,6 +389,8 @@ Model-specific architectural parameters are:
     \item \textbf{Number of Encoder-Layers} — Number of encoder layers to stack
     \item \textbf{Number of Attention-Heads} — Number of attention heads to use in each layer
 \end{itemize}
+For the Transformer models, a fixed dropout of \(p = 0.1\) was selected, as proposed by\cite{vaswani_attention_2017} in
+the original paper.
 The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}.
 
 \subsubsection{Temporal Convolution Layer}
@@ -555,6 +560,13 @@ joint parameter space in a more efficient and principled manner.
 
 \subsubsection{Training Details}\label{subsubsec:training_details}
 
+The trainings were performed on the scientific compute cluster of Leipzig University on NVIDIA A30 GPUs.
+Due to the heterogeneity of both the models and the trainings, a dynamic batch size algorithm was used, that estimated
+the batch size dynamically during training to optimize the resource usage.
+The learning rate was scaled linearly with the batch size to allow for equivalent convergence behaviour~\cite{goyal_accurate_2018}
+The batch size was capped at 2048 to avoid OOM errors on the system.
+%TODO sources, for square rule
+
 \subsection{Evaluation}\label{subsec:evaluation}
 
 A variety of evaluation metrics have been defined, to best capture each aspect of the performance of an ovulation prediction.