temperature-based-fertility-prediction-thesis/thesis/sections/methodology.tex

%! Author = alex
%! Date = 3/6/25


\section{Methodology}\label{sec:methodology}

Despite promising results in earlier studies, ovulation prediction remains constrained by small datasets,
assumptions of cycle regularity, and opaque modeling approaches.
To address these limitations, I develop a data-driven framework based on a large,
heterogeneous dataset of real-world menstrual cycles.
My approach emphasizes model transparency, adaptability to irregular patterns, and the predictive utility of
high-resolution core body temperature measurements.
The overall goal is to train a model to predict the fertility-probability and information about the ovulation for
a given day, only relying on past information prior to that day.
This section outlines the methodology used, including preprocessing, labeling, feature extraction, and model architectures.

\subsection{Data Preprocessing}\label{subsec:data_preprocessing}
This section outlines the preprocessing steps applied to the raw temperature data,
including cycle filtering and retrospective ovulation labeling.
These steps ensure that only clean, complete, and labeled cycles are used for model training.

\subsubsection{Data Filtering}\label{subsubsec:data_filtering}

As briefly mentioned in~\ref{subsec:data_background}, not all cycles in the dataset are suitable for training.
Cycles that are either too short (\textless 10 days) or too long (\textgreater 150 days) are excluded,
as they typically indicate erroneous entries, pregnancies, sensor failures, or data processing issues.

In addition, incomplete cycles are filtered out, since new cycles continuously arrive from active users
and may not contain the full sequence of data required for retrospective labeling.

Temperature values outside the physiologically plausible range, below 35\textdegree C or above 43\textdegree C,
are also excluded, as they typically result from sensor malfunction or transmission errors.

\subsubsection{Data Labeling}\label{subsubsec:data_labeling}

Supervised machine learning requires labeled data, i.e., known target values for each input.
In the context of this study, the relevant target is the ovulation day within each menstrual cycle.

The dataset contains over 40{,}000 cycles, making manual annotation unfeasible.
Instead, a retrospective ovulation detection algorithm is employed to assign labels:
(1) whether a cycle is ovulatory or anovulatory, and
(2) if ovulatory, the estimated day of ovulation.

This labeling algorithm was developed in collaboration with a gynecologist and reproductive medicine specialist.
It is trained on a curated reference set of cycles with expert-assigned labels based on domain knowledge and characteristic temperature patterns.

The algorithm operates in two stages:
\begin{enumerate}
    \item \textbf{Cycle classification:} Each cycle is classified as either monophasic (anovulatory) or biphasic (ovulatory), based on the presence of a luteal-phase temperature shift.
    \item \textbf{Ovulation estimation:} For biphasic cycles, the most likely day of ovulation is identified retrospectively using the full temperature curve.
\end{enumerate}

This retrospective labeling provides a practical and scalable proxy for ground truth, enabling training and evaluation across a large, real-world dataset,
particularly given that temperature is a well-established retrospective marker of ovulation.
In internal evaluations, the estimated ovulation day fell within a \(\pm\)2-day window of the expert reference in approximately 86\% of labeled cycles.
These labels serve as the supervisory signal for model training and evaluation.

It is important to note that the labeling algorithm infers ovulation retrospectively from the full temperature curve.
While this approach is practical for large-scale annotation,
it means that the labels represent an approximation rather than a clinical gold standard (e.g., ultrasound or daily LH testing).
Consequently, the trained models learn to reproduce these retrospective estimates, which may deviate from the true
physiological ovulation day in some cycles.
This limitation is addressed further in the Discussion.

The next section details how these labels are incorporated into feature representations and model training.

\subsection{Feature Engineering}\label{subsec:feature_engineering}

The features used as model inputs have been divided into three categories:
\begin{itemize}
    \item \textbf{Static features:} Characteristics, that remain constant across a user's cycle, such as age, height, or average ovulation day
    \item \textbf{Known features:} Inputs known a priori at each time step, such as time of day or calendar-based variables (e.g., month of the year).
    \item \textbf{Observable features:} Inputs available at the current time step, including raw and derived temperature values (e.g, rolling averages).
\end{itemize}
The target variables predicted by the model—like ovulation status or fertility probability—are described separately.

Each feature type can handle categorical and continuous features.
This allows for mixed inputs, such as scalar measurements and class labels, within the same category.

The categorization into three feature types is intended to clarify the conceptual roles of different input types.
While the current models concatenate all features into a single input stream, the distinction allows for flexibility—future models
may process each feature group differently depending on their architectural design.

\subsubsection{Static Features}\label{subsubsec:static_features}

Static features are the features that do not change over the course of a cycle.
They might even be static for all cycles from a specific user, such as age, height and weight.
Static features provide user-specific context that helps the model learn individualized cycle patterns beyond what temperature alone can reveal.

%\begin{table}[htbp]
%    \centering
%    \begin{tabular}{l>{\raggedright\arraybackslash}p{0.65\linewidth}}
%        \toprule
%        \textbf{Feature}                & \textbf{Description}                                        \\
%        \midrule
%        User age                        & Age in years; mean imputed if missing                       \\
%        User height                     & Height in centimeters; mean imputed if missing              \\
%        User weight                     & Weight in kilograms; mean imputed if missing                \\
%        Average cycle length            & Mean length of all previous cycles for this user            \\
%        Cycle length SD                 & Standard deviation of previous cycle lengths                \\
%        Average ovulation day           & Mean day of ovulation from previous cycles                  \\
%        Ovulation SD                    & Standard deviation of ovulation day of previous cycles      \\
%        Ovulatory fraction              & Proportion of prior cycles classified as ovulatory          \\
%        Cycle count                     & Number of previous completed cycles available               \\
%        Avg. pre-ovulation temperature  & Mean temperature in the follicular phase of previous cycles \\
%        Avg. post-ovulation temperature & Mean temperature in the luteal phase of previous cycles     \\
%        \bottomrule
%    \end{tabular}
%    \caption{Static features used as model inputs}
%    \label{tab:static_features}
%\end{table}
%Table~\ref{tab:static_features} shows all static features and their descriptions.

Menstrual cycle characteristics vary significantly with age and BMI~\cite{li_menstrual_2023}.
Including such information is therefore expected to improve predictive performance.

In addition, summary statistics from previous cycles, such as ovulation timing, temperature levels, or the fraction of ovulatory cycles, provide useful individual context.
These features help the model learn subject-specific variability and better estimate the likelihood and timing of ovulation in the current cycle.
If no user-specific context is available yet, population-based mean values are used.
Table~\ref{tab:feature_overview} shows the full list of static input features.

All historical features are computed using only data available prior to the current cycle, ensuring no data leakage and supporting robust, user-adaptive learning.

\subsubsection{Known Features}\label{subsubsec:known_features}
Known features encode temporal context that is available at each time step and independent of physiological measurements.
These help the model interpret observations in relation to time-based structure, including circadian and behavioral rhythms.

\begin{itemize}
    \item \textbf{Time since cycle start} — Provides the model with a relative position within the menstrual cycle.
    \item \textbf{Hour of day} — Helps distinguish between daytime and nighttime patterns, especially relevant for circadian rhythms.
    \item \textbf{Day of the week} — Encodes potential behavioral differences between weekdays and weekends.
    \item \textbf{Month of the year} — Captures seasonal variations in temperature patterns or user behavior.
\end{itemize}

Except for \textit{time since cycle start}, all features are encoded using sine and cosine transforms to preserve their cyclical nature and make them more interpretable for the model.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{resources/figures/methodology/methodology_time_feature_sine_encoded}
    \caption{Sine and cosine encoding of the day-of-week feature.}
    \label{fig:methodology_time_feature_encoding}
\end{figure}

Figure~\ref{fig:methodology_time_feature_encoding} illustrates the sine and cosine encoding of the day-of-week feature.
The cyclical nature of the variable is clearly visible in the transformation.

Although the model architectures used are sequential, the explicit inclusion of these time features allows the models to interpret each time step in a broader context.
More importantly, they enable the detection of gaps in the recording, which would otherwise not be visible from the data alone.

Menstrual cycles can be influenced by weekly rhythms~\cite{ecochard_menstrual_2024}.
For example, menstruation has been found to begin more frequently on Thursdays or Fridays, suggesting that behavioral or social factors may modulate certain events in the cycle.
Including this information could therefore improve the predictive quality of the models.

\subsubsection{Observable Features}\label{subsubsec:observable_features}

Observable features include all time-series inputs available up to the current time step.
They represent real-time physiological signals from which the model must infer ovulatory status.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{resources/figures/methodology/methodology_observable_features}
    \caption{Observable features for a cycle.}
    \label{fig:methodology_observable_features}
\end{figure}

\begin{itemize}
    \item \textbf{Temperature} — Raw intravaginal temperature as recorded by the OvulaRing sensor, sampled every 5 minutes.
    \item \textbf{Rolling Average Temperature} — The mean temperature over a 1-day (288-sample) sliding window, linearly interpolated to preserve the original input resolution.
    \item \textbf{Rolling Window Temperature Minimum} — The minimum temperature observed within a 1-day window, highlighting potential overnight lows or phase-specific dips.
    \item \textbf{Rolling Window Temperature Maximum} — The maximum temperature within a 1-day window, capturing transient peaks or elevated plateaus.
\end{itemize}

These derived features summarize local trends or extrema in the temperature signal, reducing the burden on the model to learn such patterns from raw data.
Special care was taken, so that the sliding window can only look backwards, so that no data leakage can happen.
I extend each windowed feature at the beginning with the starting value, so that the window can be calculated for the first real value already.
The 1-day window length reflects the expected circadian cycle and strikes a balance between temporal sensitivity and signal stability.

Figure~\ref{fig:methodology_observable_features} illustrates the behavior of all observable features within a single cycle.
The rolling extrema delineate the amplitude of the daily temperature variation and accentuate phase transitions.

\subsubsection{Target Features}

Target features represent the outputs that the models are trained to predict.
These can include exogenous biological outcomes or interpretable derivatives of input features.
For this study, the following exogenous variables are used:

\begin{itemize}
    \item \textbf{Fertility / Pregnancy Probability} — The estimated probability of conception from unprotected intercourse on the current day.
    \item \textbf{OV-over Indicator} — A binary variable indicating whether ovulation has already occurred in the current cycle.
\end{itemize}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{resources/figures/methodology/methodology_target_features}
    \caption{
        Target features plotted for a single cycle.
        The OV-over indicator switches on the day of ovulation;
        the fertility probability follows a curve based on known day-specific fecundability~\cite{dunson_day-specific_1999}.
    }
    \label{fig:methodology_target_features}
\end{figure}

The combination of these two targets is intended to provide the user with both physiological and practical insight:
\textit{Fertility probability} conveys the likelihood of conception, but alone does not indicate whether ovulation is yet to come or has already passed.
A fertility probability near zero could mean that ovulation is either in the past or still ahead—information the model alone cannot disambiguate.
Figure~\ref{fig:methodology_target_features} shows an example cycle where the target feature values during the course of a cycle can be seen.

The \textit{OV-over indicator}, by contrast, explicitly marks the post-ovulatory phase, but does not describe conception risk.
Together, the two outputs offer complementary information and improve interpretability for real-time user-facing applications.

As discussed in Section~\ref{sec:discussion}, all predictions are subject to further interpretation before presentation in the product interface.
The model outputs represent data-driven estimates and do not constitute medical advice or diagnostic statements.

Although the models were trained to predict both targets, and fertility-probability as well as the OV-over indicator
are relevant for actual application of the models, the primary focus of this thesis lies on the fertility-probability target,
which is more central to the research objectives.
The OV-over indicator is included due to its practical relevance for potential real-world applications,
but it is not analyzed in depth, as it lies outside the core scope of scientific investigation.

\begin{table}[htbp]
    \centering
    \begin{tabular}{@{}lp{0.62\linewidth}@{}}
        \toprule
        \textbf{Feature Name}           & \textbf{Description}                                        \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Static Features}} \\
        \midrule
        User age                        & Age in years; mean imputed if missing                       \\
        User height                     & Height in centimeters; mean imputed if missing              \\
        User weight                     & Weight in kilograms; mean imputed if missing                \\
        Average cycle length            & Mean length of all previous cycles for this user            \\
        Cycle length STD                & Standard deviation of previous cycle lengths                \\
        Ovulatory fraction              & Proportion of prior cycles classified as ovulatory          \\
        Cycle count                     & Number of previously completed cycles available             \\
        Avg. pre-ovulatory temperature  & Mean temperature in the follicular phase of previous cycles \\
        Avg. post-ovulatory temperature & Mean temperature in the luteal phase of previous cycles     \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Known (Time-Based) Features}} \\
        \midrule
        Time since cycle start          & Hours since the start of the current cycle                  \\
        Hour of day                     & Hour of the day (0–23)                                      \\
        Day of the week                 & Day of the week (0–6)                                       \\
        Month of the year               & Month of the year (0–11)                                    \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Observable Features}} \\
        \midrule
        Raw temperature                 & Temperature value recorded by the sensor                    \\
        Rolling average temperature     & 1-day rolling average (288 measurements)                    \\
        Rolling window temp. minimum    & Minimum temperature within 1-day window                     \\
        Rolling window temp. maximum    & Maximum temperature within 1-day window                     \\
        \bottomrule
    \end{tabular}
    \caption{Overview of all features and their descriptions.}
    \label{tab:feature_overview}
\end{table}

\subsubsection{Time-Series Input Representation}
\label{subsubsec:time_series_input_representation}

All features were normalized based on their empirical distributions.
A \textit{standard scaler} was applied to approximately normal features without outliers,
a \textit{robust scaler} was used for distributions with outliers, and a \textit{MinMax scaler} was used for all others.
All scalers were fit on the training set only and applied to the validation and test sets to avoid data leakage.


For the final data matrix, all features are stacked per timestep.
The static features are repeated for each timestep.
I am aware of possible inefficiencies here.
A side channel for static features might improve predictive efficiency and potential quality, but
this was left out to keep the interfaces the same for compatibility purposes between all tested models

Let
\begin{itemize}
    \item \( T \) be the sequence length (number of time steps)
    \item \( x_t^{\text{obs}} \in \mathbb{R}^{d_{\text{obs}}} \): observable features at time \( t \)
    \item \( x_t^{\text{known}} \in \mathbb{R}^{d_{\text{known}}} \): known features at time \( t \)
    \item \( x^{\text{static}} \in \mathbb{R}^{d_{\text{static}}} \): static features (repeated across time steps)
\end{itemize}

Then each input token at time \( t \in \{1, \dots, T\} \) is:
\[
    x_t = \left[ x_t^{\text{obs}} \;\middle|\; x_t^{\text{known}} \;\middle|\; x^{\text{static}} \right]
    \in \mathbb{R}^{d_{\text{obs}} + d_{\text{known}} + d_{\text{static}}}
\]

The full input sequence is then represented as a matrix:
\[
    X = \begin{bmatrix}
            x_1 \\
            x_2 \\
            \vdots \\
            x_T
    \end{bmatrix}
    \in \mathbb{R}^{T \times (d_{\text{obs}} + d_{\text{known}} + d_{\text{static}})}
\]

The high temporal resolution of the temperature data, 288 measurements per day, results in very long input sequences
that are impractical for most deep learning models to process directly.

To manage input size and evaluate the impact of temporal resolution on predictive performance, a parameterized resampling strategy is applied.
Consecutive time steps are aggregated into bins of configurable size, and each bin is reduced to a single value using a feature-specific aggregation function.

For most continuous features, the \texttt{mean} is used.
For cyclic or categorical features—such as \textit{day of the week}—the \texttt{max} or \texttt{mode} is applied to avoid introducing artifacts at bin boundaries,
where values from distinct categories (e.g., hours 23 and 0) might otherwise be averaged into a nonexistent intermediate state.

The effect of different sampling resolutions and aggregation strategies is evaluated in Section~\ref{sec:results}.

To simulate real-time prediction, each cycle is split into overlapping,
fixed-length input windows that capture all available data up to a given time step.
As the cycle progresses, these windows slide forward, allowing the model to update its prediction based on growing historical context.

For the model types used in this study, each window produces a single output vector.
By default, this corresponds to the predicted target values at the final time step of the window,
though this can be offset to predict targets several steps into the future, depending on configuration.
While the architecture could be extended to produce output sequences (e.g., one prediction per input step), this study focuses on single-vector outputs.

This setup mimics a real-time setting, enabling the model to generate predictions dynamically as new data arrives during the cycle.
For convolutional architectures, downsampling and windowing can be disabled entirely, allowing the model to learn temporal compression directly from the raw input.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{resources/figures/methodology/methodology_padding_example}
    \caption{
        Padded input window during early cycle phases,
        where historical data is still sparse.
    }
    \label{fig:methodology_padding_example}
\end{figure}

Because fixed-length windows require a minimum amount of input data, early-cycle predictions would normally be impossible.
To address this, left-padding is applied with masked tokens until sufficient real data is available—enabled here from day four onward.
Padding values are set to 0.0 across all features.
Since the feature \emph{hours since start} is also set to 0.0 for padded steps it is reinforced, that the section is not relevant
for the prediction as no information is present.

Figure~\ref{fig:methodology_padding_example} shows an example of such padding during early-cycle input preparation.

This input strategy supports efficient, temporally-aware learning and allows me to evaluate how predictive accuracy evolves over time within each cycle.

\subsection{Model Architecture and Selection}\label{subsec:model_architecture_and_selection}

The primary objective of this study is to find models that accurately predict the features introduced in Figure~\ref{fig:methodology_target_features},
based on the historical data and context variables.
This task presents several modelling challenges: capturing temporal dependencies across varying cycle lengths, handling irregular menstrual patterns
and adapting to user-specific variability.

Ultimately, the models used in this study were selected based on their ability to:
\begin{itemize}
    \item Leverage sequential input efficiently across multiple time scales
    \item Learn temporal patterns from partially observed data
    \item Generalize across users while incorporating personalized cycle context
\end{itemize}

In this section I will introduce the base architectures \textbf{Long-Short-Term-Memory Models} and \textbf{Transformer Models}.
Additionally, I will show, how the convolutional hybrids extend their functionality.

\subsubsection{LSTM-Architecture}\label{subsubsec:lstm_architecture}
\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\textwidth]{resources/figures/methodology/methodology_lstm_architecture}
    \caption{
        Schematic overview of the LSTM architecture used in this study.
    }
    \label{fig:methodology_lstm_architecture}
\end{figure}

Recurrent neural networks, particularly LSTMs, were tested for their ability to model long-term dependencies in the time series.
Their sequential memory structure allows them to retain information across cycle days, but they may struggle with high-resolution input and longer sequences.
LSTMs, in particular, have a long history of strong performance in sequence prediction tasks.
For this study, a stack of LSTM layers was followed by a linear projection layer, mapping the hidden state at the final time
step to the two target variables: fertility probability and OV-over indicator.
Figure~\ref{fig:methodology_lstm_architecture} shows the overall architecture pipeline used for the LSTM-based model.
The stacked inputs and outputs denote the batch processing of the model.

Model-specific architectural parameters are:
\begin{itemize}
    \item \textbf{Input Length} — Number of time steps included in each input sequence.
    \item \textbf{Hidden Size} — Dimensionality of the LSTM’s internal hidden state.
    \item \textbf{Number of Layers} — Depth of the LSTM stack.
\end{itemize}
The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}.
A fixed dropout rate of \(p = 0.5\) has been selected for each LSTM-layer except the last,
as was shown to be an effective value~\cite{pham_dropout_2014}

\subsubsection{Transformer Architecture}\label{subsubsec:transformer_architecture}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\textwidth]{resources/figures/methodology/methodology_transformer_architecture}
    \caption{
        Schematic overview of the Transformer architecture used in this study.
    }
    \label{fig:methodology_transformer_architecture}
\end{figure}

Self-attention models such as the Transformer were used to capture long-range dependencies in the sequence without relying on recurrence.
These models have demonstrated state-of-the-art performance in various sequential tasks and are more robust to vanishing gradients than RNN-based alternatives.

An overview of the original Transformer design is provided in Section~\ref{subsubsec:transformer_models}.
For this study, the architecture was adapted for multivariate time-series prediction.

As the input consists of continuous features rather than discrete tokens, no embedding layer is used.
However, positional encodings are still added to allow the model to interpret the relative positions of tokens—essential for effective attention computation.
While time-dependent features like \textit{hour-of-day} or \textit{time-since-cycle-start} carry positional information implicitly,
positional encoding was retained for architectural consistency.
A future direction could investigate the impact of omitting it in such naturally ordered domains.

Since the task does not require sequence-to-sequence modeling, only the encoder part of the Transformer is used.
Its output—one vector per input token—is aggregated via 1D adaptive average pooling, resulting in a single vector representation per sequence.
This vector is then passed through a linear projection layer to produce the two target outputs:
fertility probability and OV-over indicator.
Note, that in contrast to the original use case of machine-translation, no special tokens are necessary here, as no sequence-to-sequence prediction is performed here.

Figure~\ref{fig:methodology_transformer_architecture} shows the overall architecture.
The stacked inputs and outputs indicate batch processing.

Model-specific architectural parameters are:
\begin{itemize}
    \item \textbf{Input Length} — Number of time steps included in each input sequence.
    \item \textbf{Embedding dimension} — Dimensionality of the Transformer internal token representation.
    \item \textbf{Number of Encoder-Layers} — Number of encoder layers to stack
    \item \textbf{Number of Attention-Heads} — Number of attention heads to use in each layer
\end{itemize}
For the Transformer models, a fixed dropout of \(p = 0.1\) was selected, as proposed by\cite{vaswani_attention_2017} in
the original paper.
The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}.

\subsubsection{Temporal Convolution Layer}
\label{subsubsec:temporal_convolution_layer}

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.7\textwidth]{resources/figures/methodology/methodology_convolution_architecture}
    \caption{
        Schematic overview of the temporal convolution layer architecture used in this study.
    }
    \label{fig:methodology_convolution_architecture}
\end{figure}

For both the LSTM and Transformer models, long input sequences can substantially increase model complexity and training time.
While input downsampling mitigates this, static resampling risks discarding relevant temporal patterns and reducing predictive quality.

To address this, a learnable temporal downsampling module was introduced, based on 1D convolutional layers combined with adaptive pooling.
This allows the model to reduce sequence length in a data-driven manner while preserving important features.

Specifically, a two-stage convolutional block is used to reduce the input resolution from 288 to 48 measurements per day.
Each stage consists of a 1D convolution followed by an adaptive average pooling layer.
The feature dimension remains unchanged throughout the downsampling process.
Figure~\ref{fig:methodology_convolution_architecture} shows the full pipeline of this convolutional preprocessing module.
The output is subsequently passed to the LSTM or Transformer model as described in Sections~\ref{subsubsec:lstm_architecture} and~\ref{subsubsec:transformer_architecture}.

The convolutional layer is trained end-to-end with the main model, allowing gradient-based optimization
to adaptively learn which local patterns are most informative for the downstream prediction tasks.

Further research may be necessary to identify optimal parameters for the convolutional downsampling process.
Both the number of convolutional stages and the final output resolution can be tuned to balance model capacity and temporal fidelity.
Future work could also explore alternative, potentially more interpretable downsampling strategies.
In particular, a qualitative analysis of which temporal motifs are preserved—or lost—through the convolutional layers
might offer valuable insight into the interpretability and robustness of learned representations.

\subsection{Model Training}\label{subsec:model_training}

The models described in the previous sections were trained to predict fertility probability and ovulation status based on daily temperature and contextual features.
This section outlines the training process, including the overall setup, hyperparameter optimization strategies, and implementation details.
Each model type was trained using the same preprocessing pipeline and evaluation protocol to ensure comparability across architectures.

\subsubsection{Training Setup}\label{subsubsec:training_setup}

The models were trained using a configurable framework developed specifically for this study,
allowing for flexible experimentation with different architectures, input feature sets, and
hyperparameter configurations.

The training process is organized into distinct \textit{runs}, each representing a set of model experiments
(e.g., a sweep over possible input sequence lengths), with a shared base configuration (e.g., fixed input resolution for the input sequence length sweep).
Within a run, variable parameters, such as input sequence length, hidden layer size, dropout rate,
or specific feature subsets, are swept across predefined value ranges.

I define 3 runs for each base architecture and 2 for each convolutional architecture.
\begin{itemize}
    \item \textbf{Input-Sequence-Length Run:} explores different input sequence lengths
    \item \textbf{Input Resolution Run:} explores different input resolutions (only for base architectures)
    \item \textbf{Model Parameter Run:} explores different model hyperparameters, e.g.,  hidden layer size
\end{itemize}

For each combination of parameters (e.g., 20 day input sequence length, resolution of 12 values per day),
a dedicated training and evaluation procedure is performed.
This structure supports efficient hyperparameter exploration and ensures consistent, reproducible
training conditions across models.

The framework is designed to allow plug-and-play experimentation with model types (e.g., LSTM, Transformer),
and it supports automated logging, early stopping, and checkpointing.

The trainings were performed on the scientific compute cluster of Leipzig University on NVIDIA A30 GPUs,
using the PyTorch framework~\cite{noauthor_pytorch_nodate}.
All experiments were implemented in Python and run with mixed precision for improved memory efficiency.

A detailed technical report on the training framework is planned for future work.

\subsubsection{Hyperparameter Tuning}\label{subsubsec:hyperparameter_tuning}

To identify performant configurations without exhaustively searching the entire hyperparameter space,
a structured subset of sensible parameter values was defined based on prior experience and preliminary tests.
A multi-fidelity strategy with early stopping was used to reduce training time during large-scale sweeps.

Hyperparameter tuning was divided into two stages:
(1) tuning of input-related parameters such as resampling rate, and
(2) tuning of model-specific architectural parameters such as hidden size or attention heads.

\paragraph{Input Parameter Tuning:}
This stage involved identifying optimal settings for data preprocessing and input representation.
Key variables included the resampling rate (temporal resolution) and historical context length (input window size).
These parameters strongly influence the structure of the input signal and can significantly affect model performance.

The goal was to determine whether there is a sweet spot between too little and too much temporal context,
and whether incorporating data from previous cycles improves learning or introduces noise.

\begin{table}[htbp]
    \centering
    \begin{tabular}{l>{\raggedright\arraybackslash}p{0.45\linewidth}>{\raggedright\arraybackslash}p{0.3\linewidth}}
        \toprule
        \textbf{Parameter} & \textbf{Description}                           & \textbf{Values Tested}       \\
        \midrule
        Resampling rate    & Number of temperature measurements per day     & 1, 2, 4, 12, 24, 48, 72, 288 \\
        Window length      & Historical context in days (input window size) & 10, 20, 40, 80, 160          \\
        \bottomrule
    \end{tabular}
    \caption{Input-related hyperparameters for LSTM and Transformer models.}
    \label{tab:input_hyperparameters_basic}
\end{table}

\begin{table}[ht]
    \centering
    \begin{tabular}{l>{\raggedright\arraybackslash}p{0.45\linewidth}>{\raggedright\arraybackslash}p{0.3\linewidth}}
        \toprule
        \textbf{Parameter} & \textbf{Description}                           & \textbf{Values Tested} \\
        \midrule
        Window length      & Historical context in days (input window size) & 10, 20, 40, 80, 160    \\
        \bottomrule
    \end{tabular}
    \caption{Input-related hyperparameters for convolutional LSTM and Transformer hybrids.}
    \label{tab:input_hyperparameters_conv}
\end{table}


Table~\ref{tab:input_hyperparameters_basic} shows the value ranges used for the LSTM and Transformer models,
while Table~\ref{tab:input_hyperparameters_conv} lists those for the convolutional hybrid variants.
Note that convolutional models do not require an explicit resampling parameter, as they perform learned downsampling internally
(see Section~\ref{subsubsec:temporal_convolution_layer}).

\paragraph{Model Parameter Tuning}
To identify suitable configurations for each model architecture, model-specific hyperparameters were tuned with the goal of optimizing predictive performance.
The selected value ranges were intentionally broad to explore the trade-off between model complexity and generalization.
This allowed assessment of whether increased architectural depth and capacity contribute meaningfully to performance,
or whether simpler models are sufficient for the task.
Due to resource and time limitations, not all configuration permutations can be tested.
Thus, the parameters will be tested on the input configuration with the best MSE on the fertile days for each model,
as this is the metric that represents the use cases and overall intention the best.

\begin{table}[htbp]
    \centering
    \setlength{\tabcolsep}{8pt} % adjust column spacing
    \renewcommand{\arraystretch}{1.2} % more row spacing
    \begin{tabular}{@{}p{0.28\textwidth}p{0.45\textwidth}p{0.20\textwidth}@{}}
        \toprule
        \textbf{Parameter}    & \textbf{Description}          & \textbf{Values Tested}    \\
        \midrule
        Hidden Layer Size     & Size of the LSTM hidden layer & 16, 32, 64, 128, 256, 512 \\
        Number of LSTM Layers & Number of stacked LSTM layers & 1, 2, 4                   \\
        \bottomrule
    \end{tabular}
    \caption{Model hyperparameters for the LSTM and convolutional-LSTM hybrid architectures.}
    \label{tab:lstm_model_hyperparameters}
\end{table}


\begin{table}[htbp]
    \centering
    \setlength{\tabcolsep}{8pt} % adjust column spacing
    \renewcommand{\arraystretch}{1.2} % more row spacing
    \begin{tabular}{@{}p{0.28\textwidth}p{0.45\textwidth}p{0.20\textwidth}@{}}
        \toprule
        \textbf{Parameter}        & \textbf{Description}                 & \textbf{Values Tested}    \\
        \midrule
        Embedding Dimension       & Size of the internal token embedding & 16, 32, 64, 128, 256, 512 \\
        Number of Encoder Layers  & Number of stacked encoder layers     & 1, 2, 4, 8                \\
        Number of Attention Heads & Number of attention heads per layer  & 1, 2, 4, 8                \\
        \bottomrule
    \end{tabular}
    \caption{Model-related hyperparameters for the Transformer and Convolutional-Transformer-Hybrid architectures.}
    \label{tab:transformer_model_hyperparameters}
\end{table}

Tables~\ref{tab:lstm_model_hyperparameters} and~\ref{tab:transformer_model_hyperparameters} summarize the tested hyperparameters and value ranges
for the LSTM-based and Transformer-based models, respectively.
Note that the same settings were used for the hybrid models, as their architecture beyond the convolutional front end is structurally identical.

Based on the results of the model parameter search, a best parameter set will be selected for each model architecture,
based on the MSE of the fertility-probability target during the fertile days.
These model configurations will then be used for further evaluations.

\vspace{0.5em}
I acknowledge that interactions between input and model parameters may influence final model performance,
and my two-stage tuning procedure may miss globally optimal combinations.

\subsubsection{Training Details}\label{subsubsec:training_details}

Due to the heterogeneity of both the models and training configurations,
a dynamic batch size algorithm was used to optimize resource usage by estimating the batch size during training.
The learning rate was scaled linearly with the batch size to maintain equivalent convergence behavior~\cite{goyal_accurate_2018}.
The maximum batch size was capped at 2048 to avoid out-of-memory errors during data preparation.

All training runs employed a \emph{One Cycle Scheduler}, which has proven effective for recurrent, convolutional,
and transformer-based models.
This schedule begins with a warm-up phase, followed by an annealing phase, allowing the model to converge more smoothly~\cite{smith_disciplined_2018}.

Optimization was performed using \emph{AdamW}, a decoupled weight decay variant of the \emph{Adam} optimizer,
designed to correct its flawed implementation of L2 regularization~\cite{loshchilov_decoupled_2019}.
Each model was trained for up to 30 epochs, with early stopping based on validation performance.

\subsection{Evaluation}\label{subsec:evaluation}

To meaningfully compare model performance, I define a set of metrics according to the research objectives,
that capture both overall accuracy and behavior at key points in the prediction sequence as well as cover the use cases introduced in~\ref{subsubsec:practical_use_cases}.
This includes metrics for different temporal segments, such as before and after the ovulation,
enabling a more detailed understanding of model strengths and limitations.

\subsubsection{Baseline Comparisons}\label{subsubsec:baseline_comparisons}

To place model results into context, a set of simple baseline models was established.
These models rely on minimal assumptions and serve to evaluate whether the more complex models actually learn meaningful patterns beyond basic cycle regularities.

Three types of baselines are considered:
\begin{itemize}
    \item \textbf{Population-Mean OV Day}: predicts the mean ovulation day across the full population.
    \item \textbf{User-Mean OV Day}: predicts the user’s average ovulation day; uses the population mean for the first cycle.
    \item \textbf{Previous OV Day}: predicts the ovulation day from the previous cycle; uses the population mean for the first cycle.
\end{itemize}

These baselines also help assess performance on regular menstrual patterns, something often overlooked in related work (see Section~\ref{sec:related_work}).
In many cases, it remains unclear whether proposed models genuinely outperform such simple heuristics.

\subsubsection{Evaluation Metrics}\label{subsubsec:evaluation_metrics}

The base metric used for all categories is the mean squared error (MSE),
which describes the average squared deviation of the prediction from the target value,
and is defined as:
\begin{align}
    \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2
\end{align}


where \(y_i\) is the observed value and \(\hat{y}_i\) the predicted value.

MSE penalizes larger errors more heavily, making it useful for highlighting substantial deviations.
This is particularly relevant for model comparison, where disproportionate errors can skew performance.
Moreover, since the fertility probability target was trained using an MSE-based loss function,
this metric directly reflects the optimization objective.

To complement this, I also report the mean squared error (MAE):
\begin{align}
    \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|
\end{align}
where \(y_i\) is the observed value and \(\hat{y}_i\) the predicted value.

MAE was selected for its intuitive interpretability.
In particular, the fertility probability target lends itself well to an absolute error interpretation,
making MAE a natural choice for evaluating prediction accuracy.
However, I will only be using the MAE as a secondary metric, as I will base my further interpretation of
the model performances on the use case evaluations, that provide inherent real-world interpretability.

I considered including the coefficient of determination (\(R^2\)) as an evaluation metric.
However, I found that the target windows frequently exhibited very low variance,
a condition under which \(R^2\) becomes unstable and potentially misleading.
As a result, I decided to exclude it from my evaluation.

To enable a more nuanced comparison of model performance,
I complement the overall error metrics with targeted evaluations at biologically relevant subregions of the prediction sequence.

This decomposition allows me to identify localized strengths and weaknesses in different models,
for example, a model may accurately capture the onset of fertility but underperform closer to ovulation.

It also facilitates a clearer comparison of false-positive tendencies,
i.e., days that are not fertile but are misclassified as such by the model.
These insights are important for understanding practical limitations
and choosing appropriate models depending on the intended clinical or user-facing application.

In addition, I stratify each metric by the number of past cycles available as context,
to assess how model performance varies with different amounts of historical information.
This aspect is especially relevant for users with irregular cycles,
where models may learn user-specific patterns to different degrees,
potentially leading to divergent prediction behavior.

Tables~\ref{tab:fertility_mae_metrics} and~\ref{tab:ov_over_mae_metrics} summarize the metrics used for evaluating the fertility probability and OV-over targets, respectively.

\begin{table}[htbp]
    \centering
    \setlength{\tabcolsep}{10pt} % spacing between columns
    \renewcommand{\arraystretch}{1.3} % spacing between rows
    \begin{tabular}{@{}p{0.35\linewidth}p{0.60\linewidth}@{}}
        \toprule
        \textbf{Metric Name} & \textbf{Description}          \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Mean Squared Error}} \\
        \midrule
        Fertility Overall    & MSE over the entire sequence. \\
        During-Fertility     & MSE during the fertile phase. \\
        Non-Fertility        & MSE on the non-fertile days.  \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Mean Absolute Error}} \\
        \midrule
        Fertility Overall    & MAE over the entire sequence. \\
        During-Fertility     & MAE during the fertile phase. \\
        Non-Fertility        & MAE on the non-fertile days.  \\
        \bottomrule
    \end{tabular}
    \caption{Evaluation metrics of the fertility probability target based on mean squared error (MSE) and mean absolute error (MAE) at various intervals across the predicted fertility window.}
    \label{tab:fertility_mae_metrics}
\end{table}

\begin{table}[htbp]
    \centering
    \setlength{\tabcolsep}{10pt} % spacing between columns
    \renewcommand{\arraystretch}{1.3} % spacing between rows
    \begin{tabular}{@{}p{0.35\linewidth}p{0.60\linewidth}@{}}
        \toprule
        \textbf{Metric Name} & \textbf{Description}          \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Mean Squared Error}} \\
        \midrule
        OV-Over Overall      & MSE over the entire sequence. \\
        Pre-OV               & MSE before the ovulation.     \\
        Post-OV              & MSE after the ovulation.      \\
        \midrule
        \multicolumn{2}{@{}l}{\textbf{Mean Absolute Error}} \\
        \midrule
        OV-Over Overall      & MAE over the entire sequence. \\
        Pre-OV               & MAE before the ovulation.     \\
        Post-OV              & MAE after the ovulation.      \\
        \bottomrule
    \end{tabular}
    \caption{Evaluation metrics of the OV-over target based on mean squared error (MSE) and mean absolute error (MAE) at various intervals across the predicted fertility window.}
    \label{tab:ov_over_mae_metrics}
\end{table}

\subsubsection{Best Model Configuration Selection}\label{subsubsec:methodology_best_model_config_selection}
Based on the results from the input window length, input resolution,
and model parameter space exploration, I've selected a single best configuration for each model architecture to carry forward.
This step was necessary to keep the computational effort manageable,
as exhaustively testing all possible configurations for every subsequent metric would have been prohibitively expensive.

Selection was based primarily on the \textbf{Fertility-Overall MSE} metric,
as it most directly reflects the main objective of this study: predicting fertility,
and I want to penalize larger errors more, as they are much more problematic for my use case scenarios.
In cases where the difference between configurations was small (less than \(\pm\) 5\% metric value),
I prefer the option that aligned with the general tendency of the model architecture.
For example, if an architecture consistently performs better with more input data or longer sequences,
but the Fertility-Overall MSE is only marginally better for a shorter window, I'll selected the longer window.

This selection process was applied to all tested parameters—input window length, input resolution, and model complexity.
The resulting configurations were then used in the irregular cycles study and the use case evaluation.

\paragraph{Uncertainty estimation.}
I'll report point estimates on a held-out test set.
Because per-user (or per-cycle/day) predictions were not retained, I cannot compute confidence intervals or paired significance tests.
All model comparisons are therefore descriptive rather than inferential.

\paragraph{Calibration.} While MSE (Brier score) is reported, model calibration (e.g., reliability
curves) was not assessed and is left for future work alongside prospective validation.

\subsubsection{Regular and Irregular Cycles}\label{subsubsec:methodology_regular_and_irregular_cycles}

A perfectly regular menstrual pattern does not require a sophisticated algorithm to predict ovulation—
the next ovulation day is likely to be identical or very close to the previous one.
To evaluate how different algorithms handle varying degrees of cycle variability,
I compare model performance on two distinct user groups: one with highly regular cycles and another with highly irregular cycles.

These groups are defined by the standard deviation of the ovulation day across cycles.
Specifically, I have selected select the 100 users with the lowest standard deviation (regular group)
and the 100 users with the highest standard deviation (irregular group).
This comparison allows me to assess not only overall performance,
but also whether ovulation can be meaningfully predicted in the presence of strong temporal irregularity.

For a user to be included in the analysis, they must have at least five completed cycles.
Additionally, ovulation must occur no later than cycle day 150, as later values likely indicate measurement errors or
biologically atypical cases that fall outside the scope of this study.

\subsubsection{Use Case Evaluation}\label{subsubsec:use_case_evaluation}
I further evaluate the two distinct use cases introduced in Section~\ref{subsubsec:practical_use_cases}.
For this purpose, two specialized evaluation algorithms were developed,
enabling comparability between models and providing interpretable performance metrics for each scenario.

\paragraph{Contraception Use Case:}
For evaluating contraceptive effectiveness, I have developed an algorithm inspired by the classical \emph{Pearl Index},
initially proposed by~\citeauthor{pearl_factors_1933} in~\citeyear{pearl_factors_1933}~\cite{pearl_factors_1933}.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{resources/figures/methodology/methodology_use_case_contraception_decision_diagram}
    \caption{
        Decision diagram outlining the evaluation procedure for the contraception use case.
    }
    \label{fig:methodology_use_case_contraception_decision_diagram}
\end{figure}

Figure~\ref{fig:methodology_use_case_contraception_decision_diagram} illustrates the decision logic of the evaluation algorithm.
The fertility threshold can be adjusted, as will be explored in Section~\ref{subsec:rq4_use_case_evaluations}.
A day-specific probability of intercourse is computed for each user based on age distributions reported by~\cite{twenge_declines_2017}.
I assume that the users do not have any health-related or non-health-related issues affecting fertility.
If a user's age is unknown, it is randomly drawn from the overall dataset distribution.
Only users with at least one continuous year of data are included.
To get a representative result, I will use 100 randomly selected user years.

Each day of data for a full year I've counted the following states by the algorithm:
\begin{itemize}
    \item \emph{Sex}: Intercourse occurred.
    \item \emph{Pregnancy}: fertility prediction allowed intercourse during a potentially fertile period, and it led to a pregnancy.
    \item \emph{Correct Denial}: fertility prediction correctly indicated abstinence during a fertile period.
    \item \emph{Incorrect Denial}: fertility prediction incorrectly indicated abstinence during an infertile period.
\end{itemize}

This categorization captures both the contraceptive accuracy (avoiding pregnancy) and the practicality
(minimizing unnecessary abstinence) of the predictive algorithm.
An algorithm achieving perfect contraceptive accuracy by always recommending abstinence would score highly but
significantly limit user acceptability and utility.

To add a control group, I will compute the potential pregnancies and number of sexual intercourse without using the models predictions,
i.e., the group does not use any contraception.
In this case, sexual intercourse can happen unrestricted on every day.

\paragraph{Pregnancy Use Case:}

For the pregnancy use case, I have developed a complementary algorithm to evaluate model performance for users attempting to conceive.

\begin{figure}[htbp]
    \centering
    \includegraphics[width=0.9\textwidth]{resources/figures/methodology/methodology_use_case_pregnancy_decision_diagram}
    \caption{
        Decision diagram outlining the evaluation procedure for the pregnancy use case.
    }
    \label{fig:methodology_use_case_pregnancy_decision_diagram}
\end{figure}

Figure~\ref{fig:methodology_use_case_pregnancy_decision_diagram} illustrates the decision logic of this algorithm.
The fertility threshold is adjustable and is explored further in Section~\ref{subsec:rq4_use_case_evaluations}.
Since sexual intercourse frequency differs slightly for couples trying to conceive~\cite{gaskins_predictors_2018},
I assume an average frequency of six times per month.

I assume no health-related fertility impairments for comparative simplicity,
though I acknowledge that real-world fertility is influenced by numerous complex factors.
Similar to the contraception scenario, only users with at least one continuous year of data are considered.
For representative results, I will use 100 randomly selected user years.

For each day in a full year I will count occurrences of the following states:
\begin{itemize}
    \item \emph{Sex}: Intercourse occurred
    \item \emph{Pregnancy}: Correct fertile prediction, intercourse occurred, resulting in pregnancy.
    \item \emph{Correct Deferral}: Correct non-fertile prediction.
    \item \emph{Incorrect Deferral}: Incorrect non-fertile prediction, actual fertility was above threshold.
\end{itemize}

This classification measures both fertility prediction accuracy and the impact of incorrect deferrals.
A model overly predicting fertility would increase potential pregnancy rates but negatively affect usability and trust due to misdirected efforts.

For the pregnancy case, I will also compute a control group.
Here, every day is considered potentially fertile and thus allows for intercourse.

\subsection{Ethical Considerations}\label{subsec:ethical_considerations}

The use of machine learning for fertility prediction raises important ethical
considerations due to the sensitive nature of reproductive health data and
the potential consequences of incorrect predictions.
While this thesis focuses on technical feasibility, the clinical and societal context must
also be addressed. \\

\noindent\textbf{User safety and risk of harm.} Fertility prediction errors have
asymmetric consequences: a false negative in contraception use may lead to
unintended pregnancy, while a false positive in conception support may cause frustration but less direct harm.
Any system derived from this research must therefore prioritize safety, adopt conservative thresholds for contraceptive
contexts, and clearly communicate uncertainty to users.
Importantly, predictions presented here are research results only and do not constitute medical advice. \\

\noindent\textbf{Data privacy and autonomy.} The dataset contains highly sensitive, personal health information.
All data were pseudonymized, processed under strict privacy safeguards, and used exclusively for research and product
improvement by the device manufacturer.
Even so, the possibility of re-identification in health datasets cannot be fully excluded.
Future deployments must adhere to the highest standards of data protection and give
users transparent control over how their information is used. \\

\noindent\textbf{Equity and generalizability.} The dataset reflects users of a single
device and may not represent the diversity of menstrual patterns across
populations, ages, and health conditions.
Models trained here may therefore perform differently across groups.
Explicit fairness evaluations and diverse validation cohorts are essential before any clinical or consumer application. \\

\noindent\textbf{Commercial and clinical responsibility.} The data partner in this
work is a medical device company, which underscores the need to guard against commercial bias.
Results should be independently validated, and any translation into clinical or consumer use must be preceded by prospective,
peer-reviewed trials.
Without such validation, deploying fertility prediction tools risks undermining trust and causing harm. \\