579 lines
36 KiB
TeX
579 lines
36 KiB
TeX
%! Author = alex
|
||
%! Date = 3/6/25
|
||
|
||
|
||
\section{Methodology}\label{sec:methodology}
|
||
|
||
While prior studies have demonstrated the promise of physiological signals for ovulation detection and phase classification,
|
||
many are limited by small sample sizes, rigid inclusion criteria, or non-transparent methodologies.
|
||
Temperature has emerged as a potentially predictive signal, but existing work often lacks scalability or generalizability.
|
||
This study extends previous approaches by leveraging a large, heterogeneous real-world dataset of high-resolution core body temperature
|
||
readings to develop and evaluate machine learning models for real-time ovulation prediction.
|
||
In addition to model development, special emphasis is placed on evaluating performance across irregular cycles and assessing
|
||
the predictive value of low-noise, high-resolution temperature data.
|
||
|
||
The following section outlines the methodology used, including data preprocessing,
|
||
feature extraction, input encoding, and model architectures.
|
||
|
||
% why did I select tft over other methods -> include examples of time series and why I belief a complex model could help
|
||
% you did I apply it
|
||
% implementation details
|
||
|
||
\subsection{Data Preprocessing}\label{subsec:data_preprocessing}
|
||
|
||
\subsubsection{Data Filtering}\label{subsubsec:data_filtering}
|
||
|
||
As briefly mentioned in~\ref{subsec:data_background}, not all cycles in the dataset are suitable for training.
|
||
Cycles that are either too short (\textless 10 days) or too long (\textgreater 150 days) are excluded,
|
||
as they typically indicate erroneous entries, pregnancies, sensor failures, or data processing issues.
|
||
|
||
In addition, incomplete cycles are filtered out, since new cycles continuously arrive from active users
|
||
and may not contain the full sequence of data required for retrospective labeling.
|
||
|
||
Temperature values outside the physiologically plausible range—below 35\textdegree C or above 43\textdegree C—are also excluded,
|
||
as they typically result from sensor malfunction or transmission errors.
|
||
|
||
\subsubsection{Data Labeling}\label{subsubsec:data_labeling}
|
||
|
||
Supervised machine learning requires labeled data—i.e., known target values for each input.
|
||
In the context of this study, the relevant target is the ovulation day within each menstrual cycle.
|
||
|
||
The dataset contains over 40{,}000 cycles, making manual annotation unfeasible.
|
||
Instead, a retrospective ovulation detection algorithm is employed to assign labels:
|
||
(1) whether a cycle is ovulatory or anovulatory, and
|
||
(2) if ovulatory, the estimated day of ovulation.
|
||
|
||
This labeling algorithm was developed in collaboration with a gynecologist and reproductive medicine specialist.
|
||
It is trained on a curated reference set of cycles with expert-assigned labels based on domain knowledge and characteristic temperature patterns.
|
||
|
||
The algorithm operates in two stages:
|
||
\begin{enumerate}
|
||
\item \textbf{Cycle classification:} Each cycle is classified as either monophasic (anovulatory) or biphasic (ovulatory), based on the presence of a luteal-phase temperature shift.
|
||
\item \textbf{Ovulation estimation:} For biphasic cycles, the most likely day of ovulation is identified retrospectively using the full temperature curve.
|
||
\end{enumerate}
|
||
|
||
This retrospective labeling provides a practical and scalable proxy for ground truth, enabling training and evaluation across a large, real-world dataset,
|
||
especially, as temperature is at least an excellent retrospective marker for ovulation.
|
||
In internal evaluations, the estimated ovulation day fell within a \(\pm\)2-day window of the expert reference in approximately 86\% of labeled cycles.
|
||
|
||
These labels serve as the supervisory signal for model training and evaluation.
|
||
We acknowledge the limitations of this method: ambiguous or noisy temperature patterns—due to illness, dropout,
|
||
or sensor error—can lead to mislabeled examples, which may propagate to downstream models.
|
||
However, label quality is continuously reviewed and may be refined iteratively as model performance improves.
|
||
|
||
The specific usage of ovulation labels in feature construction is described in the next section.
|
||
|
||
\subsection{Feature Engineering}\label{subsec:feature_engineering}
|
||
|
||
The features used as model inputs have been divided into three categories:
|
||
\begin{itemize}
|
||
\item \textbf{Static features} - Characteristics, that remain constant across a user's cycle, such as age, height, or average ovulation day
|
||
\item \textbf{Known features} — Inputs known a priori at each time step, such as time of day or calendar-based variables.
|
||
\item \textbf{Observable features} — Inputs available at the current time step, including raw and derived temperature values.
|
||
\item \textbf{Target features} — Outputs the model is trained to predict, such as the fertility probability.
|
||
\end{itemize}
|
||
|
||
Each feature type can handle categorical and continuous features.
|
||
This allows for mixed inputs, such as scalar measurements and class labels, within the same category.
|
||
|
||
The categorization into four feature types is intended to clarify the conceptual roles of different input types.
|
||
While the current models concatenate all features into a single input stream, the distinction allows for flexibility—future models
|
||
may process each feature group differently depending on their architectural design.
|
||
|
||
\subsubsection{Static Features}\label{subsubsec:static_features}
|
||
|
||
Static features are the features that do not change over the course of a cycle.
|
||
They might even be static for all cycles from a specific user, such as age, height and weight.
|
||
|
||
The static features are supposed to create contextual information about the cycle and the user that each model can then
|
||
use to learn patterns based not only on the temperature data, but also on this context.
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\begin{tabular}{l>{\raggedright\arraybackslash}p{0.65\linewidth}}
|
||
\toprule
|
||
\textbf{Feature} & \textbf{Description} \\
|
||
\midrule
|
||
User age & Age in years; mean imputed if missing \\
|
||
User height & Height in centimeters; mean imputed if missing \\
|
||
User weight & Weight in kilograms; mean imputed if missing \\
|
||
Average cycle length & Mean length of all previous cycles for this user \\
|
||
Cycle length SD & Standard deviation of previous cycle lengths \\
|
||
Average ovulation day & Mean day of ovulation from previous cycles \\
|
||
Ovulation SD & Standard deviation of ovulation day of previous cycles \\
|
||
Ovulatory fraction & Proportion of prior cycles classified as ovulatory \\
|
||
Cycle count & Number of previous completed cycles available \\
|
||
Avg. pre-ovulation temperature & Mean temperature in the follicular phase of previous cycles \\
|
||
Avg. post-ovulation temperature & Mean temperature in the luteal phase of previous cycles \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\caption{Static features used as model inputs}
|
||
\label{tab:static_features}
|
||
\end{table}
|
||
|
||
Table~\ref{tab:static_features} shows all static features and their descriptions.
|
||
Prior research by \citeauthor{li_menstrual_2023} has shown that menstrual cycle characteristics vary significantly with age and BMI~\cite{li_menstrual_2023}.
|
||
Including such information is therefore expected to improve predictive performance.
|
||
|
||
In addition, summary statistics from previous cycles—such as ovulation timing, temperature levels, or the fraction of ovulatory cycles—provide useful individual context.
|
||
These features help the model learn subject-specific variability and better estimate the likelihood and timing of ovulation in the current cycle.
|
||
|
||
All historical features are computed using only data available prior to the current cycle, ensuring no data leakage and supporting robust, user-adaptive learning.
|
||
|
||
%
|
||
The idea here is to provide as much context information to the models as possible to help them predict the ovulation.
|
||
|
||
\subsubsection{Known Features}\label{subsubsec:known_features}
|
||
In the context of this study, known features correspond to time-dependent inputs.
|
||
These help the model place each observation in temporal context:
|
||
|
||
\begin{itemize}
|
||
\item \textbf{Time since cycle start} — Provides the model with a relative position within the menstrual cycle.
|
||
\item \textbf{Hour of day} — Helps distinguish between daytime and nighttime patterns, especially relevant for circadian rhythms.
|
||
\item \textbf{Day of the week} — Encodes potential behavioral differences between weekdays and weekends.
|
||
\item \textbf{Month of the year} — Captures seasonal variations in temperature patterns or user behavior.
|
||
\end{itemize}
|
||
|
||
Except for \textit{time since cycle start}, all features are encoded using sine and cosine transforms to preserve their cyclical nature and make them more interpretable for the model.
|
||
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.9\textwidth]{methodology_time_feature_sine_encoded}
|
||
\caption{Sine and cosine encoding of the day-of-week feature.}
|
||
\label{fig:methodology_time_feature_encoding}
|
||
\end{figure}
|
||
|
||
Figure~\ref{fig:methodology_time_feature_encoding} illustrates the sine and cosine encoding of the day-of-week feature.
|
||
The cyclical nature of the variable is clearly visible in the transformation.
|
||
|
||
Although the model architectures used are sequential, the explicit inclusion of these time features allows the models to interpret each time step in a broader context.
|
||
More importantly, they enable the detection of gaps in the recording, which would otherwise not be visible from the data alone.
|
||
|
||
Additionally, prior research has shown that the menstrual cycle may be influenced by weekly rhythms~\cite{ecochard_menstrual_2024}.
|
||
For example, menstruation has been found to begin more frequently on Thursdays or Fridays, suggesting that behavioral or social factors may modulate certain events in the cycle.
|
||
Including this information could therefore improve the predictive quality of the models.
|
||
|
||
\subsubsection{Observable Features}\label{subsubsec:observable_features}
|
||
|
||
Observable features include all time-series inputs available up to the current time step.
|
||
They represent real-time physiological signals from which the model must infer ovulatory status.
|
||
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.9\textwidth]{methodology_observable_features}
|
||
\caption{Observable features for a cycle.}
|
||
\label{fig:methodology_observable_features}
|
||
\end{figure}
|
||
|
||
\begin{itemize}
|
||
\item \textbf{Temperature} — Raw intravaginal temperature as recorded by the OvulaRing sensor, sampled every 5 minutes.
|
||
\item \textbf{Rolling Average Temperature} — The mean temperature over a 1-day (288-sample) sliding window, linearly interpolated to preserve the original input resolution.
|
||
\item \textbf{Rolling Window Temperature Minimum} — The minimum temperature observed within a 1-day window, highlighting potential overnight lows or phase-specific dips.
|
||
\item \textbf{Rolling Window Temperature Maximum} — The maximum temperature within a 1-day window, capturing transient peaks or elevated plateaus.
|
||
\end{itemize}
|
||
|
||
These derived features are intended to reduce model complexity by providing smoothed or extremal summaries of the raw signal.
|
||
The \textit{rolling average} allows the model to capture broader trends without having to learn temporal aggregation from scratch.
|
||
The \textit{rolling minimum} and \textit{maximum} support the detection of boundary behavior (e.g., temperature shifts, sustained elevation, extreme values)
|
||
without requiring explicit memory or aggregation.
|
||
Special care was taken, so that the sliding window can only look backwards, so that no data leakage can happen.
|
||
The 1-day window length reflects the expected circadian cycle and strikes a balance between temporal sensitivity and signal stability.
|
||
|
||
Figure~\ref{fig:methodology_observable_features} illustrates the behavior of all observable features within a single cycle.
|
||
The rolling extrema delineate the amplitude of the daily temperature variation and accentuate phase transitions.
|
||
|
||
\subsubsection{Target Features}
|
||
|
||
Target features represent the outputs that the models are trained to predict.
|
||
These can include exogenous biological outcomes or interpretable derivatives of input features.
|
||
For this study, the following exogenous variables are used:
|
||
|
||
\begin{itemize}
|
||
\item \textbf{Fertility / Pregnancy Probability} — The estimated probability of conception from unprotected intercourse on the current day.
|
||
\item \textbf{Ovulation-Over Indicator} — A binary variable indicating whether ovulation has already occurred in the current cycle.
|
||
\end{itemize}
|
||
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.9\textwidth]{methodology_target_features}
|
||
\caption{
|
||
Target features plotted for a single cycle.
|
||
The ovulation-over indicator switches on the day of ovulation;
|
||
the fertility probability follows a curve based on known day-specific fecundability~\cite{dunson_day-specific_1999}.
|
||
}
|
||
\label{fig:methodology_target_features}
|
||
\end{figure}
|
||
|
||
The combination of these two targets is intended to provide the user with both physiological and practical insight:
|
||
\textit{Fertility probability} conveys the likelihood of conception, but alone does not indicate whether ovulation is yet to come or has already passed.
|
||
A fertility probability near zero could mean that ovulation is either in the past or still ahead—information the model alone cannot disambiguate.
|
||
Figure~\ref{fig:methodology_target_features} shows an example cycle where the target feature values during the course of a cycle can be seen.
|
||
|
||
The \textit{ovulation-over indicator}, by contrast, explicitly marks the post-ovulatory phase, but does not describe conception risk.
|
||
Together, the two outputs offer complementary information and improve interpretability for real-time user-facing applications.
|
||
|
||
As discussed in Section~\ref{sec:discussion}, all predictions are subject to further interpretation before presentation in the product interface.
|
||
The model outputs represent data-driven estimates and do not constitute medical advice or diagnostic statements.
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\begin{tabular}{@{}lp{0.62\linewidth}@{}}
|
||
\toprule
|
||
\textbf{Feature Name} & \textbf{Description} \\
|
||
\midrule
|
||
\multicolumn{2}{@{}l}{\textbf{Static Features}} \\
|
||
\midrule
|
||
User age & Age in years; mean imputed if missing \\
|
||
User height & Height in centimeters; mean imputed if missing \\
|
||
User weight & Weight in kilograms; mean imputed if missing \\
|
||
Average cycle length & Mean length of all previous cycles for this user \\
|
||
Cycle length STD & Standard deviation of previous cycle lengths \\
|
||
Ovulatory fraction & Proportion of prior cycles classified as ovulatory \\
|
||
Cycle count & Number of previously completed cycles available \\
|
||
Avg. pre-ovulatory temperature & Mean temperature in the follicular phase of previous cycles \\
|
||
Avg. post-ovulatory temperature & Mean temperature in the luteal phase of previous cycles \\
|
||
\midrule
|
||
\multicolumn{2}{@{}l}{\textbf{Known (Time-Based) Features}} \\
|
||
\midrule
|
||
Time since cycle start & Hours since the start of the current cycle \\
|
||
Hour of day & Hour of the day (0–23) \\
|
||
Day of the week & Day of the week (0–6) \\
|
||
Month of the year & Month of the year (0–11) \\
|
||
\midrule
|
||
\multicolumn{2}{@{}l}{\textbf{Observable Features}} \\
|
||
\midrule
|
||
Raw temperature & Temperature value recorded by the sensor \\
|
||
Rolling average temperature & 1-day rolling average (288 measurements) \\
|
||
Rolling window temp. minimum & Minimum temperature within 1-day window \\
|
||
Rolling window temp. maximum & Maximum temperature within 1-day window \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\caption{Overview of all features and their descriptions.}
|
||
\label{tab:feature_overview}
|
||
\end{table}
|
||
|
||
|
||
All features were normalized based on their empirical distributions.
|
||
A \textit{standard scaler} was applied to approximately normal features without outliers,
|
||
a \textit{robust scaler} was used for distributions with outliers, and a \textit{MinMax scaler} was used for all others.
|
||
|
||
\subsubsection{Time-Series Input Representation}
|
||
\label{subsubsec:time_series_input_representation}
|
||
|
||
Due to the high temporal resolution of the temperature data (288 measurements per day), raw input sequences can become prohibitively long for most model types.
|
||
|
||
To manage input size and evaluate the impact of temporal resolution on predictive performance, a parameterized resampling strategy is applied.
|
||
Consecutive time steps are aggregated into bins of configurable size, and each bin is reduced to a single value using a feature-specific aggregation function.
|
||
|
||
For most continuous features, the \texttt{mean} is used.
|
||
For cyclic or categorical features—such as \textit{day of the week}—the \texttt{max} or \texttt{mode} is applied to avoid introducing artifacts at bin boundaries,
|
||
where values from distinct categories (e.g., hours 23 and 0) might otherwise be averaged into a nonexistent intermediate state.
|
||
|
||
The effect of different sampling resolutions and aggregation strategies is evaluated in Section~\ref{sec:results}.
|
||
|
||
To simulate real-time prediction rather than retrospective analysis, a sliding-window approach is employed.
|
||
This allows the model to make predictions based only on data available up to a specific point in the cycle.
|
||
|
||
Each cycle is split into overlapping input windows, where each window includes data from the cycle start up to a defined time step.
|
||
The window length is fixed and configurable.
|
||
As the cycle progresses, the window slides forward, allowing the model to incorporate increasing historical context over time.
|
||
|
||
For the model types used in this study, each window produces a single output vector.
|
||
By default, this corresponds to the predicted target values at the final time step of the window, though this can be offset depending on configuration.
|
||
While the architecture could be extended to produce output sequences (e.g., one prediction per input step), this study focuses on single-vector outputs.
|
||
|
||
This setup enables temporally resolved predictions at different stages of the cycle and supports analysis of how predictive accuracy evolves with increasing context.
|
||
Depending on the configuration, downsampling and windowing can be skipped to allow the raw data to be processed by the models themselves.
|
||
This is used primarily in the convolutional flavours of the models, to allow them to learn the best way of reducing the input complexity based on the data itself.
|
||
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.9\textwidth]{methodology_padding_example}
|
||
\caption{
|
||
Padded input window during early cycle phases,
|
||
where historical data is still sparse.
|
||
}
|
||
\label{fig:methodology_padding_example}
|
||
\end{figure}
|
||
|
||
Fixed-length input windows would normally prevent early-cycle predictions when insufficient data is available.
|
||
To address this, left-padding is applied using masked values.
|
||
|
||
In this study, predictions are enabled once at least four days of data are available.
|
||
A padding value of 0.0 is used for all features, and the padding length is adjusted accordingly.
|
||
This design ensures that the model learns to ignore tokens consisting entirely of padding.
|
||
|
||
The feature \textit{hours since start}, which encodes the time elapsed since cycle onset, is also set to 0.0 for all padded tokens—
|
||
explicitly indicating that these entries contain no usable information.
|
||
|
||
Figure~\ref{fig:methodology_padding_example} shows an example of such padding during early-cycle input preparation.
|
||
|
||
\subsection{Model Architecture and Selection}\label{subsec:model_architecture_and_selection}
|
||
|
||
The primary objective of this study is to find models that accurately predict the features introduced in~\ref{fig:methodology_target_features},
|
||
based on the historical data and context variables.
|
||
This task presents several modelling challenges: capturing temporal dependencies across varying cycle lengths, handling irregular menstrual patterns
|
||
and adapting to user-specific variability.
|
||
|
||
Ultimately, the models used in this study were selected based on their ability to:
|
||
\begin{itemize}
|
||
\item Leverage sequential input efficiently across multiple time scales
|
||
\item Learn temporal patterns from partially observed data
|
||
\item Generalize across users while incorporating personalized cycle context
|
||
\end{itemize}
|
||
|
||
\subsubsection{LSTM-Architecture}\label{subsubsec:lstm_architecture}
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.7\textwidth]{methodology_lstm_architecture}
|
||
\caption{
|
||
Schematic overview of the LSTM architecture used in this study.
|
||
}
|
||
\label{fig:methodology_lstm_architecture}
|
||
\end{figure}
|
||
|
||
Recurrent neural networks, particularly GRUs and LSTMs, were tested for their ability to model long-term dependencies in the time series.
|
||
Their sequential memory structure allows them to retain information across cycle days, but they may struggle with high-resolution input and longer sequences.
|
||
LSTMs, in particular, have a long history of strong performance in sequence prediction tasks.
|
||
For this study, a stack of LSTM layers was followed by a linear projection layer, mapping the hidden state at the final time
|
||
step to the two target variables: fertility probability and ovulation-over indicator.
|
||
Figure~\ref{fig:methodology_lstm_architecture} shows the overall architecture pipeline used for the LSTM-based model.
|
||
The stacked inputs and outputs denote the batch processing of the model.
|
||
|
||
Model-specific architectural parameters are:
|
||
\begin{itemize}
|
||
\item \textbf{Input Length} — Number of time steps included in each input sequence.
|
||
\item \textbf{Hidden Size} — Dimensionality of the LSTM’s internal hidden state.
|
||
\item \textbf{Number of Layers} — Depth of the LSTM stack.
|
||
\end{itemize}
|
||
The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}.
|
||
|
||
\subsubsection{Transformer Architecture}\label{subsubsec:transformer_architecture}
|
||
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.7\textwidth]{methodology_transformer_architecture}
|
||
\caption{
|
||
Schematic overview of the Transformer architecture used in this study.
|
||
}
|
||
\label{fig:methodology_transformer_architecture}
|
||
\end{figure}
|
||
|
||
Self-attention models such as the Transformer were used to capture long-range dependencies in the sequence without relying on recurrence.
|
||
These models have demonstrated state-of-the-art performance in various sequential tasks and are more robust to vanishing gradients than RNN-based alternatives.
|
||
|
||
An overview of the original Transformer design is provided in Section~\ref{subsubsec:transformer_models}.
|
||
For this study, the architecture was adapted for multivariate time-series prediction.
|
||
|
||
As the input consists of continuous features rather than discrete tokens, no embedding layer is used.
|
||
However, positional encodings are still added to allow the model to interpret the relative positions of tokens—essential for effective attention computation.
|
||
While time-dependent features like \textit{hour-of-day} or \textit{time-since-cycle-start} carry positional information implicitly,
|
||
positional encoding was retained for architectural consistency.
|
||
A future direction could investigate the impact of omitting it in such naturally ordered domains.
|
||
|
||
Since the task does not require sequence-to-sequence modeling, only the encoder part of the Transformer is used.
|
||
Its output—one vector per input token—is aggregated via 1D adaptive average pooling, resulting in a single vector representation per sequence.
|
||
This vector is then passed through a linear projection layer to produce the two target outputs:
|
||
fertility probability and ovulation-over indicator.
|
||
|
||
Figure~\ref{fig:methodology_transformer_architecture} shows the overall architecture.
|
||
The stacked inputs and outputs indicate batch processing.
|
||
|
||
Model-specific architectural parameters are:
|
||
\begin{itemize}
|
||
\item \textbf{Input Length} — Number of time steps included in each input sequence.
|
||
\item \textbf{Embedding dimension} — Dimensionality of the Transformer internal token representation.
|
||
\item \textbf{Number of Encoder-Layers} — Number of encoder layers to stack
|
||
\item \textbf{Number of Attention-Heads} — Number of attention heads to use in each layer
|
||
\end{itemize}
|
||
The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}.
|
||
|
||
\subsubsection{Temporal Convolution Layer}
|
||
\label{subsubsec:temporal_convolution_layer}
|
||
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.7\textwidth]{methodology_convolution_architecture}
|
||
\caption{
|
||
Schematic overview of the temporal convolution layer architecture used in this study.
|
||
}
|
||
\label{fig:methodology_convolution_architecture}
|
||
\end{figure}
|
||
|
||
For both the LSTM and Transformer models, long input sequences can substantially increase model complexity and training time.
|
||
While input downsampling mitigates this, static resampling risks discarding relevant temporal patterns and reducing predictive quality.
|
||
|
||
To address this, a learnable temporal downsampling module was introduced, based on 1D convolutional layers combined with adaptive pooling.
|
||
This allows the model to reduce sequence length in a data-driven manner while preserving important features.
|
||
|
||
Specifically, a two-stage convolutional block is used to reduce the input resolution from 288 to 48 measurements per day.
|
||
Each stage consists of a 1D convolution followed by an adaptive average pooling layer.
|
||
The feature dimension remains unchanged throughout the downsampling process.
|
||
Figure~\ref{fig:methodology_convolution_architecture} shows the full pipeline of this convolutional preprocessing module.
|
||
The output is subsequently passed to the LSTM or Transformer model as described in Sections~\ref{subsubsec:lstm_architecture} and~\ref{subsubsec:transformer_architecture}.
|
||
|
||
The convolutional layer is trained end-to-end with the main model, allowing gradient-based optimization
|
||
to adaptively learn which local patterns are most informative for the downstream prediction tasks.
|
||
|
||
Further research may be necessary to identify optimal parameters for the convolutional downsampling process.
|
||
Both the number of convolutional stages and the final output resolution can be tuned to balance model capacity and temporal fidelity.
|
||
Future work could also explore alternative, potentially more interpretable downsampling strategies.
|
||
In particular, a qualitative analysis of which temporal motifs are preserved—or lost—through the convolutional layers
|
||
might offer valuable insight into the interpretability and robustness of learned representations.
|
||
|
||
\subsection{Model Training}\label{subsec:model_training}
|
||
|
||
The models described in the previous sections were trained to predict fertility probability and ovulation status based on daily temperature and contextual features.
|
||
This section outlines the training process, including the overall setup, hyperparameter optimization strategies, and implementation details.
|
||
Each model type was trained using the same preprocessing pipeline and evaluation protocol to ensure comparability across architectures.
|
||
|
||
\subsubsection{Training Setup}\label{subsubsec:training_setup}
|
||
|
||
The models were trained using a configurable framework developed specifically for this study,
|
||
allowing for flexible experimentation with different architectures, input feature sets, and
|
||
hyperparameter configurations.
|
||
|
||
The training process is organized into distinct \textit{runs}, each representing a set of model experiments with a shared base configuration.
|
||
Within a run, variable parameters—such as input sequence length, hidden layer size, dropout rate,
|
||
or specific feature subsets—are systematically swept across predefined value ranges.
|
||
|
||
For each combination of parameters, a dedicated training and evaluation procedure is performed.
|
||
This structure supports efficient hyperparameter exploration and ensures consistent, reproducible
|
||
training conditions across models.
|
||
|
||
The framework is designed to allow plug-and-play experimentation with model types (e.g., LSTM, Transformer),
|
||
and it supports automated logging, early stopping, and checkpointing.
|
||
|
||
Training was conducted on a GPU cluster equipped with NVIDIA A30 GPUs using the PyTorch framework~\cite{noauthor_pytorch_nodate}.
|
||
All experiments were implemented in Python and run with mixed precision for improved memory efficiency.
|
||
|
||
A detailed technical report on the training framework is planned for future work.
|
||
|
||
\subsubsection{Hyperparameter Tuning}\label{subsubsec:hyperparameter_tuning}
|
||
|
||
To identify performant configurations without exhaustively searching the entire hyperparameter space,
|
||
a structured subset of sensible parameter values was defined based on prior experience and preliminary tests.
|
||
A multi-fidelity strategy with early stopping was used to reduce training time during large-scale sweeps.
|
||
|
||
Hyperparameter tuning was divided into two stages:
|
||
(1) tuning of input-related parameters such as resampling rate, and
|
||
(2) tuning of model-specific architectural parameters such as hidden size or attention heads.
|
||
|
||
\paragraph{Input Parameter Tuning:}
|
||
This stage involved identifying optimal settings for data preprocessing and input representation.
|
||
Key variables included the resampling rate (temporal resolution) and historical context length (input window size).
|
||
These parameters strongly influence the structure of the input signal and can significantly affect model performance.
|
||
|
||
The goal was to determine whether there is a sweet spot between too little and too much temporal context,
|
||
and whether incorporating data from previous cycles improves learning or introduces noise.
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\begin{tabular}{l>{\raggedright\arraybackslash}p{0.45\linewidth}>{\raggedright\arraybackslash}p{0.3\linewidth}}
|
||
\toprule
|
||
\textbf{Parameter} & \textbf{Description} & \textbf{Values Tested} \\
|
||
\midrule
|
||
Resampling rate & Number of temperature measurements per day & 1, 2, 4, 12, 24, 48, 72, 288 \\
|
||
Window length & Historical context in days (input window size) & 10, 20, 40, 80, 160 \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\caption{Input-related hyperparameters for LSTM and Transformer models.}
|
||
\label{tab:input_hyperparameters_basic}
|
||
\end{table}
|
||
|
||
\begin{table}[ht]
|
||
\centering
|
||
\begin{tabular}{l>{\raggedright\arraybackslash}p{0.45\linewidth}>{\raggedright\arraybackslash}p{0.3\linewidth}}
|
||
\toprule
|
||
\textbf{Parameter} & \textbf{Description} & \textbf{Values Tested} \\
|
||
\midrule
|
||
Window length & Historical context in days (input window size) & 10, 20, 40, 80, 160 \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\caption{Input-related hyperparameters for convolutional LSTM and Transformer hybrids.}
|
||
\label{tab:input_hyperparameters_conv}
|
||
\end{table}
|
||
|
||
|
||
Table~\ref{tab:input_hyperparameters_basic} shows the value ranges used for the LSTM and Transformer models,
|
||
while Table~\ref{tab:input_hyperparameters_conv} lists those for the convolutional hybrid variants.
|
||
Note that convolutional models do not require an explicit resampling parameter, as they perform learned downsampling internally
|
||
(see Section~\ref{subsubsec:temporal_convolution_layer}).
|
||
|
||
\paragraph{Model Parameter Tuning}
|
||
To identify suitable configurations for each model architecture, model-specific hyperparameters were tuned with the goal of optimizing predictive performance.
|
||
The selected value ranges were intentionally broad to explore the trade-off between model complexity and generalization.
|
||
This allowed assessment of whether increased architectural depth and capacity contribute meaningfully to performance,
|
||
or whether simpler models are sufficient for the task.
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\setlength{\tabcolsep}{8pt} % adjust column spacing
|
||
\renewcommand{\arraystretch}{1.2} % more row spacing
|
||
\begin{tabular}{@{}p{0.28\textwidth}p{0.45\textwidth}p{0.20\textwidth}@{}}
|
||
\toprule
|
||
\textbf{Parameter} & \textbf{Description} & \textbf{Values Tested} \\
|
||
\midrule
|
||
Hidden Layer Size & Size of the LSTM hidden layer & 16, 32, 64, 128, 256, 512 \\
|
||
Number of LSTM Layers & Number of stacked LSTM layers & 1, 2, 4 \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\caption{Model hyperparameters for the LSTM and convolutional-LSTM hybrid architectures.}
|
||
\label{tab:lstm_model_hyperparameters}
|
||
\end{table}
|
||
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\setlength{\tabcolsep}{8pt} % adjust column spacing
|
||
\renewcommand{\arraystretch}{1.2} % more row spacing
|
||
\begin{tabular}{@{}p{0.28\textwidth}p{0.45\textwidth}p{0.20\textwidth}@{}}
|
||
\toprule
|
||
\textbf{Parameter} & \textbf{Description} & \textbf{Values Tested} \\
|
||
\midrule
|
||
Embedding Dimension & Size of the internal token embedding & 16, 32, 64, 128, 256, 512 \\
|
||
Number of Encoder Layers & Number of stacked encoder layers & 1, 2, 4, 8 \\
|
||
Number of Attention Heads & Number of attention heads per layer & 1, 2, 4, 8 \\
|
||
\bottomrule
|
||
\end{tabular}
|
||
\caption{Model-related hyperparameters for the Transformer and Convolutional-Transformer-Hybrid architectures.}
|
||
\label{tab:transformer_model_hyperparameters}
|
||
\end{table}
|
||
|
||
Tables~\ref{tab:lstm_model_hyperparameters} and~\ref{tab:transformer_model_hyperparameters} summarize the tested hyperparameters and value ranges
|
||
for the LSTM-based and Transformer-based models, respectively.
|
||
Note that the same settings were used for the hybrid models, as their architecture beyond the convolutional front end is structurally identical.
|
||
|
||
|
||
\vspace{0.5em}
|
||
We acknowledge that interactions between input and model parameters may influence final model performance,
|
||
and our two-stage tuning procedure may miss globally optimal combinations.
|
||
|
||
Future work may incorporate more advanced hyperparameter optimization techniques, such as
|
||
Bayesian Optimization, Genetic Algorithms, or Neural Architecture Search (NAS), to better explore the
|
||
joint parameter space in a more efficient and principled manner.
|
||
|
||
\subsubsection{Training Details}\label{subsubsec:training_details}
|
||
|
||
\subsection{Evaluation}\label{subsec:evaluation}
|
||
|
||
A variety of evaluation metrics have been defined, to best capture each aspect of the performance of an ovulation prediction.
|
||
The evaluation is divided into two subgroups for the two prediction targets, fertility and past-ovulation indicator.
|
||
|
||
\subsubsection{Evaluation Metrics}\label{subsubsec:evaluation_metrics}
|
||
|
||
\subsubsection{Baseline Comparisons}\label{subsubsec:baseline_comparisons}
|
||
|
||
\subsection{Ethical Considerations}\label{subsec:ethical_considerations}
|
||
|
||
This study was conducted using pseudonymized data collected in accordance with the terms of service and privacy policy of the data provider, VivoSensMedical GmbH (Leipzig, Germany).
|
||
All users whose data were included had consented to the use of their recordings for analytical purposes at the time of data collection.
|
||
|
||
The study protocol was reviewed and approved by the provider's internal legal and scientific advisory team,
|
||
which is responsible for ensuring ethical and regulatory compliance.
|
||
|
||
All data used in this study were pseudonymized prior to access.
|
||
No personal identifiers or sensitive metadata were included.
|
||
Additional safeguards were implemented to ensure data confidentiality and integrity,
|
||
including restricted access and use solely for the purposes of model development and evaluation.
|
||
No compensation was provided to participants, as the data were originally collected as part of routine usage under the agreed terms. |