alex/temperature-based-fertility-prediction-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a2fdf2f0a3426aa87e6737dba9f04b9378967218
temperature-based-fertility…/thesis/sections/results.tex
T
Alex Blank a2fdf2f0a3 further work on discussion
2025-09-04 17:30:43 +00:00

895 lines
49 KiB
TeX
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
%! Author = alex
%! Date = 3/6/25
\section{Results}\label{sec:results}
% find introduction
\subsection{RQ1: Comparative Study of Model Architectures}\label{subsec:rq1_temp_predictive_value}
We systematically evaluated multiple model architectures to assess their performance in
predicting the targets.
We report results for two targets: (i) fertility probability and (ii) an indicator that ovulation has already occurred (OV-Over).
Each architecture was tested across a range of input window lengths, temporal resolutions, and model capacities.
The best-performing configurations for each architecture were selected for downstream analyses.
Unless stated otherwise, \textbf{MSE} is the primary metric (equivalent to the Brier score for probabilistic targets),
and \textbf{MAE} is secondary.
All metrics reported in this section are point estimates without confidence intervals; therefore, comparisons are descriptive.
\subsubsection{Fertility-Probability Prediction}\label{subsec:fertility_probability_prediction}
We analyze model performance in predicting fertility probability,
focusing on the influence of temporal context (window length), measurement resolution, and model capacity.
\paragraph{Impact of input window length.}
Across all architectures, predictive performance is sensitive to the temporal context length, but the optimal window varies by model.
We consider 10 days short, 20--40 medium and 80--160 long input windows.
The \textbf{LSTM} performs best with long context: at 160 days, it reaches the lowest fertile-day MSE (0.0102) and a competitive overall error(0.0043).
The lowest non-fertile-day MSE (0.0013) is tied between LSTM and Convolutional LSTM, both at 20 days.
The \textbf{Transformer} shows more varied behavior: its overall error is minimized at 40 days (0.0044),
fertile-day MSE at 80 days (0.0108), and non-fertile-day MSE at 160 days (0.0016).
For the \textbf{Convolutional LSTM}, the best results occur at 40 days overall (MSE 0.0045), with non-fertile-day MSE again best at 20 days.
The \textbf{Convolutional Transformer} achieves the lowest overall MSE (0.0041) at 40 days.
A summary of best-performing window lengths per architecture is shown in Table~\ref{tab:fertility_windows_compact_mse};
full results including MAE are provided in Appendix Table~\ref{tab:fertility_results_by_window_length}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(days / MSE)}
& \shortstack[c]{Fertile-day best\\(days / MSE)}
& \shortstack[c]{Non-fertile-day best\\(days / MSE)} \\
\midrule
LSTM
& 160 / 0.0043
& \textbf{160 / 0.0102}
& \textbf{20 / 0.0013} \\
Transformer
& 40 / 0.0044
& 80 / 0.0108
& 160 / 0.0016 \\
Convolutional LSTM
& 40 / 0.0045
& 40 / 0.0122
& \textbf{20 / 0.0013} \\
Convolutional Transformer
& \textbf{40 / 0.0041}
& \textbf{20 / 0.0102}
& 10, 40 / 0.0015 \\
\bottomrule
\end{tabularx}
\caption{Best-performing parameter settings per architecture for fertility probability prediction,
highlighting model-specific trade-offs between capacity and error minimization.
Bold entries are bests within a column; ties are shown with commas.
Full grid (incl.\ MAE) in Appendix, Table~\ref{tab:fertility_results_by_window_length}.}
\label{tab:fertility_windows_compact_mse}
\end{table}
\paragraph{Impact of input resolution.}
To assess the impact of temporal resolution, we fix the input window at 20 days and evaluate LSTM and Transformer models across a range of input rates.
We exclude convolutional models from this comparison, as they always receive the full-resolution input (288 values/day) and learn temporal compression internally.
We consider 1--8 values per day low resolution, 12--48 medium resolution and 72--288 high resolution.
For the \textbf{LSTM}, the lowest overall MSE (0.0046) occurs at 4 values/day,
while fertile-day MSE is minimized at full resolution (288/day, MSE 0.0110).
The best non-fertile-day MSE (0.0013) is achieved at 12/day.
The \textbf{Transformer} performs best overall at both 48 and 288/day (MSE 0.0048),
with the best fertile-day MSE at 4/day (0.0115) and best non-fertile-day MSE at 72/day (0.0014).
Overall, low to medium input resolutions (4–48/day) often minimize overall error,
though higher resolutions can be advantageous for detecting short-term fertile-day signals.
Results are summarized in Table~\ref{tab:fertility_resolution_compact_mse}; full metrics are in Appendix Table~\ref{tab:fertility_results_by_window_resolution}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(values/day / MSE)}
& \shortstack[c]{Fertile-day best\\(values/day / MSE)}
& \shortstack[c]{Non-fertile-day best\\(values/day / MSE)} \\
\midrule
LSTM
& \textbf{4 / 0.0046}
& \textbf{288 / 0.0110}
& \textbf{12 / 0.0013} \\
Transformer
& 48, 288 / 0.0048
& 4 / 0.0115
& 72 / 0.0014 \\
\bottomrule
\end{tabularx}
\caption{Fertility probability: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
Bold entries are bests within a column; ties are shown with commas. Convolutional models are excluded (fixed at 288/day).}
\label{tab:fertility_resolution_compact_mse}
\end{table}
\paragraph{Impact of model parameters.}
We next evaluate how architectural complexity, i.e., hidden size, number of layers,
and number of attention heads, influences performance under fixed input settings (160 days at 12/day for LSTM/Transformer, 40 days for convolutional models).
Parameters are reported as (hidden size × number of layers) for LSTM models and (embedding size × number of encoders × attention heads) for Transformers.
For the \textbf{LSTM}, the best overall and fertile-day MSEs are both observed at a configuration with 512 hidden units and 4 layers.
The lowest non-fertile-day MSE (0.0011) is reached at 256×4.
The \textbf{Transformer} performs best overall and on fertile days with 512-dimensional embeddings, 4 encoder layers,
and 4 attention heads (0.0043 and 0.0103, respectively).
Its lowest non-fertile-day MSE (0.0011) occurs at a larger configuration with 8 layers and 8 heads.
For the \textbf{Convolutional LSTM}, the best overall and fertile-day MSEs are reached with 256 hidden units and 4 layers,
while several lower-capacity settings tie for the best non-fertile-day MSE (0.0016).
The \textbf{Convolutional Transformer} performs best overall with either 256×4×4 or 512×4×4, and reaches its lowest non-fertile-day MSE at 512×4×4.
Detailed comparisons are shown in Table~\ref{tab:fertility_params_compact_mse},
with full results in Appendix Tables~\ref{tab:fertility_results_by_model_parameters_lstm}–\ref{tab:fertility_results_by_model_parameters_conv_transformer}.
\begin{table}[t]
\scriptsize
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(params / MSE)}
& \shortstack[c]{Fertile-day best\\(params / MSE)}
& \shortstack[c]{Non-fertile-day best\\(params / MSE)} \\
\midrule
LSTM
& \(512\times4\) / 0.0047
& \(512\times4\) / 0.0122
& \textbf{\(256\times4\) / 0.0011} \\
Transformer
& \textbf{\(512\times4\times4\) / 0.0043}
& \(512\times4\times4\) / 0.0103
& \textbf{\(512\times8\times8\) / 0.0011} \\
Convolutional LSTM
& \(256\times4\) / 0.0042
& \textbf{\(256\times4\) / 0.0100}
& \(16\times1,\,32\times2,\,128\times2,\,128\times4\) / 0.0016 \\
Convolutional Transformer
& \textbf{\(256\times4\times4,\,512\times4\times4\) / 0.0043}
& \(512\times8\times8\) / 0.0107
& \(512\times4\times4\) / 0.0013 \\
\bottomrule
\end{tabularx}
\caption{Fertility probability: best parameter settings per architecture (MSE only).
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
Bold entries are bests within a column; ties are shown with commas.
Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
\label{tab:fertility_params_compact_mse}
\end{table}
\subsubsection{Ovulation-Over Prediction}\label{subsubsec:ov_over_prediction}
We evaluate prediction of a binary indicator denoting whether ovulation has already occurred.
We examine the influence of input window length, input resolution, and model capacity.
\paragraph{Impact of input window length.}
For predicting whether ovulation has occurred (OV-over),
optimal window lengths again vary by architecture and target phase (pre- vs post-ovulation).
The \textbf{Transformer} yields the lowest overall (0.0533) and post-ovulation MSE (0.0520) at 40 days.
The \textbf{LSTM} performs best before ovulation (MSE 0.0212 at 20 days), while its overall MSE improves with longer context (160 days).
The \textbf{Convolutional LSTM} favors short windows, with the best overall MSE (0.0699), before-OV (0.0389),
and after-OV (0.0833) all occurring at 20 days.
The \textbf{Convolutional Transformer} performs best overall at 40 days and best before ovulation at 160 days (MSE 0.0286).
See Table~\ref{tab:ovover_windows_compact_mse} for a summary.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(days / MSE)}
& \shortstack[c]{Before-OV best\\(days / MSE)}
& \shortstack[c]{After-OV best\\(days / MSE)} \\
\midrule
LSTM
& 160 / 0.0557
& \textbf{20 / 0.0212}
& 160 / 0.0580 \\
Transformer
& \textbf{40 / 0.0533}
& 20 / 0.0255
& \textbf{40 / 0.0520} \\
Convolutional LSTM
& 20 / 0.0699
& 20 / 0.0389
& 20 / 0.0833 \\
Convolutional Transformer
& 40 / 0.0709
& 160 / 0.0286
& 40 / 0.0820 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best input window per architecture (MSE only) at a fixed input resolution of 12 values/day.
Bold entries are bests within a column.}
\label{tab:ovover_windows_compact_mse}
\end{table}
\paragraph{Impact of input resolution.}
Fixing the input window to 20 days, we compare input resolutions for LSTM and Transformer models.
For the \textbf{LSTM}, the lowest overall MSE (0.0633) occurs at 24/day,
with best before-OV and after-OV MSEs at 12/day (0.0212) and 48/day (0.0550), respectively.
The \textbf{Transformer} performs best overall at 72/day (0.0585),
with lowest before-OV MSE at 12/day (0.0255) and after-OV MSE at 288/day (0.0578).
Results are summarized in Table~\ref{tab:ovover_resolution_compact_mse};
full resolution grids are in Appendix Table~\ref{tab:ov_over_results_by_resolution}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(values/day / MSE)}
& \shortstack[c]{Before-OV best\\(values/day / MSE)}
& \shortstack[c]{After-OV best\\(values/day / MSE)} \\
\midrule
LSTM
& 24 / 0.0633
& \textbf{12 / 0.0212}
& \textbf{48 / 0.0550} \\
Transformer
& \textbf{72 / 0.0585}
& 12 / 0.0255
& 288 / 0.0578 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
Bold entries are bests within a column. Convolutional models are excluded (they consume 288 values/day internally).}
\label{tab:ovover_resolution_compact_mse}
\end{table}
\paragraph{Impact of model parameters.}
Using fixed input settings (160 days at 12/day for LSTM/Transformer; 40 days for convolutional models), we evaluate model scaling.
The \textbf{Transformer} achieves the lowest overall (0.0543) and after-ovulation MSE (0.0410) at a large configuration (512×8×8).
Before-OV MSE is lowest at 256×4×4 (0.0293).
For the \textbf{LSTM}, the best overall and after-OV performance is at 512×4,
while the best before-OV MSE occurs at 128×2 (0.0274).
The \textbf{Convolutional LSTM} performs best overall and after-OV at 256×4, with the best before-OV MSE at 128×2.
Finally, the \textbf{Convolutional Transformer} achieves its lowest overall and after-OV MSE at 512×4×4, and best before-OV MSE at 512×8×8.
Table~\ref{tab:ovover_params_compact_mse} summarizes these parameter-dependent results;
full comparisons are included in Appendix Tables~\ref{tab:ov_over_results_by_model_parameters_lstm}–\ref{tab:ov_over_results_by_model_parameters_conv_transformer}.
\begin{table}[t]
\scriptsize
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(params / MSE)}
& \shortstack[c]{Before-OV best\\(params / MSE)}
& \shortstack[c]{After-OV best\\(params / MSE)} \\
\midrule
LSTM
& \(512\times4\) / 0.0616
& \textbf{\(128\times2\) / 0.0274}
& \(512\times4\) / 0.0613 \\
Transformer
& \textbf{\(512\times8\times8\) / 0.0543}
& \(256\times4\times4\) / 0.0293
& \textbf{\(512\times8\times8\) / 0.0410} \\
Convolutional LSTM
& \(256\times4\) / 0.0687
& \(128\times2\) / 0.0357
& \(256\times4\) / 0.0715 \\
Convolutional Transformer
& \(512\times4\times4\) / 0.0703
& \(512\times8\times8\) / 0.0339
& \(512\times4\times4\) / 0.0814 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best parameter settings per architecture (MSE only).
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
Bold values indicate bests within a column. Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
\label{tab:ovover_params_compact_mse}
\end{table}
\subsubsection{Best Model Configuration Selection}\label{subsubsec:results_best_model_config_selection}
To identify the best configuration for each model architecture, we followed the procedure described in Section~\ref{subsubsec:methodology_best_model_config_selection}.
Configurations were selected based on the overall fertility MSE on the validation set,
while also considering the general performance trends of each model.
Tables~\ref{tab:best_configs_lstm} and~\ref{tab:best_configs_transformer} summarize the selected hyperparameters for each architecture,
including input window length, temporal resolution, and model complexity.
For convolutional models, the input resolution was always fixed at the full 288 measurements per day.
For the Transformer architecture, we made the decision to use longer input windows, as the performance on longer
input windows was competitive to the best measured at 40 days, but includes much more context information.
\begin{table}[htbp]
\centering
\scriptsize
\begin{tabularx}{\linewidth}{lXXXX}
\toprule
\textbf{Architecture} &
\textbf{Input Window Length} &
\textbf{Input Resolution} &
\textbf{Hidden Layer Size} &
\textbf{\# LSTM Layers} \\
\midrule
LSTM & 160 & 12 & 512 & 4 \\
Convolutional LSTM & 40 & 288 & 512 & 4 \\
\bottomrule
\end{tabularx}
\caption{Selected configurations for LSTM-based architectures. Input window length is given in days; input resolution in samples per day.}
\label{tab:best_configs_lstm}
\end{table}
\begin{table}[htbp]
\centering
\scriptsize
\begin{tabularx}{\linewidth}{lXXXXX}
\toprule
\textbf{Architecture} &
\textbf{Input Window Length} &
\textbf{Input Resolution} &
\textbf{Embedding Size} &
\textbf{\# Encoder Layers} &
\textbf{\# Attention Heads} \\
\midrule
Transformer & 160 & 12 & 512 & 4 & 4 \\
Convolutional Transformer & 40 & 288 & 512 & 4 & 4 \\
\bottomrule
\end{tabularx}
\caption{Selected configurations for Transformer-based architectures. Input window length is given in days; input resolution in samples per day.}
\label{tab:best_configs_transformer}
\end{table}
Final evaluation of the selected configurations was performed on a held-out test set with no reuse of training or validation data.
Table~\ref{tab:results_model_selection_metrics} reports the MSE values for both prediction targets:
the fertility probability and the OV-over indicator.
All trained architectures outperform the baseline models across all evaluation metrics.
\begin{table}
\scriptsize
\begin{tabularx}{\linewidth}{l*{6}{X}}
\toprule
\multirow{1}{*}{Model} & \multicolumn{6}{c}{MSE} \\
\cmidrule(r){2-7}
& Fertility Overall & Fertile Days & Non-Fertile Days & OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
\midrule
Transformer & 0.0037 & \textbf{0.0089} & 0.0017 & 0.0508 & 0.0236 & 0.0581 \\
LSTM & \textbf{0.0036} & 0.0104 & \textbf{0.0008} & \textbf{0.0437} & \textbf{0.0233} & \textbf{0.0492} \\
Convolutional LSTM & 0.0037 & 0.0092 & 0.0014 & 0.0499 & 0.0281 & 0.0563 \\
Convolutional Transformer & 0.0038 & 0.0098 & 0.0014 & 0.0517 & 0.0305 & 0.0566 \\
User-Based-Mean Baseline & 0.0064 & 0.0154 & 0.0028 & 0.1072 & 0.0872 & 0.0983 \\
Last-Cycle Baseline & 0.0080 & 0.0202 & 0.0031 & 0.1453 & 0.1099 & 0.1762 \\
Population-Mean Baseline & 0.0127 & 0.0258 & 0.0074 & 0.2145 & 0.0400 & 0.3749 \\
\bottomrule
\end{tabularx}
\caption{Evaluation metrics (MSE) for fertility probability and OV-over targets using the selected model configurations.
Bold values indicate the best-performing architecture for each metric.}
\label{tab:results_model_selection_metrics}
\end{table}
The LSTM model achieves the best overall performance, with the lowest MSE for overall fertility (0.0036),
non-fertile days (0.0008), and all post-ovulatory metrics.
The Transformer model, however, yields the lowest MSE on fertile days (0.0089), slightly outperforming the LSTM in that subset.
Compared to the strongest baseline (User-Mean), the LSTM reduces the overall fertility MSE by 44\% and the post-ovulatory MSE by 59\%.
The performance gap is even larger when compared to the population-mean and last-cycle baselines,
confirming the advantage of personalized and temporally-aware modeling.
% -------------------------------------------------------------------------------------------------------------------------------
\subsection{RQ2: Factors and Patterns that Influence Predictions}\label{subsec:rq2_factors_and_patterns}
\subsubsection{Representative predictions}\label{subsubsec:rq2_examples}
All following prediction were made with the best model for each architecture as selected in the previous section.
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{resources/figures/results/regular_cycle_pattern_fertility_prediction_example}
\caption{
Temperature rolling average and fertility-probability prediction for a user with a regular cycle pattern. (Values are scaled features)
}
\label{fig:results_rq2_regular_cycle_predictions_example}
\end{figure}
To visualize potential patterns in the predictions, we will show some representative prediction plots.
All models have a similar predictive behaviour throughout both the regular and irregular cycles.
Figure~\ref{fig:results_rq2_regular_cycle_predictions_example} shows, that all models come very close to the ground truth
for a regular cycle pattern.
Slight deviations from the regular cycle length, such as in cycle 5 (Measurement 150--220) show, that the models
tend to overestimate the fertility-probability in such cases.
Additionally, all models never reach the full range of fertility-probability as indicated by the ground-truth.
These predictions show a clear correlation with a temperature drop before the ovulation.
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{resources/figures/results/temperature_drop_pattern}
\caption{
Temperature rolling average and fertility-probability prediction showing a correlation between fertility and
the pre-ovulator temperature drop.
}
\label{fig:results_temperature_drop_pattern}
\end{figure}
This correlation between a pre-ovulator temperature drop and the fertility can be seen more pronounced in Figure~\ref{fig:results_temperature_drop_pattern}.
The ground-truth fertility seems to be centered around this temperature drop and all models seem to be able
to pick it up, if it is clearly distinguishable and visible in the data.
For regular cycles, this temperature drop seems to be much easier distinguishable,
as there is not as much temperature variability throughout the first cycle phase.
Additionally, there also seems to be a correlation between the height of the temperature drop and the corresponding
fertility-probability prediction by the models.
The second cycle (Measurement 35--90) has a smaller temperature drop and all models predict a smaller fertility-probability
as for the following two cycles (Measurements 90--145).
This association of temperature-drop height and fertility-probability can also be seen in Figure~\ref{fig:results_rq2_regular_cycle_predictions_example}.
The first two cycles have a smaller temperature drop compared to the following cycles and also a less pronounced prediction.
Figure~\ref{fig:results_rq2_irregular_cycle_predictions_example} shows an irregular cycle pattern
and the fertility-probability predictions for it.
For irregular cycles, all models show much more struggle in determining the ground-truth fertility.
The pre-ovulator temperature drop is far less distinguishable.
The fifth cycle (measurement 450--700) shows a much more gradual temperature drop and all model's predictions
are far off target.
The previous 2 cycles (cycle 3 and 4) are shorter and have a much more distinguishable temperature drop
and thus all models show indication for a detection of the fertile phase.
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{resources/figures/results/irregular_cycle_pattern_fertility_prediction_example}
\caption{
Temperature rolling average and fertility-probability prediction for a user with an irregular cycle pattern. (Values are scaled features)
}
\label{fig:results_rq2_irregular_cycle_predictions_example}
\end{figure}
All models seem to be easily confused by anomalies during this characteristic temperature drop, even for short and regular cycles.
Figure~\ref{fig:results_rq2_anomaly_in_temperature_drop} shows a cycle with a clear anomaly during the pre-ovulatory drop in cycle 3 (Measurements 90--125).
This confuses all models into ending the fertile phase earlier, even though fertility is likely still elevated.
\begin{figure}[htbp]
\centering
\includegraphics[width=1.0\textwidth]{resources/figures/results/spike_in_temperature_drop}
\caption{
Temperature rolling average and fertility-probability prediction for a user with an anomaly during the characteristic temperature drop. (Values are scaled features)
}
\label{fig:results_rq2_anomaly_in_temperature_drop}
\end{figure}
\subsubsection{Effect of user history depth}\label{subsubsec:rq2_history_depth}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/performance_on_different_historical_contexts}
\caption{
Plots showing the MSE for the fertility probability and OV-over targets over a growing size of past cycles available
for each model architecture, as well as a trend line for each metric.
}
\label{fig:results_performance_on_different_historical_context}
\end{figure}
Figure~\ref{fig:results_performance_on_different_historical_context} shows the MSE for the fertility probability
and OV-over targets as a function of the number of past cycles available per user, for all model architectures and baselines.
All four models improve slightly on both metrics as the amount of historical data increases.
The LSTM-based models show a larger relative improvement with longer user history than the Transformer-based variants.
All baselines seem produce worse prediction with larger historical contexts.
The last-cycle and population-mean baselines show the largest decrease in performance,
especially for the fertility-probability prediction target.
The user-mean baselines offers largely unchanged fertility prediction performance,
but worsens on the ov-over prediction with more historical cycles available.
Across all models and baselines, the variability of results increases with greater history depth:
results are more tightly clustered around the trend line with short histories,
but show greater scatter for users with longer data records.
% -------------------------------------------------------------
\subsection{RQ3: Performance across Regular and Irregular Cycles}\label{subsec:rq3_regular_vs_irregular}
\begin{table}[htbp]
\centering
\small
\begin{tabularx}{\linewidth}{l*{3}{X}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{3}{c}{MSE} \\
\cmidrule(r){2-4}
& Fertility Overall & Fertile Days & Non-Fertile Days \\
\midrule
\multicolumn{4}{c}{\textbf{Regular Cycle Group}} \\
\midrule
LSTM & 0.002563 & 0.007850 & \textbf{0.000415} \\
Transformer & \textbf{0.002376} & \textbf{0.006651} & 0.000626 \\
Convolutional LSTM & 0.002460 & 0.007313 & 0.000573 \\
Convolutional Transformer & 0.002424 & 0.006919 & 0.000655 \\
Last-Cycle Baseline & 0.006592 & 0.014177 & 0.003198 \\
Population Mean Baseline & 0.016657 & 0.027671 & 0.011292 \\
User Mean Baseline & 0.005195 & 0.011715 & 0.002312 \\
\midrule
\multicolumn{4}{c}{\textbf{Irregular Cycle Group}} \\
\midrule
LSTM & 0.004490 & 0.013415 & \textbf{0.001076} \\
Transformer & 0.004122 & \textbf{0.010774} & 0.001783 \\
Convolutional LSTM & \textbf{0.004096} & 0.011188 & 0.001538 \\
Convolutional Transformer & 0.004281 & 0.011374 & 0.001639 \\
Last-Cycle Baseline & 0.009423 & 0.023650 & 0.004613 \\
Population Mean Baseline & 0.009795 & 0.022330 & 0.005751 \\
User Mean Baseline & 0.010548 & 0.023935 & 0.006159 \\
\bottomrule
\end{tabularx}
\caption{MSEs for fertility probability predictions across different model architectures for regular and irregular cycle groups. Bold values denote the best scores per column.}
\label{tab:fertility_mse_regular_irregular}
\end{table}
\noindent
Table~\ref{tab:fertility_mse_regular_irregular} shows the MSE metric results for all models on the regular and irregular
cycle groups for the fertility-probability target.
For this target, all learned models substantially outperform the baseline methods in both regular and irregular cycle groups.
In the regular group, the Transformer achieves the lowest overall MSE and MSE on fertile days, while the LSTM excels on non-fertile days.
Convolutional variants also perform competitively.
Among baselines, the user-specific mean performs better than both the last-cycle and population mean, but still falls short of all learned models.
In the irregular group, model performance is uniformly lower (i.e., MSEs are higher).
The Convolutional LSTM shows the lowest overall MSE, while the Transformer performs best on fertile days.
The LSTM remains the most effective on non-fertile days.
Baselines degrade more sharply in this group.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_fertility_results}.
\vspace{1em}
\begin{table}[htbp]
\centering
\small
\begin{tabularx}{\linewidth}{l*{3}{X}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{3}{c}{MSE} \\
\cmidrule(r){2-4}
& OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
\midrule
\multicolumn{4}{c}{\textbf{Regular Cycle Group}} \\
\midrule
LSTM & \textbf{0.028519} & 0.031695 & \textbf{0.026116} \\
Transformer & 0.032929 & \textbf{0.024774} & 0.033411 \\
Convolutional LSTM & 0.034224 & 0.028962 & 0.033929 \\
Convolutional Transformer & 0.033333 & 0.039045 & 0.029329 \\
Last-Cycle Baseline & 0.087916 & 0.093465 & 0.074191 \\
Population Mean Baseline & 0.285887 & 0.008876 & 0.418879 \\
User Mean Baseline & 0.069245 & 0.069152 & 0.055724 \\
\midrule
\multicolumn{4}{c}{\textbf{Irregular Cycle Group}} \\
\midrule
LSTM & \textbf{0.051747} & \textbf{0.016777} & 0.070419 \\
Transformer & 0.055990 & 0.026159 & 0.067520 \\
Convolutional LSTM & 0.054523 & 0.030092 & \textbf{0.066643} \\
Convolutional Transformer & 0.056075 & 0.029638 & 0.067965 \\
Last-Cycle Baseline & 0.222906 & 0.134945 & 0.258047 \\
Population Mean Baseline & 0.171368 & 0.130366 & 0.140363 \\
User Mean Baseline & 0.180702 & 0.088884 & 0.217144 \\
\bottomrule
\end{tabularx}
\caption{MSEs for ovulation-over target across different model architectures for regular and irregular cycle groups. Bold values denote the best scores per column.}
\label{tab:ov_over_mse_regular_irregular}
\end{table}
\noindent
Table~\ref{tab:ov_over_mse_regular_irregular} shows the MSE metric results for all models on the regular and irregular cycle groups for the OV-over target.
For this target, performance patterns differ more distinctly across phases.
In the regular group, the LSTM performs best overall and after ovulation, while the Transformer performs best before ovulation.
Convolutional models perform slightly worse than their recurrent and transformer-based counterparts across all phases.
The population mean baseline shows a spurious low MSE before ovulation, but this is not consistent across phases, suggesting it is not reliable.
In general, baselines are notably weaker than learned models across all splits.
For irregular cycles, all model performances deteriorate relative to regular cycles.
The LSTM remains the most robust, achieving the best MSEs both overall and before ovulation.
Convolutional LSTM performs best after ovulation.
Transformer-based models perform less consistently in this group, suggesting decreased robustness to cycle irregularity.
As before, all learned models outperform the baselines by a wide margin.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_ov_over_results}.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_regular_cycles}
\caption{
MSE performance trends across increasing historical data availability for fertility and ovulation-over targets on regular cycles.
}
\label{fig:results_performance_on_regular_cycles}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_irregular_cycles}
\caption{
MSE performance trends across increasing historical data availability for fertility and ovulation-over targets on irregular cycles.
}
\label{fig:results_performance_on_irregular_cycles}
\end{figure}
\paragraph{Evaluation on Historical Context Depth.}
Figures~\ref{fig:results_performance_on_regular_cycles} and~\ref{fig:results_performance_on_irregular_cycles}
visualize performance improvements with increasing numbers of past cycles.
For regular cycles, all models except the population-mean baseline benefit from more historical data, showing consistent MSE reductions for both targets.
Interestingly, the population-mean baseline performs worse as more cycles are added.
The largest improvement can be seen for the last-cycle baseline and user-mean baseline models.
The outliers in the last-cycle and user-mean baseline are MSE values of 0.0, which result in these vertical lines.
In contrast, irregular cycles exhibit more variability and less performance gain with more cycles available.
All trained models improve on both targets with more context.
The population-mean and user-mean baselines show improvement on the OV-over target.
In contrast, their MSE increases with more historical context for the fertility-probability target.
The last-cycle baseline shows marginal performance improvement for the OV-over target and the same increase for the fertility-probability
target as the other two baselines.
The selected irregular cycle group has cycles with longer context, thus the irregular cycle history reaches up to 80 cycles,
while regular cycle history only reaches up to 35.
All models show increased output variability as the number of available past cycles grows,
as indicated by the wider deviations from the trend lines.
% -------------------------------------------------------------
\subsection{RQ4: Use-Case Evaluations}\label{subsec:rq4_use_case_evaluations}
We evaluated the use-case scenarios described in Section~\ref{subsubsec:practical_use_cases} using the algorithms
in Section~\ref{subsubsec:use_case_evaluation} across multiple fertility thresholds.
Each evaluation used the same test set of 100 users (100 user-years) and was repeated for 200 iterations
to get statistically more meaningful results.
All results represent the means over all runs.
\subsubsection{Contraception Use-Case Results}\label{subsubsec:use_case_contraception_results}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/contraception_use_case_results_by_fertility_threshold}
\caption{
Contraception use-case evaluation study results by fertility threshold.
}
\label{fig:results_contraception_use_case_results_by_fertility_threshold}
\end{figure}
Figure~\ref{fig:results_contraception_use_case_results_by_fertility_threshold} shows the outcomes of
the contraception use-case across fertility thresholds for each model architecture, along with the baselines.
The four evaluated metrics are: number of pregnancies, number of intercourse events,
number of correct denials (i.e., appropriately flagged fertile days),
and number of incorrect denials (i.e., fertile days incorrectly flagged as infertile).
\paragraph{Pregnancies.}
All trained models start with very low pregnancy rates (Transformer: 0, LSTM: 4)
and then increase roughly linearly with the threshold,
reaching between 99 (Convolutional Transformer) and 122 (Transformer) at the highest setting.
The LSTM performs worst overall in this metric.
The Transformer starts with the lowest pregnancy rate but rises steeply,
ending with the highest number of pregnancies at threshold 0.1.
By comparison, the baselines show much higher pregnancy rates throughout,
starting between 103 (user-mean) and 150 (population-mean) and increasing by about 50 pregnancies over the range of thresholds.
\paragraph{Intercourse Events.}
All trained models except the LSTM begin with low values of about 1000 intercourse events.
The LSTM starts much higher, around 3000, and increases sharply, converging toward approximately 6100.
All trained model variants show curved growth resembling logarithmic convergence, approaching the same upper range.
In contrast, the baselines start near 6000 events and converge toward the same levels as the trained models at higher thresholds.
\paragraph{Correct Denials.}
The number of correct denials decreases by about half across the tested thresholds.
All trained models cluster closely, starting at around 1800 correct denials,
with the Transformer and its convolutional variant slightly outperforming the LSTMs.
The baselines start at substantially lower levels: 1012 (user-mean), 902 (last-cycle), and 737 (population-mean).
\paragraph{Incorrect Denials.}
For incorrect denials, the Transformer-based models (except LSTM) start at very high values near 5000
but sharply decrease and converge between 500 and 1000.
The LSTM shows the same convergence pattern but starts considerably lower at 2654.
Baselines follow a different trend: the last-cycle and user-mean baselines decrease
modestly by about 100 over the range (starting at 749 and 762, respectively),
while the population-mean baseline instead decreases from 1102 to around 879.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/contraception_use_case_pregnancy_statistics}
\caption{
Contraception use-case results: pregnancies per 100 users per year and per 1000 intercourse events, stratified by fertility threshold.
}
\label{fig:results_contraception_use_case_pregnancy_statistics}
\end{figure}
To facilitate comparison, Figure~\ref{fig:results_contraception_use_case_pregnancy_statistics} presents two
normalized pregnancy metrics across fertility thresholds: the number of pregnancies per 100 users per year (only one potential pregnancy is counted per user),
and the number of pregnancies per 1000 intercourse events.
Here, the difference between the models becomes clearer.
The baselines start at 62 to 73 pregnancies per 100 users and become linearly worse over the threshold range.
The LSTM performs worse than all other trained models, except for the upper end of the threshold interval.
The Transformer, while being the best model for small thresholds, becomes worse over growing values until being on par with
the LSTM for the maximum tested value (0.1).
\paragraph{Summary.}
In the contraception use-case, the trained models achieve substantially lower pregnancy rates than the baselines,
though at the cost of fewer intercourse opportunities and more denials, both correct and incorrect.
This results in less pregnancies for both 100 users for a year and per 1000 intercourse events for all trained models.
Within the trained models, Transformers and their convolutional variants perform best overall, while LSTMs lag behind across most metrics.
There is an edge for the non-convolutional models for smaller thresholds that switches for higher values.
\subsubsection{Pregnancy Use-Case Results}\label{subsubsec:use_case_pregnancy_results}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/pregnancy_use_case_results_by_fertility_threshold}
\caption{
Contraception use-case evaluation study results by fertility threshold.
}
\label{fig:results_pregnancy_use_case_results_by_fertility_threshold}
\end{figure}
Figure~\ref{fig:results_pregnancy_use_case_results_by_fertility_threshold} presents the results for the pregnancy use-case
across varying fertility thresholds, comparing all model architectures and baselines.
The four reported metrics are: number of pregnancies, number of intercourse events, number of correct deferrals (i.e., correctly flagged infertile days),
and number of incorrect deferrals (i.e., fertile days incorrectly flagged as infertile).
The overall pattern mirrors the contraception use-case, but the optimization goal is reversed:
here, a higher number of pregnancies is desirable.
\paragraph{Pregnancies.}
Trained models achieve substantially higher pregnancy rates than the baselines.
They begin at around 275 pregnancies for low thresholds and decline linearly to between 157 and 176 at higher thresholds.
By contrast, the baselines start much lower (user-mean: 159, last-cycle: 139, population-mean: 117) and also decrease linearly,
reaching 119, 102, and 71 pregnancies, respectively.
This places all trained models well above baseline performance across the full threshold range.
\paragraph{Intercourse Events.}
Trained models start with high numbers of intercourse events, near 7000, except for the LSTM which begins lower at 4252.
All decline sublinearly and converge toward approximately 1500 at the highest thresholds.
In contrast, the baselines start far lower, between 1700 and 1900, and decrease linearly to about 1300.
Thus, the improved pregnancy rates of trained models come at the cost of substantially higher intercourse event counts.
\paragraph{Correct Deferrals.}
The baselines initially show higher numbers of correct deferrals, with values that increase linearly as thresholds rise.
Trained models begin much lower but follow a logarithmic-like growth pattern,
eventually converging with the baselines at high thresholds.
Within the trained group, the LSTM starts with noticeably higher values than the other models, though all converge toward a similar range.
\paragraph{Incorrect Deferrals.}
For trained models, incorrect deferrals start very low at small thresholds
(Transformer: 11, LSTM: 190) and grow with a curved, logarithmic-like pattern.
At high thresholds they reach between 1906 (Convolutional Transformer) and 2378 (Transformer).
The LSTM consistently produces more incorrect deferrals than the other trained models, except upper end of the threshold value range.
The Transformer once again shows the familiar pattern of excelling at low thresholds
but converging to the weakest performance among trained models at higher thresholds.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/pregnancy_use_case_pregnancy_statistics}
\caption{
Contraception use-case evaluation study results by fertility threshold.
}
\label{fig:results_pregnancy_use_case_pregnancy_statistics}
\end{figure}
To facilitate comparison here as well, Figure~\ref{fig:results_pregnancy_use_case_pregnancy_statistics} presents two
normalized pregnancy metrics across fertility thresholds: the number of pregnancies per 100 users per year,
and the number of pregnancies per 1000 intercourse events.
All trained models clearly result in more pregnancies per 100 users overall.
However, for lower fertility thresholds, there are fewer pregnancies per 1000 intercourse events for the trained models
compared to the baselines.
\paragraph{Summary.}
In the pregnancy use-case, trained models clearly outperform baselines in terms of pregnancy rates,
though at the expense of more intercourse events and higher incorrect deferrals.
This results in less pregnancies per 1000 intercourse events for the trained models on smaller thresholds.
However, for 100 user-years, the pregnancy rates are noticeably higher than the baselines.
The LSTM underperforms relative to the Transformer-based architectures across most metrics,
while the Transformer itself exhibits strong performance at low thresholds but deteriorates more rapidly with increasing thresholds.
There is an edge for the non-convolutional models for smaller thresholds that switches for higher values.
% -------------------------------------------------------------
\subsection{Summary of key results}\label{subsec:key_results}
% Keep this as a tight bulleted list mirroring RQs; strictly findings, no causes/interpretations.
%\begin{itemize}[leftmargin=*]
% \item \textbf{RQ1:} Temperature-based models achieve lowest MSEs on regular cycles; errors increase on irregular cycles.
% More history generally reduces MSE, with greater variance at higher history depth.
% \item \textbf{RQ2:} In contraception and conception scenarios, sequential models (LSTM/Transformer) yield
% the most favorable access–outcome trade-offs at threshold 0.05.
% \item \textbf{RQ3:} All ML models outperform rule-based baselines across targets and splits;
% the largest margins occur on post-ovulation metrics.
%\end{itemize}
\noindent\textbf{RQ1 (Architectures, context, and capacity).}
\begin{itemize}
\item \emph{Architectures.} All learned models beat baselines by a wide margin.
The \textbf{LSTM} is the most reliable overall and post-ovulation; the \textbf{Transformer} is strongest on fertile days.
Convolutional variants are competitive but rarely best.
\item \emph{Temporal context.} Performance depends on window length and differs by model.
LSTM benefits from long context (up to 160\,days), Transformers peak around 40–80\,days; conv models prefer mid-range (40\,days).
(Tables~\ref{tab:fertility_windows_compact_mse},~\ref{tab:ovover_windows_compact_mse}.)
\item \emph{Input resolution.} Low–medium rates (4–48/day) minimize \emph{overall} error for sequence models; high rates help fertile-day signals.
Conv models learn their own compression from full resolution. (Tables~\ref{tab:fertility_resolution_compact_mse},~\ref{tab:ovover_resolution_compact_mse}.)
\item \emph{Capacity.} Moderate-to-large configurations improve accuracy but with target- and phase-specific optima
(e.g., LSTM \(512\times4\); Transformer \(512\times4\times4\) for fertility overall, \(512\times8\times8\) for OV-over after-OV).
(Tables~\ref{tab:fertility_params_compact_mse}, \ref{tab:ovover_params_compact_mse}.)
\item Best configs (Tables~\ref{tab:best_configs_lstm},~\ref{tab:best_configs_transformer})
generalize on the held-out test set (Table~\ref{tab:results_model_selection_metrics}).
\item LSTM is best \emph{overall}: fertility MSE \(0.0036\) (best overall; best non-fertile \(0.0008\)) and OV-over overall \(0.0437\)
(best; best after-OV \(0.0492\)). Transformer is best on fertile days (MSE \(0.0089\)).
\item Versus the strongest baseline (User-Mean), LSTM cuts fertility MSE by \(\sim 44\%\) and post-ovulatory MSE by \(\sim 59\%\).
\end{itemize}
\noindent\textbf{RQ2 (Factors and Patterns).}
\begin{itemize}
\item Models consistently key on the \emph{pre-ovulatory temperature drop}; larger drops yield higher predicted fertility.
This pattern weakens in irregular cycles with noisier temperature traces (Figures~\ref{fig:results_rq2_regular_cycle_predictions_example}–\ref{fig:results_rq2_irregular_cycle_predictions_example}, \ref{fig:results_temperature_drop_pattern}).
\item More user history helps modestly; LSTM variants benefit most.
Baselines do not: Last-Cycle and Population-Mean often degrade with more history (Figure~\ref{fig:results_performance_on_different_historical_context}).
\end{itemize}
\noindent\textbf{RQ3 (Regular vs.\ Irregular).}
\begin{itemize}
\item All models are better on \emph{regular} cycles.
For fertility, Transformer leads overall and on fertile days; LSTM leads on non-fertile days.
For OV-over, LSTM leads overall and after-OV; Transformer leads before-OV (Tables~\ref{tab:fertility_mse_regular_irregular},~\ref{tab:ov_over_mse_regular_irregular}).
\item In \emph{irregular} cycles, errors rise across the board.
LSTM remains most robust; Convolutional LSTM is competitive for fertility overall; Transformers are less stable before/after phase splits.
\item Variance of predictions grow with longer context lengths.
\end{itemize}
\noindent\textbf{RQ4 (Use-case Evaluations).}
\begin{itemize}
\item \emph{Contraception:} Trained models cut pregnancies sharply vs baselines,
at the cost of more denials and fewer intercourse opportunities.
Transformer variants are best at low thresholds; LSTM lags overall.
Convolutional models excel at higher thresholds
(Figures~\ref{fig:results_contraception_use_case_results_by_fertility_threshold},~\ref{fig:results_contraception_use_case_pregnancy_statistics}).
\item \emph{Pregnancy seeking:} Trained models yield many more pregnancies than baselines but require
more intercourse events and accept more incorrect deferrals at high thresholds.
Transformer excels at low thresholds; LSTM underperforms overall.
Convolutional models again excel at higher thresholds
(Figures~\ref{fig:results_pregnancy_use_case_results_by_fertility_threshold},~\ref{fig:results_pregnancy_use_case_pregnancy_statistics}).
\end{itemize}
Reference in New Issue View Git Blame Copy Permalink