alex/temperature-based-fertility-prediction-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
56bfc2b95d06cdd006bb74076069d779607d41f4
temperature-based-fertility…/thesis/sections/results.tex
T
Alex Blank 56bfc2b95d further work on discussion
2025-08-25 11:38:26 +00:00

824 lines
46 KiB
TeX
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
%! Author = alex
%! Date = 3/6/25
\section{Results}\label{sec:results}
We report results for two targets: (i) fertility probability and (ii) an indicator that ovulation has already occurred (OV-Over).
Unless stated otherwise, \textbf{MSE} is the primary metric (equivalent to the Brier score for probabilistic targets),
and \textbf{MAE} is secondary.
We first summarize overall performance across architectures, then analyze sensitivity to input window length,
input resolution, and model capacity.
Finally, we compare the best configurations to three baselines and present stratified and use-case analyses.
\subsection{Overall Model Performance Across Architectures}\label{subsec:overall_model_performance_across_architectures}
We systematically evaluated multiple model architectures to assess their performance in
predicting fertility probability and ovulation-over targets.
Each architecture was tested across a range of input window lengths, temporal resolutions, and model capacities.
The best-performing configurations for each architecture were selected for downstream analyses.
All metrics reported in this section are point estimates without confidence intervals; therefore, comparisons are descriptive.
\subsubsection{Fertility-Probability Prediction}\label{subsec:fertility_probability_prediction}
We analyze model performance in predicting fertility probability,
focusing on the influence of temporal context (window length), measurement resolution, and model capacity.
\paragraph{Impact of input window length.}
Across all architectures, predictive performance is sensitive to the temporal context length, but the optimal window varies by model.
We consider 10 days short, 20--40 medium and 80--160 long input windows.
The \textbf{LSTM} performs best with long context: at 160 days, it reaches the lowest fertile-day MSE (0.0102) and a competitive overall error(0.0043).
The lowest non-fertile-day MSE (0.0013) is tied between LSTM and Convolutional LSTM, both at 20 days.
The \textbf{Transformer} shows more varied behavior: its overall error is minimized at 40 days (0.0044),
fertile-day MSE at 80 days (0.0108), and non-fertile-day MSE at 160 days (0.0016).
For the \textbf{Convolutional LSTM}, the best results occur at 40 days overall (MSE 0.0045), with non-fertile-day MSE again best at 20 days.
The \textbf{Convolutional Transformer} achieves the lowest overall MSE (0.0041) at 40 days.
A summary of best-performing window lengths per architecture is shown in Table~\ref{tab:fertility_windows_compact_mse};
full results including MAE are provided in Appendix Table~\ref{tab:fertility_results_by_window_length}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(days / MSE)}
& \shortstack[c]{Fertile-day best\\(days / MSE)}
& \shortstack[c]{Non-fertile-day best\\(days / MSE)} \\
\midrule
LSTM
& 160 / 0.0043
& \textbf{160 / 0.0102}
& \textbf{20 / 0.0013} \\
Transformer
& 40 / 0.0044
& 80 / 0.0108
& 160 / 0.0016 \\
Convolutional LSTM
& 40 / 0.0045
& 40 / 0.0122
& \textbf{20 / 0.0013} \\
Convolutional Transformer
& \textbf{40 / 0.0041}
& \textbf{20 / 0.0102}
& 10, 40 / 0.0015 \\
\bottomrule
\end{tabularx}
\caption{Best-performing parameter settings per architecture for fertility probability prediction,
highlighting model-specific trade-offs between capacity and error minimization.
Bold entries are bests within a column; ties are shown with commas.
Full grid (incl.\ MAE) in Appendix, Table~\ref{tab:fertility_results_by_window_length}.}
\label{tab:fertility_windows_compact_mse}
\end{table}
\paragraph{Impact of input resolution.}
To assess the impact of temporal resolution, we fix the input window at 20 days and evaluate LSTM and Transformer models across a range of input rates.
We exclude convolutional models from this comparison, as they always receive the full-resolution input (288 values/day) and learn temporal compression internally.
We consider 1--8 values per day low resolution, 12--48 medium resolution and 72--288 high resolution.
For the \textbf{LSTM}, the lowest overall MSE (0.0046) occurs at 4 values/day,
while fertile-day MSE is minimized at full resolution (288/day, MSE 0.0110).
The best non-fertile-day MSE (0.0013) is achieved at 12/day.
The \textbf{Transformer} performs best overall at both 48 and 288/day (MSE 0.0048),
with the best fertile-day MSE at 4/day (0.0115) and best non-fertile-day MSE at 72/day (0.0014).
Overall, low to medium input resolutions (4–48/day) often minimize overall error,
though higher resolutions can be advantageous for detecting short-term fertile-day signals.
Results are summarized in Table~\ref{tab:fertility_resolution_compact_mse}; full metrics are in Appendix Table~\ref{tab:fertility_results_by_window_resolution}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(values/day / MSE)}
& \shortstack[c]{Fertile-day best\\(values/day / MSE)}
& \shortstack[c]{Non-fertile-day best\\(values/day / MSE)} \\
\midrule
LSTM
& \textbf{4 / 0.0046}
& \textbf{288 / 0.0110}
& \textbf{12 / 0.0013} \\
Transformer
& 48, 288 / 0.0048
& 4 / 0.0115
& 72 / 0.0014 \\
\bottomrule
\end{tabularx}
\caption{Fertility probability: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
Bold entries are bests within a column; ties are shown with commas. Convolutional models are excluded (fixed at 288/day).}
\label{tab:fertility_resolution_compact_mse}
\end{table}
\paragraph{Impact of model parameters.}
We next evaluate how architectural complexity, i.e., hidden size, number of layers,
and number of attention heads—influences performance under fixed input settings (160 days at 12/day for LSTM/Transformer, 40 days for convolutional models).
Parameters are reported as (hidden size × number of layers) for LSTM models and (embedding size × number of encoders × attention heads) for Transformers.
For the \textbf{LSTM}, the best overall and fertile-day MSEs are both observed at a configuration with 512 hidden units and 4 layers.
The lowest non-fertile-day MSE (0.0011) is reached at 256×4.
The \textbf{Transformer} performs best overall and on fertile days with 512-dimensional embeddings, 4 encoder layers,
and 4 attention heads (0.0043 and 0.0103, respectively).
Its lowest non-fertile-day MSE (0.0011) occurs at a larger configuration with 8 layers and 8 heads.
For the \textbf{Convolutional LSTM}, the best overall and fertile-day MSEs are reached with 256 hidden units and 4 layers,
while several lower-capacity settings tie for the best non-fertile-day MSE (0.0016).
The \textbf{Convolutional Transformer} performs best overall with either 256×4×4 or 512×4×4, and reaches its lowest non-fertile-day MSE at 512×4×4.
Detailed comparisons are shown in Table~\ref{tab:fertility_params_compact_mse},
with full results in Appendix Tables~\ref{tab:fertility_results_by_model_parameters_lstm}–\ref{tab:fertility_results_by_model_parameters_conv_transformer}.
\begin{table}[t]
\scriptsize
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(params / MSE)}
& \shortstack[c]{Fertile-day best\\(params / MSE)}
& \shortstack[c]{Non-fertile-day best\\(params / MSE)} \\
\midrule
LSTM
& \(512\times4\) / 0.0047
& \(512\times4\) / 0.0122
& \textbf{\(256\times4\) / 0.0011} \\
Transformer
& \textbf{\(512\times4\times4\) / 0.0043}
& \(512\times4\times4\) / 0.0103
& \textbf{\(512\times8\times8\) / 0.0011} \\
Convolutional LSTM
& \(256\times4\) / 0.0042
& \textbf{\(256\times4\) / 0.0100}
& \(16\times1,\,32\times2,\,128\times2,\,128\times4\) / 0.0016 \\
Convolutional Transformer
& \textbf{\(256\times4\times4,\,512\times4\times4\) / 0.0043}
& \(512\times8\times8\) / 0.0107
& \(512\times4\times4\) / 0.0013 \\
\bottomrule
\end{tabularx}
\caption{Fertility probability: best parameter settings per architecture (MSE only).
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
Bold entries are bests within a column; ties are shown with commas.
Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
\label{tab:fertility_params_compact_mse}
\end{table}
\subsubsection{Ovulation-Over Prediction}\label{subsubsec:ov_over_prediction}
We evaluate prediction of a binary indicator denoting whether ovulation has already occurred.
We examine the influence of input window length, input resolution, and model capacity.
\paragraph{Impact of input window length.}
For predicting whether ovulation has occurred (OV-over),
optimal window lengths again vary by architecture and target phase (pre- vs post-ovulation).
The \textbf{Transformer} yields the lowest overall (0.0533) and post-ovulation MSE (0.0520) at 40 days.
The \textbf{LSTM} performs best before ovulation (MSE 0.0212 at 20 days), while its overall MSE improves with longer context (160 days).
The \textbf{Convolutional LSTM} favors short windows, with the best overall MSE (0.0699), before-OV (0.0389),
and after-OV (0.0833) all occurring at 20 days.
The \textbf{Convolutional Transformer} performs best overall at 40 days and best before ovulation at 160 days (MSE 0.0286).
These trends suggest that intermediate windows (20–40 days) often balance short- and long-term signal,
while long windows (e.g., 160 days) help capture post-ovulation patterns.
See Table~\ref{tab:ovover_windows_compact_mse} for a summary.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(days / MSE)}
& \shortstack[c]{Before-OV best\\(days / MSE)}
& \shortstack[c]{After-OV best\\(days / MSE)} \\
\midrule
LSTM
& 160 / 0.0557
& \textbf{20 / 0.0212}
& 160 / 0.0580 \\
Transformer
& \textbf{40 / 0.0533}
& 20 / 0.0255
& \textbf{40 / 0.0520} \\
Convolutional LSTM
& 20 / 0.0699
& 20 / 0.0389
& 20 / 0.0833 \\
Convolutional Transformer
& 40 / 0.0709
& 160 / 0.0286
& 40 / 0.0820 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best input window per architecture (MSE only) at a fixed input resolution of 12 values/day.
Bold entries are bests within a column.}
\label{tab:ovover_windows_compact_mse}
\end{table}
\paragraph{Impact of input resolution.}
Fixing the input window to 20 days, we compare input resolutions for LSTM and Transformer models.
For the \textbf{LSTM}, the lowest overall MSE (0.0633) occurs at 24/day,
with best before-OV and after-OV MSEs at 12/day (0.0212) and 48/day (0.0550), respectively.
The \textbf{Transformer} performs best overall at 72/day (0.0585),
with lowest before-OV MSE at 12/day (0.0255) and after-OV MSE at 288/day (0.0578).
This indicates that high resolution benefits post-ovulation prediction, while lower rates suffice for pre-ovulation.
Results are summarized in Table~\ref{tab:ovover_resolution_compact_mse};
full resolution grids are in Appendix Table~\ref{tab:ov_over_results_by_resolution}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(values/day / MSE)}
& \shortstack[c]{Before-OV best\\(values/day / MSE)}
& \shortstack[c]{After-OV best\\(values/day / MSE)} \\
\midrule
LSTM
& 24 / 0.0633
& \textbf{12 / 0.0212}
& \textbf{48 / 0.0550} \\
Transformer
& \textbf{72 / 0.0585}
& 12 / 0.0255
& 288 / 0.0578 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
Bold entries are bests within a column. Convolutional models are excluded (they consume 288 values/day internally).}
\label{tab:ovover_resolution_compact_mse}
\end{table}
\paragraph{Impact of model parameters.}
Using fixed input settings (160 days at 12/day for LSTM/Transformer; 40 days for convolutional models), we evaluate model scaling.
The \textbf{Transformer} achieves the lowest overall (0.0543) and after-ovulation MSE (0.0410) at a large configuration (512×8×8).
Before-OV MSE is lowest at 256×4×4 (0.0293).
For the \textbf{LSTM}, the best overall and after-OV performance is at 512×4,
while the best before-OV MSE occurs at 128×2 (0.0274).
The \textbf{Convolutional LSTM} performs best overall and after-OV at 256×4, with the best before-OV MSE at 128×2.
Finally, the \textbf{Convolutional Transformer} achieves its lowest overall and after-OV MSE at 512×4×4, and best before-OV MSE at 512×8×8.
Table~\ref{tab:ovover_params_compact_mse} summarizes these parameter-dependent results;
full comparisons are included in Appendix Tables~\ref{tab:ov_over_results_by_model_parameters_lstm}–\ref{tab:ov_over_results_by_model_parameters_conv_transformer}.
\begin{table}[t]
\scriptsize
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(params / MSE)}
& \shortstack[c]{Before-OV best\\(params / MSE)}
& \shortstack[c]{After-OV best\\(params / MSE)} \\
\midrule
LSTM
& \(512\times4\) / 0.0616
& \textbf{\(128\times2\) / 0.0274}
& \(512\times4\) / 0.0613 \\
Transformer
& \textbf{\(512\times8\times8\) / 0.0543}
& \(256\times4\times4\) / 0.0293
& \textbf{\(512\times8\times8\) / 0.0410} \\
Convolutional LSTM
& \(256\times4\) / 0.0687
& \(128\times2\) / 0.0357
& \(256\times4\) / 0.0715 \\
Convolutional Transformer
& \(512\times4\times4\) / 0.0703
& \(512\times8\times8\) / 0.0339
& \(512\times4\times4\) / 0.0814 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best parameter settings per architecture (MSE only).
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
Bold values indicate bests within a column. Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
\label{tab:ovover_params_compact_mse}
\end{table}
\subsubsection{Best Model Configuration Selection}\label{subsubsec:results_best_model_config_selection}
To identify the best configuration for each model architecture, we followed the procedure described in Section~\ref{subsubsec:methodology_best_model_config_selection}.
Configurations were selected based on the overall fertility MSE on the validation set,
while also considering the general performance trends of each model.
Tables~\ref{tab:best_configs_lstm} and~\ref{tab:best_configs_transformer} summarize the selected hyperparameters for each architecture,
including input window length, temporal resolution, and model complexity.
For convolutional models, the input resolution was always fixed at the full 288 measurements per day.
\begin{table}[htbp]
\centering
\scriptsize
\begin{tabularx}{\linewidth}{lXXXX}
\toprule
\textbf{Architecture} &
\textbf{Input Window Length} &
\textbf{Input Resolution} &
\textbf{Hidden Layer Size} &
\textbf{\# LSTM Layers} \\
\midrule
LSTM & 160 & 12 & 512 & 4 \\
Convolutional LSTM & 40 & 288 & 512 & 4 \\
\bottomrule
\end{tabularx}
\caption{Selected configurations for LSTM-based architectures. Input window length is given in days; input resolution in samples per day.}
\label{tab:best_configs_lstm}
\end{table}
\begin{table}[htbp]
\centering
\scriptsize
\begin{tabularx}{\linewidth}{lXXXXX}
\toprule
\textbf{Architecture} &
\textbf{Input Window Length} &
\textbf{Input Resolution} &
\textbf{Embedding Size} &
\textbf{\# Encoder Layers} &
\textbf{\# Attention Heads} \\
\midrule
Transformer & 160 & 12 & 512 & 4 & 4 \\
Convolutional Transformer & 40 & 288 & 512 & 4 & 4 \\
\bottomrule
\end{tabularx}
\caption{Selected configurations for Transformer-based architectures. Input window length is given in days; input resolution in samples per day.}
\label{tab:best_configs_transformer}
\end{table}
Final evaluation of the selected configurations was performed on a held-out test set with no reuse of training or validation data.
Table~\ref{tab:results_model_selection_metrics} reports the MSE values for both prediction targets:
the fertility probability and the OV-over indicator.
All trained architectures outperform the baseline models across all evaluation metrics.
\begin{table}
\scriptsize
\begin{tabularx}{\linewidth}{l*{6}{X}}
\toprule
\multirow{1}{*}{Model} & \multicolumn{6}{c}{MSE} \\
\cmidrule(r){2-7}
& Fertility Overall & Fertile Days & Non-Fertile Days & OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
\midrule
Transformer & 0.0037 & \textbf{0.0089} & 0.0017 & 0.0508 & 0.0236 & 0.0581 \\
Convolutional Transformer & 0.0038 & 0.0098 & 0.0014 & 0.0517 & 0.0305 & 0.0566 \\
LSTM & \textbf{0.0036} & 0.0104 & \textbf{0.0008} & \textbf{0.0437} & \textbf{0.0233} & \textbf{0.0492} \\
Convolutional LSTM & 0.0037 & 0.0092 & 0.0014 & 0.0499 & 0.0281 & 0.0563 \\
User-Based-Mean Baseline & 0.0064 & 0.0154 & 0.0028 & 0.1072 & 0.0872 & 0.0983 \\
Last-Cycle Baseline & 0.0080 & 0.0202 & 0.0031 & 0.1453 & 0.1099 & 0.1762 \\
Population-Mean Baseline & 0.0127 & 0.0258 & 0.0074 & 0.2145 & 0.0400 & 0.3749 \\
\bottomrule
\end{tabularx}
\caption{Evaluation metrics (MSE) for fertility probability and OV-over targets using the selected model configurations.
Bold values indicate the best-performing architecture for each metric.}
\label{tab:results_model_selection_metrics}
\end{table}
The LSTM model achieves the best overall performance, with the lowest MSE for overall fertility (0.0036),
non-fertile days (0.0008), and all post-ovulatory metrics.
The Transformer model, however, yields the lowest MSE on fertile days (0.0089), slightly outperforming the LSTM in that subset.
Compared to the strongest baseline (User-Mean), the LSTM reduces the overall fertility MSE by 44\% and the post-ovulatory MSE by 59\%.
The performance gap is even larger when compared to the population-mean and last-cycle baselines,
confirming the advantage of personalized and temporally-aware modeling.
\subsection{Stratified Analysis}\label{subsec:stratified_analysis}
\subsubsection{Influence of User History Depth}\label{subsubsec:influence_of_past_user_data}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/performance_on_different_historical_contexts}
\caption{
Plots showing the MSE for the fertility probability and OV-over targets over a growing size of past cycles available
for each model architecture, as well as a trend line for each metric.
}
\label{fig:results_performance_on_different_historical_context}
\end{figure}
Figure~\ref{fig:results_performance_on_different_historical_context} shows the MSE for the fertility probability
and OV-over targets as a function of the number of past cycles available per user, for all model architectures and baselines.
All four models improve on both metrics as the amount of historical data increases.
The LSTM-based models show a larger relative improvement with longer user history than the Transformer-based variants.
The User-Mean Baseline also improves substantially, with the Last-Cycle Baseline showing a smaller but still notable gain.
In contrast, the Population-Mean Baseline performs worse (i.e., MSE increases) as more historical cycles are included.
Across all models and baselines, the variability of results increases with greater history depth:
results are more tightly clustered around the trend line with short histories,
but show greater scatter for users with longer data records.
\subsubsection{Regular vs Irregular Cycles}\label{subsubsec:regular_vs_irregular_cycles}
\begin{table}[htbp]
\centering
\small
\begin{tabularx}{\linewidth}{l*{3}{X}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{3}{c}{MSE} \\
\cmidrule(r){2-4}
& Fertility Overall & Fertile Days & Non-Fertile Days \\
\midrule
\multicolumn{4}{c}{\textbf{Regular Cycle Group}} \\
\midrule
LSTM & 0.002563 & 0.007850 & \textbf{0.000415} \\
Transformer & \textbf{0.002376} & \textbf{0.006651} & 0.000626 \\
Convolutional LSTM & 0.002460 & 0.007313 & 0.000573 \\
Convolutional Transformer & 0.002424 & 0.006919 & 0.000655 \\
Last-Cycle Baseline & 0.006592 & 0.014177 & 0.003198 \\
Population Mean Baseline & 0.016657 & 0.027671 & 0.011292 \\
User Mean Baseline & 0.005195 & 0.011715 & 0.002312 \\
\midrule
\multicolumn{4}{c}{\textbf{Irregular Cycle Group}} \\
\midrule
LSTM & 0.004490 & 0.013415 & \textbf{0.001076} \\
Transformer & 0.004122 & \textbf{0.010774} & 0.001783 \\
Convolutional LSTM & \textbf{0.004096} & 0.011188 & 0.001538 \\
Convolutional Transformer & 0.004281 & 0.011374 & 0.001639 \\
Last-Cycle Baseline & 0.009423 & 0.023650 & 0.004613 \\
Population Mean Baseline & 0.009795 & 0.022330 & 0.005751 \\
User Mean Baseline & 0.010548 & 0.023935 & 0.006159 \\
\bottomrule
\end{tabularx}
\caption{MSEs for fertility probability predictions across different model architectures for regular and irregular cycle groups. Bold values denote the best scores per column.}
\label{tab:fertility_mse_regular_irregular}
\end{table}
\noindent
Table~\ref{tab:fertility_mse_regular_irregular} shows the MSE metric results for all models on the regular and irregular
cycle groups for the fertility-probability target.
For this target, all learned models substantially outperform the baseline methods in both regular and irregular cycle groups.
In the regular group, the Transformer achieves the lowest overall MSE and MSE on fertile days, while the LSTM excels on non-fertile days.
Convolutional variants also perform competitively.
Among baselines, the user-specific mean performs better than both the last-cycle and population mean, but still falls short of all learned models.
In the irregular group, model performance is uniformly lower (i.e., MSEs are higher).
The Convolutional LSTM shows the lowest overall MSE, while the Transformer performs best on fertile days.
The LSTM remains the most effective on non-fertile days.
Baselines degrade more sharply in this group.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_fertility_results}.
\vspace{1em}
\begin{table}[htbp]
\centering
\small
\begin{tabularx}{\linewidth}{l*{3}{X}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{3}{c}{MSE} \\
\cmidrule(r){2-4}
& OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
\midrule
\multicolumn{4}{c}{\textbf{Regular Cycle Group}} \\
\midrule
LSTM & \textbf{0.028519} & 0.031695 & \textbf{0.026116} \\
Transformer & 0.032929 & \textbf{0.024774} & 0.033411 \\
Convolutional LSTM & 0.034224 & 0.028962 & 0.033929 \\
Convolutional Transformer & 0.033333 & 0.039045 & 0.029329 \\
Last-Cycle Baseline & 0.087916 & 0.093465 & 0.074191 \\
Population Mean Baseline & 0.285887 & 0.008876 & 0.418879 \\
User Mean Baseline & 0.069245 & 0.069152 & 0.055724 \\
\midrule
\multicolumn{4}{c}{\textbf{Irregular Cycle Group}} \\
\midrule
LSTM & \textbf{0.051747} & \textbf{0.016777} & 0.070419 \\
Transformer & 0.055990 & 0.026159 & 0.067520 \\
Convolutional LSTM & 0.054523 & 0.030092 & \textbf{0.066643} \\
Convolutional Transformer & 0.056075 & 0.029638 & 0.067965 \\
Last-Cycle Baseline & 0.222906 & 0.134945 & 0.258047 \\
Population Mean Baseline & 0.171368 & 0.130366 & 0.140363 \\
User Mean Baseline & 0.180702 & 0.088884 & 0.217144 \\
\bottomrule
\end{tabularx}
\caption{MSEs for ovulation-over target across different model architectures for regular and irregular cycle groups. Bold values denote the best scores per column.}
\label{tab:ov_over_mse_regular_irregular}
\end{table}
\noindent
Table~\ref{tab:ov_over_mse_regular_irregular} shows the MSE metric results for all models on the regular and irregular cycle groups for the OV-over target.
For this target, performance patterns differ more distinctly across phases.
In the regular group, the LSTM performs best overall and after ovulation, while the Transformer performs best before ovulation.
Convolutional models perform slightly worse than their recurrent and transformer-based counterparts across all phases.
The population mean baseline shows a spurious low MSE before ovulation, but this is not consistent across phases, suggesting it is not reliable.
In general, baselines are notably weaker than learned models across all splits.
For irregular cycles, all model performances deteriorate relative to regular cycles.
The LSTM remains the most robust, achieving the best MSEs both overall and before ovulation.
Convolutional LSTM performs best after ovulation.
Transformer-based models perform less consistently in this group, suggesting decreased robustness to cycle irregularity.
As before, all learned models outperform the baselines by a wide margin.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_ov_over_results}.
\vspace{1em}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_regular_cycles}
\caption{
MSE performance trends across increasing historical data availability for fertility and ovulation-over targets on regular cycles.
}
\label{fig:results_performance_on_regular_cycles}
\end{figure}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_irregular_cycles}
\caption{
MSE performance trends across increasing historical data availability for fertility and ovulation-over targets on irregular cycles.
}
\label{fig:results_performance_on_irregular_cycles}
\end{figure}
\paragraph{Evaluation on Historical Context Depth.}
Figures~\ref{fig:results_performance_on_regular_cycles} and~\ref{fig:results_performance_on_irregular_cycles} visualize performance improvements with increasing numbers of past cycles.
For regular cycles, all models except the population-mean baseline benefit from more historical data, showing consistent MSE reductions for both targets.
Interestingly, the population-mean baseline performs worse as more cycles are added.
The largest improvement can be seen for the last-cycle baseline and user-mean baseline models.
In contrast, irregular cycles exhibit more variability and less performance gain with more cycles available.
All trained models improve on both targets with more context.
The population-mean and user-mean baselines show improvement on the OV-over target.
In contrast, their MSE increases with more historical context for the fertility-probability target.
The last-cycle baseline shows marginal performance improvement for the OV-over target and the same increase for the fertility-probability
target as the other two baselines.
All models show increased output variability as the number of available past cycles grows,
as indicated by the wider deviations from the trend lines.
\subsection{Use-Case Evaluation Results}\label{subsec:use_case_evaluation_results}
We evaluated the use-case scenarios described in Section~\ref{subsubsec:practical_use_cases} using the algorithms
in Section~\ref{subsubsec:use_case_evaluation} across multiple fertility thresholds.
Each evaluation used a test set of 100 users (100 user-years) and was repeated for 200 iterations;
we report means and 95\% confidence intervals (CIs).
\subsubsection{Contraception Use-Case Results}\label{subsubsec:use_case_contraception_results}
\paragraph{Threshold 0.01 (Table~\ref{tab:results_contraception_use_case_0_01}).}
At the strictest threshold of 0.01, Transformer and LSTM models achieve the lowest pregnancy rates, only 4.4 to 4.6
on average, despite allowing approximately 3,200 intercourse events.
This equates to about 1.4 pregnancies per 1,000 intercourse events, a strong result for contraceptive reliability.
In contrast, convolutional variants (ConvLSTM and ConvTransformer) restrict intercourse events to around 1,150,
leading to roughly 34–35 pregnancies, or about 30 per 1,000 events—far less efficient in terms of balance between access and protection.
All baseline models perform substantially worse, with pregnancy counts exceeding 100 in all cases, confirming the value of personalized predictions.
\begin{table}
\centering
\begin{subtable}{\textwidth}
\centering
\scriptsize
\begin{tabularx}{\linewidth}{l*{5}{X}}
\toprule
Model & No. of Intercourse Events & Pregnancies & Correct Denials & Incorrect Denials \\
\midrule
Convolutional LSTM & 1154 (1149-1158) & 34.9 (34.1-35.7) & 1615 (1611-1620) & 4963 (4954-4971) \\
Transformer & 3200 (3194-3207) & 4.4 (4.1-4.7) & 1817 (1812-1822) & 2717 (2710-2724) \\
Convolutional Transformer & 1156 (1151-1160) & 34.3 (33.5-35.0) & 1613 (1608-1619) & 4974 (4965-4983) \\
LSTM & 3206 (3199-3213) & 4.6 (4.3-4.9) & 1832 (1827-1838) & 2711 (2705-2717) \\
Last-Cycle Baseline & 6066 (6057-6075) & 127.5 (125.9-129.1) & 918 (915-922) & 762 (759-766) \\
Population-Mean Baseline & 5874 (5865-5883) & 153.7 (151.9-155.5) & 749 (746-752) & 1121 (1116-1125) \\
User-Mean Baseline & 5933 (5924-5942) & 105.8 (104.4-107.2) & 1030 (1026-1034) & 776 (772-779) \\
\bottomrule
\end{tabularx}
\caption{Contraception metrics at threshold \textbf{0.01} for all models.}
\label{tab:results_contraception_use_case_0_01}
\end{subtable}
\vspace{1.5em}
\begin{subtable}{\textwidth}
\centering
\scriptsize
\begin{tabularx}{\linewidth}{l*{5}{X}}
\toprule
Model & No. of Intercourse Events & Pregnancies & Correct Denials & Incorrect Denials \\
\midrule
Convolutional LSTM & 2509 (2502-2515) & 55.0 (54.0-56.0) & 1448 (1444-1452) & 3768 (3760-3775) \\
Transformer & 3801 (3793-3810) & 8.2 (7.8-8.6) & 1768 (1763-1773) & 2162 (2157-2168) \\
Convolutional Transformer & 2510 (2503-2516) & 55.6 (54.5-56.6) & 1448 (1443-1453) & 3776 (3769-3783) \\
LSTM & 3801 (3794-3809) & 8.2 (7.8-8.7) & 1772 (1766-1777) & 2160 (2154-2165) \\
Last-Cycle Baseline & 6106 (6096-6116) & 130.2 (128.7-131.7) & 888 (885-892) & 750 (746-753) \\
Population-Mean Baseline & 5924 (5914-5933) & 156.1 (154.3-157.8) & 716 (713-720) & 1100 (1096-1104) \\
User-Mean Baseline & 5988 (5978-5997) & 108.8 (107.3-110.3) & 998 (994-1002) & 764 (761-768) \\
\bottomrule
\end{tabularx}
\caption{Contraception metrics at threshold \textbf{0.05} for all models.}
\label{tab:results_contraception_use_case_0_05}
\end{subtable}
\vspace{1.5em}
\begin{subtable}{\textwidth}
\centering
\scriptsize
\begin{tabularx}{\linewidth}{l*{5}{X}}
\toprule
Model & No. of Intercourse Events & Pregnancies & Correct Denials & Incorrect Denials \\
\midrule
Convolutional LSTM & 3596 (3588-3603) & 75.2 (74.0-76.4) & 1292 (1288-1297) & 2847 (2840-2853) \\
Transformer & 4226 (4217-4234) & 13.3 (12.8-13.7) & 1692 (1687-1696) & 1822 (1817-1827) \\
Convolutional Transformer & 3592 (3584-3599) & 73.8 (72.7-75.0) & 1296 (1292-1301) & 2846 (2840-2852) \\
LSTM & 4222 (4214-4230) & 13.3 (12.8-13.8) & 1694 (1688-1699) & 1818 (1812-1823) \\
Last-Cycle Baseline & 6145 (6135-6155) & 134.8 (133.3-136.3) & 865 (861-868) & 744 (740-747) \\
Population-Mean Baseline & 5960 (5951-5969) & 160.3 (158.5-162.1) & 694 (691-697) & 1088 (1084-1093) \\
User-Mean Baseline & 6014 (6005-6022) & 111.8 (110.3-113.3) & 969 (965-973) & 758 (755-762) \\
\bottomrule
\end{tabularx}
\caption{Contraception metrics at threshold \textbf{0.10} for all models.}
\label{tab:results_contraception_use_case_0_10}
\end{subtable}
\caption{grouped contraception metrics at thresholds 0.01, 0.05, and 0.10. values are means over 200 iterations; 95\% confidence intervals in parentheses.}
\label{tab:results_contraception_grouped}
\end{table}
\paragraph{Threshold 0.05 (Table~\ref{tab:results_contraception_use_case_0_05}).}
Increasing the threshold to 0.05 improves access: Transformer and LSTM models now permit approximately 3,800 intercourse
events while still limiting pregnancies to around 8.2 (2.2 per 1,000).
This represents a ~19\% increase in events over threshold 0.01, at the cost of a modest rise in pregnancies (+3.8 absolute, +86\% relative).
Convolutional models also allow more events (~2,510) but continue to produce significantly more pregnancies (~55),
yielding a less favorable risk-benefit profile.
Baselines remain underperforming.
\paragraph{Threshold 0.10 (Table~\ref{tab:results_contraception_use_case_0_10}).}
A further increase to 0.10 raises Transformer/LSTM events to ~4,220, but also raises pregnancies to ~13.3 (3.1 per 1,000).
This is a ~11\% gain in access compared to 0.05, but the pregnancy count increases by ~62\%.
Meanwhile, convolutional models permit ~3,590 events and result in 74–75 pregnancies, indicating a consistent trade-off in favor of sequential models.
\paragraph{Contraception—Recommended Threshold.}
Threshold 0.05 achieves a favorable balance between access and effectiveness.
Transformer and LSTM models perform best, allowing a relatively high number of intercourse events while keeping pregnancies low.
This threshold offers the best compromise and is selected as the most promising setting for contraceptive use.
\subsubsection{Pregnancy Use-Case Results}\label{subsubsec:use_case_pregnancy_results}
\begin{table}
\centering
\begin{subtable}{\textwidth}
\centering
\scriptsize
\begin{tabularx}{\linewidth}{l*{5}{X}}
\toprule
Model & No. of Intercourse Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\
\midrule
Convolutional LSTM & 4899 (4890-4908) & 216.9 (214.8-219.1) & 10038 & 1734 \\
Transformer & 3685 (3678-3692) & 264.9 (262.7-267.0) & 17489 & 350 \\
Convolutional Transformer & 4903 (4894-4912) & 214.3 (212.2-216.4) & 10038 & 1734 \\
LSTM & 3688 (3680-3696) & 266.9 (264.7-269.1) & 17489 & 350 \\
Last-Cycle Baseline & 1536 (1531-1541) & 138.7 (137.1-140.2) & 24593 & 4038 \\
Population-Mean Baseline & 1704 (1699-1709) & 113.5 (112.1-114.9) & 22956 & 4841 \\
User-Mean Baseline & 1652 (1647-1656) & 158.4 (156.8-160.0) & 24528 & 3523 \\
\bottomrule
\end{tabularx}
\caption{Pregnancy metrics at threshold \textbf{0.01} for all models. }
\label{tab:results_pregnancy_use_case_0_01}
\end{subtable}
\vspace{1.5em}
\begin{subtable}{\textwidth}
\centering
\scriptsize
\begin{tabularx}{\linewidth}{l*{5}{X}}
\toprule
Model & No. of Intercourse Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\
\midrule
Convolutional LSTM & 6174 (6164-6184) & 236.1 (234.1-238.2) & 4332 & 1076 \\
Transformer & 4253 (4245-4261) & 270.9 (268.6-273.1) & 14825 & 190 \\
Convolutional Transformer & 6173 (6164-6182) & 235.7 (233.6-237.9) & 4332 & 1076 \\
LSTM & 4255 (4247-4262) & 273.0 (270.6-275.3) & 14825 & 190 \\
Last-Cycle Baseline & 1581 (1576-1586) & 139.8 (138.2-141.3) & 24327 & 4091 \\
Population-Mean Baseline & 1752 (1747-1757) & 117.1 (115.6-118.6) & 22655 & 4888 \\
User-Mean Baseline & 1695 (1690-1701) & 158.3 (156.6-160.1) & 24271 & 3567 \\
\bottomrule
\end{tabularx}
\caption{Pregnancy metrics at threshold \textbf{0.05} for all models.}
\label{tab:results_pregnancy_use_case_0_05}
\end{subtable}
\vspace{1.5em}
\begin{subtable}{\textwidth}
\centering
\scriptsize
\begin{tabularx}{\linewidth}{l*{5}{X}}
\toprule
Model & No. of Intercourse Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\
\midrule
Convolutional LSTM & 3887 (3880-3895) & 198.0 (196.1-199.9) & 14544 & 2296 \\
Transformer & 3291 (3284-3298) & 260.7 (258.6-262.7) & 19261 & 551 \\
Convolutional Transformer & 3883 (3875-3891) & 198.4 (196.6-200.3) & 14544 & 2296 \\
LSTM & 3296 (3289-3303) & 261.1 (259.1-263.2) & 19261 & 551 \\
Last-Cycle Baseline & 1509 (1504-1513) & 135.2 (133.6-136.8) & 24778 & 4010 \\
Population-Mean Baseline & 1668 (1663-1673) & 109.5 (108.1-110.9) & 23156 & 4811 \\
User-Mean Baseline & 1619 (1614-1624) & 154.0 (152.4-155.6) & 24717 & 3501 \\
\bottomrule
\end{tabularx}
\caption{Pregnancy metrics at threshold \textbf{0.10} for all models.}
\label{tab:results_pregnancy_use_case_0_10}
\end{subtable}
\caption{grouped pregnancy metrics at thresholds 0.01, 0.05, and 0.10. values are means over 200 iterations; 95\% confidence intervals in parentheses.
Under our simulation at a fixed threshold, correct/incorrect deferrals are deterministic; CIs are therefore omitted for these columns.}
\label{tab:results_pregnancy_grouped}
\end{table}
\paragraph{Threshold 0.01 (Table~\ref{tab:results_pregnancy_use_case_0_01}).}
In the conception use-case, Transformer and LSTM models generate the highest pregnancy counts (~265–267) at ~3,685 intercourse events (72 per 1,000).
Convolutional models yield fewer pregnancies (~214–217) but allow ~4,900 intercourse events, resulting in ~44 pregnancies per 1,000.
Baselines underperform on both metrics, allowing fewer events and achieving lower pregnancy counts, suggesting they are
overly conservative without yielding benefits in effectiveness.
\paragraph{Threshold 0.05 (Table~\ref{tab:results_pregnancy_use_case_0_05}).}
At threshold 0.05, Transformer/LSTM models slightly increase intercourse access (~4,250 events) with pregnancies rising to ~271–273 (64 per 1,000).
Notably, these models also achieve very low incorrect deferral counts (~190), indicating they rarely block opportunities for conception when they shouldn’t.
Convolutional models allow substantially more intercourse (~6,170) with lower pregnancy counts (~236),
but at the cost of higher incorrect deferrals (~1,076).
This suggests they are more permissive but less selective.
\paragraph{Threshold 0.10 (Table~\ref{tab:results_pregnancy_use_case_0_10}).}
At the highest threshold, Transformer/LSTM models see a drop in access (~3,290 events) and in pregnancies (~261),
but with an increase in incorrect deferrals (~551).
Convolutional models again show a more permissive profile (~3,880 events, ~198 pregnancies), but with higher rates of incorrect deferrals (~2,296).
This threshold leads to fewer pregnancies and more unnecessary blocks, especially for the Transformer/LSTM models.
\paragraph{Pregnancy—Recommended Threshold.}
Threshold 0.05 strikes the best balance for conception as well.
Transformer and LSTM models provide high pregnancy counts and good access with minimal incorrect deferrals.
Compared to 0.10, it results in more successful conceptions with fewer missed opportunities, making it the optimal setting for this use-case.
\subsection{Summary of Key Findings}\label{subsec:summary_of_key_findings}
\paragraph{Model Performance.}
LSTM models consistently achieved the lowest overall mean squared errors (MSE) across both prediction targets,
fertility probability and ovulation-over (OV-over), on the held-out test set.
Transformer models slightly outperformed LSTMs on fertile-day predictions, suggesting higher sensitivity to short-term signals.
Convolutional models (ConvLSTM and ConvTransformer) performed competitively in some configurations but generally
exhibited higher error and less favorable trade-offs in practical scenarios.
\paragraph{Effect of Input Settings.}
\textit{Input window length} significantly influenced performance.
LSTMs and Transformers benefited from longer windows (e.g., 160\,days), especially for post-ovulation detection,
while convolutional models favored shorter windows (e.g., 20--40\,days).
\textit{Input resolution} showed that intermediate sampling rates (4--48\,values/day) often minimized overall error,
though full resolution (288/day) was most useful for detecting fertile windows in LSTM models.
\paragraph{Model Scaling.}
Larger configurations (e.g., 512 hidden units, 4--8 layers, 4--8 attention heads) generally improved performance across architectures,
especially for Transformers in OV-over prediction.
\paragraph{Comparison to Baselines.}
All trained models substantially outperformed baseline predictors (user mean, last-cycle, population mean) across metrics and targets.
The LSTM model reduced fertility MSE by 44\% and post-ovulation MSE by 59\% compared to the strongest baseline.
\paragraph{Stratified Analyses.}
\textit{User history depth:} More historical cycles consistently improved prediction accuracy, especially for LSTM models.
However, output variability increased with longer histories.
\textit{Cycle regularity:} All models performed better on regular cycles than irregular ones.
LSTM remained the most robust across both groups.
\paragraph{Use-Case Scenarios.}
In \textit{contraceptive settings}, Transformer and LSTM models maintained pregnancy rates below 2.2 per 1{,}000 intercourse events
at a 0.05 threshold---demonstrating both high reliability and user access.
In \textit{conception settings}, the same models yielded the highest pregnancy rates ($\sim$64 per 1{,}000 events)
with minimal missed opportunities, again at the 0.05 threshold.
Convolutional models allowed more events but were less selective,
resulting in higher pregnancy rates in contraception and lower in conception scenarios.
Baseline methods were consistently outperformed in both use-cases, often sacrificing either effectiveness or access.
Reference in New Issue View Git Blame Copy Permalink