470 lines
25 KiB
TeX
470 lines
25 KiB
TeX
%! Author = alex
|
||
%! Date = 3/6/25
|
||
|
||
|
||
\section{Results}\label{sec:results}
|
||
We report results for two targets: (i) fertility probability and (ii) an indicator that ovulation has already occurred (OV-Over).
|
||
Unless stated otherwise, \textbf{MSE} is the primary metric (equivalent to the Brier score for probabilistic targets),
|
||
and \textbf{MAE} is secondary.
|
||
We first summarize overall performance across architectures, then analyze sensitivity to input window length,
|
||
input resolution, and model capacity.
|
||
Finally, we compare the best configurations to three baselines and present stratified and use-case analyses.
|
||
|
||
\subsection{Overall Model Performance Across Architectures}\label{subsec:overall_model_performance_across_architectures}
|
||
|
||
We evaluated multiple model architectures to compare their effectiveness in predicting the fertility probability and ovulation-over targets.
|
||
Based on these results, we selected the best-performing configuration for each architecture for further analysis.
|
||
|
||
All metrics below are point estimates without uncertainty bounds; differences between models should be interpreted descriptively.
|
||
|
||
\subsubsection{Fertility-Probability Prediction}\label{subsec:fertility_probability_prediction}
|
||
|
||
This section examines model performance in predicting the probability of fertility,
|
||
focusing on the effects of input window length, input resolution, and key architecture parameters.
|
||
|
||
\paragraph{Impact of input window length.}
|
||
Across architectures, the optimal context is model-dependent.
|
||
Intermediate windows (20–40\,days) often minimize overall error, with the \textbf{Convolutional Transformer}
|
||
achieving the best withing grid at 40\,days (MAE 0.0376; MSE 0.0041).
|
||
The \textbf{LSTM} benefits from long context (160\,days: overall MAE 0.0378; MSE 0.0043) and ties the
|
||
global-best fertile-day MSE (0.0102), whereas the lowest non-fertile-day MSE occurs at 20\,days (0.0013; LSTM/Convolutional LSTM).
|
||
For the \textbf{Transformer}, overall MAE is lowest at 160\,days (0.0411), fertile-day error at 80\,days (MAE 0.0859; MSE 0.0108),
|
||
and non-fertile-day MSE at 160\,days (0.0016).
|
||
The \textbf{Convolutional LSTM} peaks at 40\,days (MAE 0.0394; MSE 0.0045), with non-fertile-day MSE best at 20\,days (0.0013).
|
||
Table~\ref{tab:fertility_windows_compact_mse} shows the compact results for the different architectures,
|
||
full grids (incl.\ MAE) are in Appendix, Table~\ref{tab:fertility_results_by_window_length}.
|
||
|
||
|
||
\begin{table}[t]
|
||
\small
|
||
\renewcommand{\arraystretch}{1.15}
|
||
\setlength{\tabcolsep}{6pt}
|
||
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
|
||
\toprule
|
||
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
|
||
\cmidrule(r){2-4}
|
||
& \shortstack[c]{Overall best\\(days / MSE)}
|
||
& \shortstack[c]{Fertile-day best\\(days / MSE)}
|
||
& \shortstack[c]{Non-fertile-day best\\(days / MSE)} \\
|
||
\midrule
|
||
LSTM
|
||
& 160 / 0.0043
|
||
& \textbf{160 / 0.0102}
|
||
& \textbf{20 / 0.0013} \\
|
||
Transformer
|
||
& 40 / 0.0044
|
||
& 80 / 0.0108
|
||
& 160 / 0.0016 \\
|
||
Convolutional LSTM
|
||
& 40 / 0.0045
|
||
& 40 / 0.0122
|
||
& \textbf{20 / 0.0013} \\
|
||
Convolutional Transformer
|
||
& \textbf{40 / 0.0041}
|
||
& \textbf{20 / 0.0102}
|
||
& 10, 40 / 0.0015 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Fertility probability: best input window per architecture (MSE only) at a fixed input resolution of 12 values/day.
|
||
Bold entries are bests within a column; ties are shown with commas.
|
||
Full grid (incl.\ MAE) in Appendix, Table~\ref{tab:fertility_results_by_window_length}.}
|
||
\label{tab:fertility_windows_compact_mse}
|
||
\end{table}
|
||
|
||
\paragraph{Impact of input resolution.}
|
||
We vary input resolution at a fixed 20\,day window and exclude convolutional architectures (they consume the full 288 values/day and learn downsampling internally).
|
||
For \textbf{LSTM}, overall MAE is lowest at 48 values/day (0.0402), while overall MSE is globally best at 4/day (0.0046).
|
||
Fertile-day errors are minimized at 288/day (MAE 0.0872; MSE 0.0110, best within grid), whereas non-fertile days are best at 12/day (MAE 0.0180; MSE 0.0013, both within grid).
|
||
For the \textbf{Transformer}, overall MAE is lowest at 48/day (0.0447); overall MSE ties at 48 and 288/day (0.0048).
|
||
Its fertile-day best is at 4/day (MAE 0.0891; MSE 0.0115), and non-fertile-day bests occur at 48/day (MAE 0.0254) and 72/day (MSE 0.0014).
|
||
In summary, intermediate resolutions (4–48/day) tend to minimize overall error, while an extreme (288/day) benefits LSTM on fertile-day performance.
|
||
Table~\ref{tab:fertility_resolution_compact_mse} shows the condensed results per architecture,
|
||
full grids (incl.\ MAE) are in Appendix, Table~\ref{tab:fertility_results_by_window_resolution}.
|
||
\begin{table}[t]
|
||
\small
|
||
\renewcommand{\arraystretch}{1.15}
|
||
\setlength{\tabcolsep}{6pt}
|
||
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
|
||
\toprule
|
||
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
|
||
\cmidrule(r){2-4}
|
||
& \shortstack[c]{Overall best\\(values/day / MSE)}
|
||
& \shortstack[c]{Fertile-day best\\(values/day / MSE)}
|
||
& \shortstack[c]{Non-fertile-day best\\(values/day / MSE)} \\
|
||
\midrule
|
||
LSTM
|
||
& \textbf{4 / 0.0046}
|
||
& \textbf{288 / 0.0110}
|
||
& \textbf{12 / 0.0013} \\
|
||
Transformer
|
||
& 48, 288 / 0.0048
|
||
& 4 / 0.0115
|
||
& 72 / 0.0014 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Fertility probability: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
|
||
Bold entries are bests within a column; ties are shown with commas. Convolutional models are excluded (fixed at 288/day).}
|
||
\label{tab:fertility_resolution_compact_mse}
|
||
\end{table}
|
||
|
||
\paragraph{Impact of model parameters.}
|
||
At fixed input settings (LSTM/Transformer: 160\,days, 12 values/day; Convolutional models: 40\,days),
|
||
larger capacities generally improve performance.
|
||
For \textbf{LSTM}, the best overall MSE is at \(512\times4\) (Hidden Layer Size, Number of LSTM Layers) (0.0047),
|
||
with the lowest fertile-day MSE also at \(512\times4\) (0.0122); the best non-fertile-day MSE occurs at \(256\times4\) (\,0.0011\,).
|
||
For the \textbf{Transformer}, overall MSE is minimized at \(512\times4\times4\) (Size of Embeddings, Number of Encoders, Number of Attention Heads)
|
||
(0.0043), fertile-day MSE at the same setting (0.0103), and non-fertile-day MSE at \(512\times8\times8\) (\,0.0011\,).
|
||
For the \textbf{Convolutional LSTM}, \(256\times4\) (Hidden Layer Size, Number of LSTM Layers) yields the lowest overall MSE (0.0042)
|
||
and the global-best fertile-day MSE (0.0100), while several smaller settings tie for the best non-fertile-day MSE (0.0016).
|
||
For the \textbf{Convolutional Transformer}, overall MSE ties at \(256\times4\times4\) and \(512\times4\times4\)
|
||
(Size of Embeddings, Number of Encoders, Number of Attention Heads) (\,0.0043\,);
|
||
fertile-day MSE is best at \(512\times8\times8\) (0.0107), and the non-fertile-day MSE is minimized at \(512\times4\times4\) (0.0013).
|
||
Table~\ref{tab:fertility_params_compact_mse} shows the compact results for the parameter search per architecture,
|
||
full parameter grids (incl.\ MAE) are in Appendix, Tables~\ref{tab:fertility_results_by_model_parameters_lstm}–\ref{tab:fertility_results_by_model_parameters_conv_transformer}.
|
||
|
||
\begin{table}[t]
|
||
\scriptsize
|
||
\renewcommand{\arraystretch}{1.15}
|
||
\setlength{\tabcolsep}{6pt}
|
||
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
|
||
\toprule
|
||
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
|
||
\cmidrule(r){2-4}
|
||
& \shortstack[c]{Overall best\\(params / MSE)}
|
||
& \shortstack[c]{Fertile-day best\\(params / MSE)}
|
||
& \shortstack[c]{Non-fertile-day best\\(params / MSE)} \\
|
||
\midrule
|
||
LSTM
|
||
& \(512\times4\) / 0.0047
|
||
& \(512\times4\) / 0.0122
|
||
& \textbf{\(256\times4\) / 0.0011} \\
|
||
Transformer
|
||
& \textbf{\(512\times4\times4\) / 0.0043}
|
||
& \(512\times4\times4\) / 0.0103
|
||
& \textbf{\(512\times8\times8\) / 0.0011} \\
|
||
Convolutional LSTM
|
||
& \(256\times4\) / 0.0042
|
||
& \textbf{\(256\times4\) / 0.0100}
|
||
& \(16\times1,\,32\times2,\,128\times2,\,128\times4\) / 0.0016 \\
|
||
Convolutional Transformer
|
||
& \textbf{\(256\times4\times4,\,512\times4\times4\) / 0.0043}
|
||
& \(512\times8\times8\) / 0.0107
|
||
& \(512\times4\times4\) / 0.0013 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Fertility probability: best parameter settings per architecture (MSE only).
|
||
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
|
||
Bold entries are bests within a column; ties are shown with commas.
|
||
Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
|
||
\label{tab:fertility_params_compact_mse}
|
||
\end{table}
|
||
|
||
\subsubsection{Ovulation-Over Prediction}\label{subsubsec:ov_over_prediction}
|
||
|
||
We evaluate prediction of a binary indicator denoting whether ovulation has already occurred.
|
||
We examine the influence of input window length, input resolution, and model capacity.
|
||
|
||
\paragraph{Impact of input window length.}
|
||
At a fixed input resolution of 12 values/day, the optimal window length for OV-Over is architecture- and phase-dependent.
|
||
The \textbf{Transformer} achieves the best overall and after-ovulation errors at 40\,days
|
||
(MAE 0.1017; MSE 0.0533 overall; MAE 0.0883; MSE 0.0520 after-OV).
|
||
The \textbf{LSTM} benefits from long context (160\,days: overall MAE 0.1039; MSE 0.0557; after-OV MSE 0.0580)
|
||
but attains the best before-ovulation scores at 20\,days (MAE 0.0745; MSE 0.0212).
|
||
For the \textbf{Convolutional LSTM}, short/intermediate windows work best (20–40\,days):
|
||
overall MSE 0.0699 at 20\,days; before-OV MSE 0.0389 and after-OV MSE 0.0833 at 20\,days.
|
||
The \textbf{Convolutional Transformer} peaks overall at 40\,days (MSE 0.0709; after-OV MSE 0.0820),
|
||
with its best before-OV MSE at 160\,days (0.0286).
|
||
In summary, \textit{intermediate} windows (20–40\,days) are often optimal—especially
|
||
pre-ovulation—while \textit{long} windows (160\,days) can improve post-ovulation for LSTM (and Convolutional Transformer before-OV MSE).
|
||
Table~\ref{tab:ovover_windows_compact_mse} shows the compact results per architecture,
|
||
full parameter grids (incl.\ MAE) are in Appendix, Tables~\ref{tab:ov_over_results_by_window_length}.
|
||
|
||
\begin{table}[t]
|
||
\small
|
||
\renewcommand{\arraystretch}{1.15}
|
||
\setlength{\tabcolsep}{6pt}
|
||
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
|
||
\toprule
|
||
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
|
||
\cmidrule(r){2-4}
|
||
& \shortstack[c]{Overall best\\(days / MSE)}
|
||
& \shortstack[c]{Before-OV best\\(days / MSE)}
|
||
& \shortstack[c]{After-OV best\\(days / MSE)} \\
|
||
\midrule
|
||
LSTM
|
||
& 160 / 0.0557
|
||
& \textbf{20 / 0.0212}
|
||
& 160 / 0.0580 \\
|
||
Transformer
|
||
& \textbf{40 / 0.0533}
|
||
& 20 / 0.0255
|
||
& \textbf{40 / 0.0520} \\
|
||
Convolutional LSTM
|
||
& 20 / 0.0699
|
||
& 20 / 0.0389
|
||
& 20 / 0.0833 \\
|
||
Convolutional Transformer
|
||
& 40 / 0.0709
|
||
& 160 / 0.0286
|
||
& 40 / 0.0820 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{OV-Over: best input window per architecture (MSE only) at a fixed input resolution of 12 values/day.
|
||
Bold entries are bests within a column.}
|
||
\label{tab:ovover_windows_compact_mse}
|
||
\end{table}
|
||
|
||
\paragraph{Impact of input resolution.}
|
||
With the window fixed at 20\,days, optimal resolution for OV-Over is architecture- and phase-specific.
|
||
For \textbf{LSTM}, overall MAE is lowest at 12/day (0.1153) and overall MSE at 24/day (0.0633);
|
||
the \emph{before-ovulation} MSE reaches the best within grid at 12/day (0.0212) and
|
||
the \emph{after-ovulation} MSE reaches the best within grid at 48/day (0.0550).
|
||
For the \textbf{Transformer}, both overall MAE (0.1041) and overall MSE achieve bests within grid at 72/day (0.0585),
|
||
with the best before-OV MSE at 12/day (0.0255) and the best after-OV MSE at 288/day (0.0578).
|
||
Overall, intermediate resolutions (12–72/day) tend to minimize overall error; 12/day favors pre-ovulation,
|
||
while 48–72/day improves post-ovulation, with 288/day giving a modest after-OV edge for the Transformer.
|
||
Table~\ref{tab:ovover_resolution_compact_mse} shows the condensed results per architecture,
|
||
full grids (incl.\ MAE) are in Appendix, Table~\ref{tab:ov_over_results_by_resolution}.
|
||
|
||
\begin{table}[t]
|
||
\small
|
||
\renewcommand{\arraystretch}{1.15}
|
||
\setlength{\tabcolsep}{6pt}
|
||
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
|
||
\toprule
|
||
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
|
||
\cmidrule(r){2-4}
|
||
& \shortstack[c]{Overall best\\(values/day / MSE)}
|
||
& \shortstack[c]{Before-OV best\\(values/day / MSE)}
|
||
& \shortstack[c]{After-OV best\\(values/day / MSE)} \\
|
||
\midrule
|
||
LSTM
|
||
& 24 / 0.0633
|
||
& \textbf{12 / 0.0212}
|
||
& \textbf{48 / 0.0550} \\
|
||
Transformer
|
||
& \textbf{72 / 0.0585}
|
||
& 12 / 0.0255
|
||
& 288 / 0.0578 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{OV-Over: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
|
||
Bold entries are bests within a column. Convolutional models are excluded (they consume 288 values/day internally).}
|
||
\label{tab:ovover_resolution_compact_mse}
|
||
\end{table}
|
||
|
||
\paragraph{Impact of model parameters.}
|
||
At fixed input settings (LSTM/Transformer: 160 days with 12 values/day; Convolutional models: 40 days),
|
||
larger capacities generally improve OV-Over performance.
|
||
For \textbf{LSTM}, the best overall MSE is at \(512\times4\) (Hidden Layer Size, Number of LSTM Layers)
|
||
(0.0616) with the global-best \emph{before-OV} MSE at \(128\times2\) (0.0274) and the best \emph{after-OV} MSE at \(512\times4\) (0.0613).
|
||
For the \textbf{Transformer}, scaling to \(512\times8\times8\) (Size of Embeddings, Number of Encoders, Number of Attention Heads)
|
||
yields the global-best overall MSE (0.0543) and \emph{after-OV} MSE (0.0410); the best \emph{before-OV} MSE is at \(256\times4\times4\) (0.0293).
|
||
For the \textbf{Convolutional LSTM}, \(256\times4\) (Hidden Layer Size, Number of LSTM Layers) minimizes
|
||
overall (0.0687) and \emph{after-OV} MSE (0.0715), while \(128\times2\) is best \emph{before-OV} (0.0357).
|
||
For the \textbf{Convolutional Transformer}, \(512\times4\times4\) (Size of Embeddings, Number of Encoders, Number of Attention
|
||
Heads) is best overall (0.0703) and \emph{after-OV} (0.0814), with \(512\times8\times8\) best \emph{before-OV} (0.0339).
|
||
Table~\ref{tab:ovover_params_compact_mse} shows the compact results for the parameter search per architecture,
|
||
full parameter grids (incl.\ MAE) are in Appendix, Tables~\ref{tab:ov_over_results_by_model_parameters_lstm}–\ref{tab:ov_over_results_by_model_parameters_conv_transformer}.
|
||
|
||
\begin{table}[t]
|
||
\scriptsize
|
||
\renewcommand{\arraystretch}{1.15}
|
||
\setlength{\tabcolsep}{6pt}
|
||
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
|
||
\toprule
|
||
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
|
||
\cmidrule(r){2-4}
|
||
& \shortstack[c]{Overall best\\(params / MSE)}
|
||
& \shortstack[c]{Before-OV best\\(params / MSE)}
|
||
& \shortstack[c]{After-OV best\\(params / MSE)} \\
|
||
\midrule
|
||
LSTM
|
||
& \(512\times4\) / 0.0616
|
||
& \textbf{\(128\times2\) / 0.0274}
|
||
& \(512\times4\) / 0.0613 \\
|
||
Transformer
|
||
& \textbf{\(512\times8\times8\) / 0.0543}
|
||
& \(256\times4\times4\) / 0.0293
|
||
& \textbf{\(512\times8\times8\) / 0.0410} \\
|
||
Convolutional LSTM
|
||
& \(256\times4\) / 0.0687
|
||
& \(128\times2\) / 0.0357
|
||
& \(256\times4\) / 0.0715 \\
|
||
Convolutional Transformer
|
||
& \(512\times4\times4\) / 0.0703
|
||
& \(512\times8\times8\) / 0.0339
|
||
& \(512\times4\times4\) / 0.0814 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{OV-Over: best parameter settings per architecture (MSE only).
|
||
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
|
||
Bold values indicate bests within a column. Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
|
||
\label{tab:ovover_params_compact_mse}
|
||
\end{table}
|
||
|
||
\subsubsection{Best Model Configuration Selection}\label{subsubsec:results_best_model_config_selection}
|
||
|
||
Following the selection procedure described in Section~\ref{subsubsec:methodology_best_model_config_selection}
|
||
the best configuration for each model architecture was identified based on the fertility overall MSE and, where applicable,
|
||
the general tendencies of the model.
|
||
Table~\ref{tab:best_configs_lstm} and~\ref{tab:best_configs_transformer} summarize the selected input window length,
|
||
input resolution, and model complexity for each architecture.
|
||
These configurations are used in all subsequent experiments, including the irregular cycles analysis and the use case evaluation.
|
||
Note, that the convolutional models always get the full resolution data (288 measurements per day).
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\scriptsize
|
||
\begin{tabularx}{\linewidth}{lXXXX}
|
||
\toprule
|
||
\textbf{Architecture} &
|
||
\textbf{Input Window Length} &
|
||
\textbf{Input Resolution} &
|
||
\textbf{Hidden Layer Size} &
|
||
\textbf{\# LSTM Layers} \\
|
||
\midrule
|
||
LSTM & 160 & 12 & 512 & 4 \\
|
||
Convolutional LSTM & 40 & 288 & 512 & 4 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Selected configurations for LSTM-based architectures. Input Window Length is given in days and input resolution in values per day.}
|
||
\label{tab:best_configs_lstm}
|
||
\end{table}
|
||
|
||
\begin{table}[htbp]
|
||
\centering
|
||
\scriptsize
|
||
\begin{tabularx}{\linewidth}{lXXXXX}
|
||
\toprule
|
||
\textbf{Architecture} &
|
||
\textbf{Input Window Length} &
|
||
\textbf{Input Resolution} &
|
||
\textbf{Embedding Size} &
|
||
\textbf{\# Encoder Layers} &
|
||
\textbf{\# Attention Heads} \\
|
||
\midrule
|
||
Transformer & 160 & 12 & 512 & 4 & 4 \\
|
||
Convolutional Transformer & 40 & 288 & 512 & 4 & 4 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Selected configurations for Transformer-based architectures. Input Window Length is given in days and input resolution in values per day.}
|
||
\label{tab:best_configs_transformer}
|
||
\end{table}
|
||
|
||
\begin{table}
|
||
\scriptsize
|
||
\begin{tabularx}{\linewidth}{l*{6}{X}}
|
||
\toprule
|
||
\multirow{1}{*}{Model} & \multicolumn{6}{c}{MSE} \\
|
||
\cmidrule(r){2-7}
|
||
& Fertility Overall & Fertile Days & Non-Fertile Days & OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
|
||
\midrule
|
||
Transformer & 0.0037 & \textbf{0.0089} & 0.0017 & 0.0508 & 0.0236 & 0.0581 \\
|
||
Convolutional Transformer & 0.0038 & 0.0098 & 0.0014 & 0.0517 & 0.0305 & 0.0566 \\
|
||
LSTM & \textbf{0.0036} & 0.0104 & \textbf{0.0008} & \textbf{0.0437} & \textbf{0.0233} & \textbf{0.0492} \\
|
||
Convolutional LSTM & 0.0037 & 0.0092 & 0.0014 & 0.0499 & 0.0281 & 0.0563 \\
|
||
User-Based-Mean Baseline & 0.0064 & 0.0154 & 0.0028 & 0.1072 & 0.0872 & 0.0983 \\
|
||
Last-Cycle Baseline & 0.0080 & 0.0202 & 0.0031 & 0.1453 & 0.1099 & 0.1762 \\
|
||
Population-Mean Baseline & 0.0127 & 0.0258 & 0.0074 & 0.2145 & 0.0400 & 0.3749 \\
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Evaluation Metrics for the fertility probability and OV-over targets (MSE) for the best model selection and the baselines for comparison.
|
||
\textbf{Bold} values represent the best values across all models for a given metric.}
|
||
\label{tab:results_model_selection_metrics}
|
||
\end{table}
|
||
|
||
Table~\ref{tab:results_model_selection_metrics} shows the performance of the best selected models for each architecture,
|
||
as well as the baselines for comparison.
|
||
The final numbers in Table~\ref{tab:results_model_selection_metrics} are on a held-out test set with no reuse.
|
||
All trained model architectures outperform the baselines across all metrics.
|
||
The LSTM achieves the global best MSE for both targets (\mbox{0.0036} fertility; \mbox{0.0437} OV-Over)
|
||
and the best non-fertile-day MSE (\mbox{0.0008}).
|
||
The Transformer attains the global best fertile-day MSE (\mbox{0.0089}).
|
||
Relative to the User-Mean baseline, the best LSTM reduces fertility MSE by \mbox{44\%} and OV-Over MSE by \mbox{59\%}.
|
||
|
||
\subsection{Stratified Analysis}\label{subsec:stratified_analysis}
|
||
|
||
\subsubsection{Influence of User History Depth}\label{subsubsec:influence_of_past_user_data}
|
||
% don't forget to also add baseline to tables
|
||
\begin{figure}[htbp]
|
||
\centering
|
||
\includegraphics[width=0.8\textwidth]{resources/figures/results/performance_on_different_historical_contexts}
|
||
\caption{
|
||
Plots showing the MSE for the fertility probability and OV-over targets over a growing size of past cycles available
|
||
for each model architecture, as well as a trend line for each metric.
|
||
}
|
||
\label{fig:results_performance_on_different_historical_context}
|
||
\end{figure}
|
||
|
||
Figure~\ref{fig:results_performance_on_different_historical_context} shows the MSE for the fertility probability
|
||
and OV-over targets as a function of the number of past cycles available per user, for all model architectures and baselines.
|
||
|
||
All four models improve on both metrics as the amount of historical data increases.
|
||
The LSTM-based models show a larger relative improvement with longer user history than the Transformer-based variants.
|
||
The User-Mean Baseline also improves substantially, with the Last-Cycle Baseline showing a smaller but still notable gain.
|
||
In contrast, the Population-Mean Baseline performs worse (i.e., MSE increases) as more historical cycles are included.
|
||
|
||
Across all models and baselines, the variability of results increases with greater history depth:
|
||
results are more tightly clustered around the trend line with short histories,
|
||
but show greater scatter for users with longer data records.
|
||
This suggests that longer histories introduce additional heterogeneity, which may reflect increased user-specific variability.
|
||
|
||
\subsubsection{Regular vs Irregular Cycles}\label{subsubsec:regular_vs_irregular_cycles}
|
||
\begin{landscape}
|
||
\begin{table}
|
||
\small
|
||
\begin{tabularx}{\linewidth}{l*{6}{X}}
|
||
\toprule
|
||
\multirow{2}{*}{Model Architecture} & \multicolumn{3}{c}{MAE} & \multicolumn{3}{c}{MSE} \\
|
||
\cmidrule(r){2-4} \cmidrule(r){5-7}
|
||
& Fertility Overall & Fertile Days & Non-Fertile Days & Fertility Overall & Fertile Days & Non-Fertile Days \\
|
||
\midrule
|
||
\multicolumn{7}{c}{\textbf{Regular Cycle Group}} \\
|
||
\midrule
|
||
|
||
\midrule
|
||
\multicolumn{7}{c}{\textbf{Irregular Cycle Group}} \\
|
||
\midrule
|
||
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Evaluation Metrics for the fertility probability target across Different Model Architectures for the Regular and Irregular Cycle Groups.
|
||
\underline{Underlined} values represent the best value for each metric within a model.
|
||
\textbf{\underline{Bold + Underlined}} values represent the best values across all models for a given metric.}
|
||
\label{tab:regular_vs_irregular_fertility_results}
|
||
\end{table}
|
||
\end{landscape}
|
||
\begin{landscape}
|
||
\begin{table}
|
||
\small
|
||
\begin{tabularx}{\linewidth}{l*{6}{X}}
|
||
\toprule
|
||
\multirow{2}{*}{Model} & \multicolumn{3}{c}{MAE} & \multicolumn{3}{c}{MSE} \\
|
||
\cmidrule(r){2-4} \cmidrule(r){5-7}
|
||
& OV-Over Overall & OV-Over Before OV & OV-Over After OV & OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
|
||
\midrule
|
||
\multicolumn{7}{c}{\textbf{Regular Cycle Group}} \\
|
||
\midrule
|
||
|
||
\midrule
|
||
\multicolumn{7}{c}{\textbf{Irregular Cycle Group}} \\
|
||
\midrule
|
||
|
||
\bottomrule
|
||
\end{tabularx}
|
||
\caption{Evaluation Metrics for the Ovulation-Over Target across Different Model Architectures for the Regular and Irregular Cycle Groups.
|
||
\underline{Underlined} values represent the best value for each metric within a model.
|
||
\textbf{\underline{Bold + Underlined}} values represent the best values across all models for a given metric.}
|
||
\label{tab:regular_vs_irregular_ov_over_results}
|
||
\end{table}
|
||
\end{landscape}
|
||
|
||
\subsection{Use-Case Evaluation Results}\label{subsec:use_case_evaluation_results}
|
||
|
||
\subsubsection{Contraception Use-Case Results}\label{subsubsec:use_case_contraception_results}
|
||
|
||
\subsubsection{Pregnancy Use-Case Results}\label{subsubsec:use_case_pregnancy_results}
|
||
|
||
\subsection{Summary of Key Findings}\label{subsec:summary_of_key_findings}
|
||
%best model per target, typical window lengths (20–40 d often optimal; LSTM benefits from 160 d), best resolutions (4–48/day for fertility; 12–72/day for OV-Over), and %-gains over baselines. |