alex/temperature-based-fertility-prediction-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

hopefully final commit

Browse Source
This commit is contained in:
Alex Blank
2025-09-05 20:02:41 +00:00
parent a2fdf2f0a3
commit bf2e79cc4c
15 changed files with 709 additions and 449 deletions
Download Patch File Download Diff File Expand all files Collapse all files
thesis/sections/results.tex
+28 -211
View File
@@ -6,11 +6,14 @@
% find introduction
\subsection{RQ1: Comparative Study of Model Architectures}\label{subsec:rq1_temp_predictive_value}
\subsection{Objective 1: Comparative Study of Model Architectures}\label{subsec:rq1_temp_predictive_value}
We systematically evaluated multiple model architectures to assess their performance in
predicting the targets.
We report results for two targets: (i) fertility probability and (ii) an indicator that ovulation has already occurred (OV-Over).
We mainly report results for the fertility probability target.
Results for the second target, an indicator that ovulation has already occurred (OV-Over),
can be found in Appendix~\ref{sec:appendix_ov_over_results}
Each architecture was tested across a range of input window lengths, temporal resolutions, and model capacities.
The best-performing configurations for each architecture were selected for downstream analyses.
Unless stated otherwise, \textbf{MSE} is the primary metric (equivalent to the Brier score for probabilistic targets),
@@ -178,148 +181,6 @@ with full results in Appendix Tables~\ref{tab:fertility_results_by_model_paramet
\label{tab:fertility_params_compact_mse}
\end{table}
\subsubsection{Ovulation-Over Prediction}\label{subsubsec:ov_over_prediction}
We evaluate prediction of a binary indicator denoting whether ovulation has already occurred.
We examine the influence of input window length, input resolution, and model capacity.
\paragraph{Impact of input window length.}
For predicting whether ovulation has occurred (OV-over),
optimal window lengths again vary by architecture and target phase (pre- vs post-ovulation).
The \textbf{Transformer} yields the lowest overall (0.0533) and post-ovulation MSE (0.0520) at 40 days.
The \textbf{LSTM} performs best before ovulation (MSE 0.0212 at 20 days), while its overall MSE improves with longer context (160 days).
The \textbf{Convolutional LSTM} favors short windows, with the best overall MSE (0.0699), before-OV (0.0389),
and after-OV (0.0833) all occurring at 20 days.
The \textbf{Convolutional Transformer} performs best overall at 40 days and best before ovulation at 160 days (MSE 0.0286).
See Table~\ref{tab:ovover_windows_compact_mse} for a summary.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(days / MSE)}
& \shortstack[c]{Before-OV best\\(days / MSE)}
& \shortstack[c]{After-OV best\\(days / MSE)} \\
\midrule
LSTM
& 160 / 0.0557
& \textbf{20 / 0.0212}
& 160 / 0.0580 \\
Transformer
& \textbf{40 / 0.0533}
& 20 / 0.0255
& \textbf{40 / 0.0520} \\
Convolutional LSTM
& 20 / 0.0699
& 20 / 0.0389
& 20 / 0.0833 \\
Convolutional Transformer
& 40 / 0.0709
& 160 / 0.0286
& 40 / 0.0820 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best input window per architecture (MSE only) at a fixed input resolution of 12 values/day.
Bold entries are bests within a column.}
\label{tab:ovover_windows_compact_mse}
\end{table}
\paragraph{Impact of input resolution.}
Fixing the input window to 20 days, we compare input resolutions for LSTM and Transformer models.
For the \textbf{LSTM}, the lowest overall MSE (0.0633) occurs at 24/day,
with best before-OV and after-OV MSEs at 12/day (0.0212) and 48/day (0.0550), respectively.
The \textbf{Transformer} performs best overall at 72/day (0.0585),
with lowest before-OV MSE at 12/day (0.0255) and after-OV MSE at 288/day (0.0578).
Results are summarized in Table~\ref{tab:ovover_resolution_compact_mse};
full resolution grids are in Appendix Table~\ref{tab:ov_over_results_by_resolution}.
\begin{table}[t]
\small
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(values/day / MSE)}
& \shortstack[c]{Before-OV best\\(values/day / MSE)}
& \shortstack[c]{After-OV best\\(values/day / MSE)} \\
\midrule
LSTM
& 24 / 0.0633
& \textbf{12 / 0.0212}
& \textbf{48 / 0.0550} \\
Transformer
& \textbf{72 / 0.0585}
& 12 / 0.0255
& 288 / 0.0578 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best input resolution per architecture (MSE only) at a fixed input-window length of 20\,days.
Bold entries are bests within a column. Convolutional models are excluded (they consume 288 values/day internally).}
\label{tab:ovover_resolution_compact_mse}
\end{table}
\paragraph{Impact of model parameters.}
Using fixed input settings (160 days at 12/day for LSTM/Transformer; 40 days for convolutional models), we evaluate model scaling.
The \textbf{Transformer} achieves the lowest overall (0.0543) and after-ovulation MSE (0.0410) at a large configuration (512×8×8).
Before-OV MSE is lowest at 256×4×4 (0.0293).
For the \textbf{LSTM}, the best overall and after-OV performance is at 512×4,
while the best before-OV MSE occurs at 128×2 (0.0274).
The \textbf{Convolutional LSTM} performs best overall and after-OV at 256×4, with the best before-OV MSE at 128×2.
Finally, the \textbf{Convolutional Transformer} achieves its lowest overall and after-OV MSE at 512×4×4, and best before-OV MSE at 512×8×8.
Table~\ref{tab:ovover_params_compact_mse} summarizes these parameter-dependent results;
full comparisons are included in Appendix Tables~\ref{tab:ov_over_results_by_model_parameters_lstm}–\ref{tab:ov_over_results_by_model_parameters_conv_transformer}.
\begin{table}[t]
\scriptsize
\renewcommand{\arraystretch}{1.15}
\setlength{\tabcolsep}{6pt}
\begin{tabularx}{\linewidth}{l*{3}{>{\centering\arraybackslash}X}}
\toprule
\textbf{Architecture} & \multicolumn{3}{c}{\textbf{MSE}} \\
\cmidrule(r){2-4}
& \shortstack[c]{Overall best\\(params / MSE)}
& \shortstack[c]{Before-OV best\\(params / MSE)}
& \shortstack[c]{After-OV best\\(params / MSE)} \\
\midrule
LSTM
& \(512\times4\) / 0.0616
& \textbf{\(128\times2\) / 0.0274}
& \(512\times4\) / 0.0613 \\
Transformer
& \textbf{\(512\times8\times8\) / 0.0543}
& \(256\times4\times4\) / 0.0293
& \textbf{\(512\times8\times8\) / 0.0410} \\
Convolutional LSTM
& \(256\times4\) / 0.0687
& \(128\times2\) / 0.0357
& \(256\times4\) / 0.0715 \\
Convolutional Transformer
& \(512\times4\times4\) / 0.0703
& \(512\times8\times8\) / 0.0339
& \(512\times4\times4\) / 0.0814 \\
\bottomrule
\end{tabularx}
\caption{OV-Over: best parameter settings per architecture (MSE only).
Parameters are formatted as \(\text{hidden}\times\text{layers}\) (LSTM/Conv.\ LSTM) and \(\text{embedding}\times\text{encoder layers}\times\text{heads}\) (Transformer/Conv.\ Transformer).
Bold values indicate bests within a column. Fixed input settings: 160\,days with 12 values/day for LSTM/Transformer; 40\,days for convolutional models.}
\label{tab:ovover_params_compact_mse}
\end{table}
\subsubsection{Best Model Configuration Selection}\label{subsubsec:results_best_model_config_selection}
To identify the best configuration for each model architecture, we followed the procedure described in Section~\ref{subsubsec:methodology_best_model_config_selection}.
@@ -408,7 +269,7 @@ confirming the advantage of personalized and temporally-aware modeling.
% -------------------------------------------------------------------------------------------------------------------------------
\subsection{RQ2: Factors and Patterns that Influence Predictions}\label{subsec:rq2_factors_and_patterns}
\subsection{Objective 2: Factors and Patterns that Influence Predictions}\label{subsec:rq2_factors_and_patterns}
\subsubsection{Representative predictions}\label{subsubsec:rq2_examples}
@@ -517,7 +378,7 @@ but show greater scatter for users with longer data records.
% -------------------------------------------------------------
\subsection{RQ3: Performance across Regular and Irregular Cycles}\label{subsec:rq3_regular_vs_irregular}
\subsection{Objective 3: Performance across Regular and Irregular Cycles}\label{subsec:rq3_regular_vs_irregular}
\begin{table}[htbp]
\centering
@@ -568,59 +429,6 @@ Baselines degrade more sharply in this group.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_fertility_results}.
\vspace{1em}
\begin{table}[htbp]
\centering
\small
\begin{tabularx}{\linewidth}{l*{3}{X}}
\toprule
\multirow{2}{*}{Model} & \multicolumn{3}{c}{MSE} \\
\cmidrule(r){2-4}
& OV-Over Overall & OV-Over Before OV & OV-Over After OV \\
\midrule
\multicolumn{4}{c}{\textbf{Regular Cycle Group}} \\
\midrule
LSTM & \textbf{0.028519} & 0.031695 & \textbf{0.026116} \\
Transformer & 0.032929 & \textbf{0.024774} & 0.033411 \\
Convolutional LSTM & 0.034224 & 0.028962 & 0.033929 \\
Convolutional Transformer & 0.033333 & 0.039045 & 0.029329 \\
Last-Cycle Baseline & 0.087916 & 0.093465 & 0.074191 \\
Population Mean Baseline & 0.285887 & 0.008876 & 0.418879 \\
User Mean Baseline & 0.069245 & 0.069152 & 0.055724 \\
\midrule
\multicolumn{4}{c}{\textbf{Irregular Cycle Group}} \\
\midrule
LSTM & \textbf{0.051747} & \textbf{0.016777} & 0.070419 \\
Transformer & 0.055990 & 0.026159 & 0.067520 \\
Convolutional LSTM & 0.054523 & 0.030092 & \textbf{0.066643} \\
Convolutional Transformer & 0.056075 & 0.029638 & 0.067965 \\
Last-Cycle Baseline & 0.222906 & 0.134945 & 0.258047 \\
Population Mean Baseline & 0.171368 & 0.130366 & 0.140363 \\
User Mean Baseline & 0.180702 & 0.088884 & 0.217144 \\
\bottomrule
\end{tabularx}
\caption{MSEs for ovulation-over target across different model architectures for regular and irregular cycle groups. Bold values denote the best scores per column.}
\label{tab:ov_over_mse_regular_irregular}
\end{table}
\noindent
Table~\ref{tab:ov_over_mse_regular_irregular} shows the MSE metric results for all models on the regular and irregular cycle groups for the OV-over target.
For this target, performance patterns differ more distinctly across phases.
In the regular group, the LSTM performs best overall and after ovulation, while the Transformer performs best before ovulation.
Convolutional models perform slightly worse than their recurrent and transformer-based counterparts across all phases.
The population mean baseline shows a spurious low MSE before ovulation, but this is not consistent across phases, suggesting it is not reliable.
In general, baselines are notably weaker than learned models across all splits.
For irregular cycles, all model performances deteriorate relative to regular cycles.
The LSTM remains the most robust, achieving the best MSEs both overall and before ovulation.
Convolutional LSTM performs best after ovulation.
Transformer-based models perform less consistently in this group, suggesting decreased robustness to cycle irregularity.
As before, all learned models outperform the baselines by a wide margin.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_ov_over_results}.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_regular_cycles}
@@ -645,7 +453,7 @@ visualize performance improvements with increasing numbers of past cycles.
For regular cycles, all models except the population-mean baseline benefit from more historical data, showing consistent MSE reductions for both targets.
Interestingly, the population-mean baseline performs worse as more cycles are added.
The largest improvement can be seen for the last-cycle baseline and user-mean baseline models.
The outliers in the last-cycle and user-mean baseline are MSE values of 0.0, which result in these vertical lines.
The outliers in the last-cycle and user-mean baseline are MSE values of 0.0, which result in vertical lines based on the logarithmic scaling of the axis.
In contrast, irregular cycles exhibit more variability and less performance gain with more cycles available.
All trained models improve on both targets with more context.
@@ -662,7 +470,7 @@ as indicated by the wider deviations from the trend lines.
% -------------------------------------------------------------
\subsection{RQ4: Use-Case Evaluations}\label{subsec:rq4_use_case_evaluations}
\subsection{Objective 4: Evaluating Use-Case Scenarios}\label{subsec:rq4_use_case_evaluations}
We evaluated the use-case scenarios described in Section~\ref{subsubsec:practical_use_cases} using the algorithms
in Section~\ref{subsubsec:use_case_evaluation} across multiple fertility thresholds.
@@ -687,7 +495,7 @@ The four evaluated metrics are: number of pregnancies, number of intercourse eve
number of correct denials (i.e., appropriately flagged fertile days),
and number of incorrect denials (i.e., fertile days incorrectly flagged as infertile).
\paragraph{Pregnancies.}
\paragraph{Potential Pregnancies.}
All trained models start with very low pregnancy rates (Transformer: 0, LSTM: 4)
and then increase roughly linearly with the threshold,
reaching between 99 (Convolutional Transformer) and 122 (Transformer) at the highest setting.
@@ -696,12 +504,16 @@ The Transformer starts with the lowest pregnancy rate but rises steeply,
ending with the highest number of pregnancies at threshold 0.1.
By comparison, the baselines show much higher pregnancy rates throughout,
starting between 103 (user-mean) and 150 (population-mean) and increasing by about 50 pregnancies over the range of thresholds.
The control group, which does not use any predictions shows very high potential pregnancy rates at 274 for 100 user years.
Note, that the thresholds do not alter the control group, as intercourse can happen on any day by design.
\paragraph{Intercourse Events.}
All trained models except the LSTM begin with low values of about 1000 intercourse events.
The LSTM starts much higher, around 3000, and increases sharply, converging toward approximately 6100.
All trained model variants show curved growth resembling logarithmic convergence, approaching the same upper range.
In contrast, the baselines start near 6000 events and converge toward the same levels as the trained models at higher thresholds.
Here, the control group allows for 7610 intercourse events, which is also the maximum for 100 user years by design,
since no denials happen.
\paragraph{Correct Denials.}
The number of correct denials decreases by about half across the tested thresholds.
@@ -732,6 +544,7 @@ and the number of pregnancies per 1000 intercourse events.
Here, the difference between the models becomes clearer.
The baselines start at 62 to 73 pregnancies per 100 users and become linearly worse over the threshold range.
The control group results in 90 pregnant users of 100 for a year.
The LSTM performs worse than all other trained models, except for the upper end of the threshold interval.
The Transformer, while being the best model for small thresholds, becomes worse over growing values until being on par with
@@ -764,18 +577,20 @@ and number of incorrect deferrals (i.e., fertile days incorrectly flagged as inf
The overall pattern mirrors the contraception use-case, but the optimization goal is reversed:
here, a higher number of pregnancies is desirable.
\paragraph{Pregnancies.}
\paragraph{Potential Pregnancies.}
Trained models achieve substantially higher pregnancy rates than the baselines.
They begin at around 275 pregnancies for low thresholds and decline linearly to between 157 and 176 at higher thresholds.
By contrast, the baselines start much lower (user-mean: 159, last-cycle: 139, population-mean: 117) and also decrease linearly,
reaching 119, 102, and 71 pregnancies, respectively.
This places all trained models well above baseline performance across the full threshold range.
The control group results in 261 potential pregnancies for 100 user years.
\paragraph{Intercourse Events.}
Trained models start with high numbers of intercourse events, near 7000, except for the LSTM which begins lower at 4252.
All decline sublinearly and converge toward approximately 1500 at the highest thresholds.
In contrast, the baselines start far lower, between 1700 and 1900, and decrease linearly to about 1300.
Thus, the improved pregnancy rates of trained models come at the cost of substantially higher intercourse event counts.
Here, the control group allows for 7258 intercourse events, which is again the maximum possible for that interval by design.
\paragraph{Correct Deferrals.}
The baselines initially show higher numbers of correct deferrals, with values that increase linearly as thresholds rise.
@@ -804,9 +619,11 @@ To facilitate comparison here as well, Figure~\ref{fig:results_pregnancy_use_cas
normalized pregnancy metrics across fertility thresholds: the number of pregnancies per 100 users per year,
and the number of pregnancies per 1000 intercourse events.
All trained models clearly result in more pregnancies per 100 users overall.
All trained models clearly result in more pregnancies per 100 users than the baselines.
However, for lower fertility thresholds, there are fewer pregnancies per 1000 intercourse events for the trained models
compared to the baselines.
The control group attains 90 pregnant users per 100 for a year, which is on par with the trained models for low thresholds.
The efficiency is much lower than both trained and baseline models with 36 pregnancies per 1000 intercourse events.
\paragraph{Summary.}
In the pregnancy use-case, trained models clearly outperform baselines in terms of pregnancy rates,
@@ -820,7 +637,7 @@ There is an edge for the non-convolutional models for smaller thresholds that sw
% -------------------------------------------------------------
\subsection{Summary of key results}\label{subsec:key_results}
\subsection{Summary of Key Results}\label{subsec:key_results}
% Keep this as a tight bulleted list mirroring RQs; strictly findings, no causes/interpretations.
%\begin{itemize}[leftmargin=*]
@@ -832,10 +649,10 @@ There is an edge for the non-convolutional models for smaller thresholds that sw
% the largest margins occur on post-ovulation metrics.
%\end{itemize}
\noindent\textbf{RQ1 (Architectures, context, and capacity).}
\noindent\textbf{Objective 1: Architectures, context, and capacity.}
\begin{itemize}
\item \emph{Architectures.} All learned models beat baselines by a wide margin.
The \textbf{LSTM} is the most reliable overall and post-ovulation; the \textbf{Transformer} is strongest on fertile days.
The LSTM is the most reliable overall and post-ovulation; the Transformer is strongest on fertile days.
Convolutional variants are competitive but rarely best.
\item \emph{Temporal context.} Performance depends on window length and differs by model.
@@ -855,10 +672,10 @@ There is an edge for the non-convolutional models for smaller thresholds that sw
\item LSTM is best \emph{overall}: fertility MSE \(0.0036\) (best overall; best non-fertile \(0.0008\)) and OV-over overall \(0.0437\)
(best; best after-OV \(0.0492\)). Transformer is best on fertile days (MSE \(0.0089\)).
\item Versus the strongest baseline (User-Mean), LSTM cuts fertility MSE by \(\sim 44\%\) and post-ovulatory MSE by \(\sim 59\%\).
\item Versus the strongest baseline (User-Mean), LSTM cuts fertility MSE by \(\sim 44\%\) and ov-over MSE by \(\sim 59\%\).
\end{itemize}
\noindent\textbf{RQ2 (Factors and Patterns).}
\noindent\textbf{Objective 2: Factors and Patterns.}
\begin{itemize}
\item Models consistently key on the \emph{pre-ovulatory temperature drop}; larger drops yield higher predicted fertility.
This pattern weakens in irregular cycles with noisier temperature traces (Figures~\ref{fig:results_rq2_regular_cycle_predictions_example}–\ref{fig:results_rq2_irregular_cycle_predictions_example}, \ref{fig:results_temperature_drop_pattern}).
@@ -867,7 +684,7 @@ There is an edge for the non-convolutional models for smaller thresholds that sw
Baselines do not: Last-Cycle and Population-Mean often degrade with more history (Figure~\ref{fig:results_performance_on_different_historical_context}).
\end{itemize}
\noindent\textbf{RQ3 (Regular vs.\ Irregular).}
\noindent\textbf{Objective 3: Regular vs.\ Irregular.}
\begin{itemize}
\item All models are better on \emph{regular} cycles.
For fertility, Transformer leads overall and on fertile days; LSTM leads on non-fertile days.
@@ -879,7 +696,7 @@ There is an edge for the non-convolutional models for smaller thresholds that sw
\item Variance of predictions grow with longer context lengths.
\end{itemize}
\noindent\textbf{RQ4 (Use-case Evaluations).}
\noindent\textbf{Objective 4: Use-case Evaluations.}
\begin{itemize}
\item \emph{Contraception:} Trained models cut pregnancies sharply vs baselines,
at the cost of more denials and fewer intercourse opportunities.