alex/temperature-based-fertility-prediction-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fixes and improvements

Browse Source
This commit is contained in:
Alex Blank
2025-09-09 13:44:45 +00:00
parent bf2e79cc4c
commit e71f1cd2e3
10 changed files with 354 additions and 234 deletions
Download Patch File Download Diff File Expand all files Collapse all files
thesis/sections/results.tex
+44 -35
View File
@@ -8,9 +8,9 @@
\subsection{Objective 1: Comparative Study of Model Architectures}\label{subsec:rq1_temp_predictive_value}
We systematically evaluated multiple model architectures to assess their performance in
I have systematically evaluated multiple model architectures to assess their performance in
predicting the targets.
We mainly report results for the fertility probability target.
I mainly report results for the fertility probability target.
Results for the second target, an indicator that ovulation has already occurred (OV-Over),
can be found in Appendix~\ref{sec:appendix_ov_over_results}
@@ -19,17 +19,21 @@ The best-performing configurations for each architecture were selected for downs
Unless stated otherwise, \textbf{MSE} is the primary metric (equivalent to the Brier score for probabilistic targets),
and \textbf{MAE} is secondary.
All metrics reported in this section are point estimates without confidence intervals; therefore, comparisons are descriptive.
While models were compared on the same user-disjoint test set, predictions were not retained at the per-user level,
which prevents the computation of confidence intervals or paired significance tests.
Reported differences between models and thresholds should therefore be interpreted descriptively rather than inferentially.
Future evaluations should retain user-level predictions to enable bootstrap confidence intervals and paired comparisons,
ensuring robust quantification of uncertainty.
\subsubsection{Fertility-Probability Prediction}\label{subsec:fertility_probability_prediction}
We analyze model performance in predicting fertility probability,
I have analysed model performance in predicting fertility probability,
focusing on the influence of temporal context (window length), measurement resolution, and model capacity.
\paragraph{Impact of input window length.}
Across all architectures, predictive performance is sensitive to the temporal context length, but the optimal window varies by model.
We consider 10 days short, 20--40 medium and 80--160 long input windows.
I consider 10 days short, 20--40 medium and 80--160 long input windows.
The \textbf{LSTM} performs best with long context: at 160 days, it reaches the lowest fertile-day MSE (0.0102) and a competitive overall error(0.0043).
The lowest non-fertile-day MSE (0.0013) is tied between LSTM and Convolutional LSTM, both at 20 days.
@@ -79,10 +83,10 @@ full results including MAE are provided in Appendix Table~\ref{tab:fertility_res
\end{table}
\paragraph{Impact of input resolution.}
To assess the impact of temporal resolution, we fix the input window at 20 days and evaluate LSTM and Transformer models across a range of input rates.
We exclude convolutional models from this comparison, as they always receive the full-resolution input (288 values/day) and learn temporal compression internally.
To assess the impact of temporal resolution, I have fixed the input window at 20 days and evaluate LSTM and Transformer models across a range of input rates.
I have excluded convolutional models from this comparison, as they always receive the full-resolution input (288 values/day) and learn temporal compression internally.
We consider 1--8 values per day low resolution, 12--48 medium resolution and 72--288 high resolution.
I consider 1--8 values per day low resolution, 12--48 medium resolution and 72--288 high resolution.
For the \textbf{LSTM}, the lowest overall MSE (0.0046) occurs at 4 values/day,
while fertile-day MSE is minimized at full resolution (288/day, MSE 0.0110).
@@ -125,7 +129,7 @@ Results are summarized in Table~\ref{tab:fertility_resolution_compact_mse}; full
\paragraph{Impact of model parameters.}
We next evaluate how architectural complexity, i.e., hidden size, number of layers,
Next, I have evaluated how architectural complexity, i.e., hidden size, number of layers,
and number of attention heads, influences performance under fixed input settings (160 days at 12/day for LSTM/Transformer, 40 days for convolutional models).
Parameters are reported as (hidden size × number of layers) for LSTM models and (embedding size × number of encoders × attention heads) for Transformers.
@@ -183,14 +187,14 @@ with full results in Appendix Tables~\ref{tab:fertility_results_by_model_paramet
\subsubsection{Best Model Configuration Selection}\label{subsubsec:results_best_model_config_selection}
To identify the best configuration for each model architecture, we followed the procedure described in Section~\ref{subsubsec:methodology_best_model_config_selection}.
To identify the best configuration for each model architecture, I have followed the procedure described in Section~\ref{subsubsec:methodology_best_model_config_selection}.
Configurations were selected based on the overall fertility MSE on the validation set,
while also considering the general performance trends of each model.
Tables~\ref{tab:best_configs_lstm} and~\ref{tab:best_configs_transformer} summarize the selected hyperparameters for each architecture,
including input window length, temporal resolution, and model complexity.
For convolutional models, the input resolution was always fixed at the full 288 measurements per day.
For the Transformer architecture, we made the decision to use longer input windows, as the performance on longer
For the Transformer architecture, I have made the decision to use longer input windows, as the performance on longer
input windows was competitive to the best measured at 40 days, but includes much more context information.
\begin{table}[htbp]
@@ -284,7 +288,7 @@ All following prediction were made with the best model for each architecture as
\label{fig:results_rq2_regular_cycle_predictions_example}
\end{figure}
To visualize potential patterns in the predictions, we will show some representative prediction plots.
To visualize potential patterns in the predictions, I will show some representative prediction plots.
All models have a similar predictive behaviour throughout both the regular and irregular cycles.
Figure~\ref{fig:results_rq2_regular_cycle_predictions_example} shows, that all models come very close to the ground truth
@@ -427,13 +431,13 @@ The Convolutional LSTM shows the lowest overall MSE, while the Transformer perfo
The LSTM remains the most effective on non-fertile days.
Baselines degrade more sharply in this group.
The full table with MSE and MAE for all models can be found in the appendix, Table~\ref{tab:regular_vs_irregular_fertility_results}.
The full table with MSE and MAE for all models can be found in the appendix Table~\ref{tab:regular_vs_irregular_fertility_results}.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_regular_cycles}
\caption{
MSE performance trends across increasing historical data availability for fertility and ovulation-over targets on regular cycles.
MSE performance trends across increasing historical data availability for fertility and OV-over targets on regular cycles.
}
\label{fig:results_performance_on_regular_cycles}
\end{figure}
@@ -442,7 +446,7 @@ The full table with MSE and MAE for all models can be found in the appendix, Tab
\centering
\includegraphics[width=0.8\textwidth]{resources/figures/results/model_results_on_irregular_cycles}
\caption{
MSE performance trends across increasing historical data availability for fertility and ovulation-over targets on irregular cycles.
MSE performance trends across increasing historical data availability for fertility and OV-over targets on irregular cycles.
}
\label{fig:results_performance_on_irregular_cycles}
\end{figure}
@@ -470,32 +474,36 @@ as indicated by the wider deviations from the trend lines.
% -------------------------------------------------------------
\subsection{Objective 4: Evaluating Use-Case Scenarios}\label{subsec:rq4_use_case_evaluations}
\subsection{Objective 4: Evaluating Use Case Scenarios}\label{subsec:rq4_use_case_evaluations}
We evaluated the use-case scenarios described in Section~\ref{subsubsec:practical_use_cases} using the algorithms
I have evaluated the use case scenarios described in Section~\ref{subsubsec:practical_use_cases} using the algorithms
in Section~\ref{subsubsec:use_case_evaluation} across multiple fertility thresholds.
Each evaluation used the same test set of 100 users (100 user-years) and was repeated for 200 iterations
to get statistically more meaningful results.
All results represent the means over all runs.
\subsubsection{Contraception Use-Case Results}\label{subsubsec:use_case_contraception_results}
\subsubsection{Contraception Use Case Results}\label{subsubsec:use_case_contraception_results}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/contraception_use_case_results_by_fertility_threshold}
\caption{
Contraception use-case evaluation study results by fertility threshold.
Contraception use case evaluation study results by fertility threshold.
}
\label{fig:results_contraception_use_case_results_by_fertility_threshold}
\end{figure}
Figure~\ref{fig:results_contraception_use_case_results_by_fertility_threshold} shows the outcomes of
the contraception use-case across fertility thresholds for each model architecture, along with the baselines.
the contraception use case across fertility thresholds for each model architecture, along with the baselines.
The four evaluated metrics are: number of pregnancies, number of intercourse events,
number of correct denials (i.e., appropriately flagged fertile days),
and number of incorrect denials (i.e., fertile days incorrectly flagged as infertile).
\paragraph{Potential Pregnancies.}
Potential pregnancies represent days that led to a pregnancy for a user.
This means, that each user could have multiple potential pregnancies.
For a per-user pregnancy metric, see normalized values below (pregnant users per 100-user year).
All trained models start with very low pregnancy rates (Transformer: 0, LSTM: 4)
and then increase roughly linearly with the threshold,
reaching between 99 (Convolutional Transformer) and 122 (Transformer) at the highest setting.
@@ -504,7 +512,7 @@ The Transformer starts with the lowest pregnancy rate but rises steeply,
ending with the highest number of pregnancies at threshold 0.1.
By comparison, the baselines show much higher pregnancy rates throughout,
starting between 103 (user-mean) and 150 (population-mean) and increasing by about 50 pregnancies over the range of thresholds.
The control group, which does not use any predictions shows very high potential pregnancy rates at 274 for 100 user years.
The control group, which does not use any predictions shows very high potential pregnancy rates at 274 for 100 user-years.
Note, that the thresholds do not alter the control group, as intercourse can happen on any day by design.
\paragraph{Intercourse Events.}
@@ -512,7 +520,7 @@ All trained models except the LSTM begin with low values of about 1000 intercour
The LSTM starts much higher, around 3000, and increases sharply, converging toward approximately 6100.
All trained model variants show curved growth resembling logarithmic convergence, approaching the same upper range.
In contrast, the baselines start near 6000 events and converge toward the same levels as the trained models at higher thresholds.
Here, the control group allows for 7610 intercourse events, which is also the maximum for 100 user years by design,
Here, the control group allows for 7610 intercourse events, which is also the maximum for 100 user-years by design,
since no denials happen.
\paragraph{Correct Denials.}
@@ -533,7 +541,7 @@ while the population-mean baseline instead decreases from 1102 to around 879.
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/contraception_use_case_pregnancy_statistics}
\caption{
Contraception use-case results: pregnancies per 100 users per year and per 1000 intercourse events, stratified by fertility threshold.
Contraception use case results: pregnancies per 100 users per year and per 1000 intercourse events, stratified by fertility threshold.
}
\label{fig:results_contraception_use_case_pregnancy_statistics}
\end{figure}
@@ -551,39 +559,40 @@ The Transformer, while being the best model for small thresholds, becomes worse
the LSTM for the maximum tested value (0.1).
\paragraph{Summary.}
In the contraception use-case, the trained models achieve substantially lower pregnancy rates than the baselines,
In the contraception use case, the trained models achieve substantially lower pregnancy rates than the baselines,
though at the cost of fewer intercourse opportunities and more denials, both correct and incorrect.
This results in less pregnancies for both 100 users for a year and per 1000 intercourse events for all trained models.
This results in fewer pregnancies for both 100 users for a year and per 1000 intercourse events for all trained models.
Within the trained models, Transformers and their convolutional variants perform best overall, while LSTMs lag behind across most metrics.
There is an edge for the non-convolutional models for smaller thresholds that switches for higher values.
\subsubsection{Pregnancy Use-Case Results}\label{subsubsec:use_case_pregnancy_results}
\subsubsection{Pregnancy Use Case Results}\label{subsubsec:use_case_pregnancy_results}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/pregnancy_use_case_results_by_fertility_threshold}
\caption{
Contraception use-case evaluation study results by fertility threshold.
Pregnancy use case evaluation study results by fertility threshold.
}
\label{fig:results_pregnancy_use_case_results_by_fertility_threshold}
\end{figure}
Figure~\ref{fig:results_pregnancy_use_case_results_by_fertility_threshold} presents the results for the pregnancy use-case
Figure~\ref{fig:results_pregnancy_use_case_results_by_fertility_threshold} presents the results for the pregnancy use case
across varying fertility thresholds, comparing all model architectures and baselines.
The four reported metrics are: number of pregnancies, number of intercourse events, number of correct deferrals (i.e., correctly flagged infertile days),
and number of incorrect deferrals (i.e., fertile days incorrectly flagged as infertile).
The overall pattern mirrors the contraception use-case, but the optimization goal is reversed:
The overall pattern mirrors the contraception use case, but the optimization goal is reversed:
here, a higher number of pregnancies is desirable.
\paragraph{Potential Pregnancies.}
Trained models achieve substantially higher pregnancy rates than the baselines.
Trained models achieve substantially higher potential pregnancy rates than the baselines.
They begin at around 275 pregnancies for low thresholds and decline linearly to between 157 and 176 at higher thresholds.
By contrast, the baselines start much lower (user-mean: 159, last-cycle: 139, population-mean: 117) and also decrease linearly,
reaching 119, 102, and 71 pregnancies, respectively.
This places all trained models well above baseline performance across the full threshold range.
The control group results in 261 potential pregnancies for 100 user years.
The control group results in 261 potential pregnancies for 100 user-years.
\paragraph{Intercourse Events.}
Trained models start with high numbers of intercourse events, near 7000, except for the LSTM which begins lower at 4252.
@@ -610,7 +619,7 @@ but converging to the weakest performance among trained models at higher thresho
\centering
\includegraphics[width=0.9\textwidth]{resources/figures/results/pregnancy_use_case_pregnancy_statistics}
\caption{
Contraception use-case evaluation study results by fertility threshold.
Pregnancy use case evaluation study results by fertility threshold.
}
\label{fig:results_pregnancy_use_case_pregnancy_statistics}
\end{figure}
@@ -626,9 +635,9 @@ The control group attains 90 pregnant users per 100 for a year, which is on par
The efficiency is much lower than both trained and baseline models with 36 pregnancies per 1000 intercourse events.
\paragraph{Summary.}
In the pregnancy use-case, trained models clearly outperform baselines in terms of pregnancy rates,
In the pregnancy use case, trained models clearly outperform baselines in terms of pregnancy rates,
though at the expense of more intercourse events and higher incorrect deferrals.
This results in less pregnancies per 1000 intercourse events for the trained models on smaller thresholds.
This results in fewer pregnancies per 1000 intercourse events for the trained models on smaller thresholds.
However, for 100 user-years, the pregnancy rates are noticeably higher than the baselines.
The LSTM underperforms relative to the Transformer-based architectures across most metrics,
while the Transformer itself exhibits strong performance at low thresholds but deteriorates more rapidly with increasing thresholds.
@@ -696,7 +705,7 @@ There is an edge for the non-convolutional models for smaller thresholds that sw
\item Variance of predictions grow with longer context lengths.
\end{itemize}
\noindent\textbf{Objective 4: Use-case Evaluations.}
\noindent\textbf{Objective 4: Use Case Evaluations.}
\begin{itemize}
\item \emph{Contraception:} Trained models cut pregnancies sharply vs baselines,
at the cost of more denials and fewer intercourse opportunities.