Merge branch 'main' of gitlab.com:blankinator/master-thesis

thesis/main.tex

@@ -8,6 +8,7 @@
 % Packages
 \usepackage{amsmath}
 \usepackage{array}
+\usepackage{amsfonts}
 \usepackage[a4paper, margin=1in]{geometry}
 
 \usepackage{graphicx}
@@ -17,7 +18,6 @@
 \usepackage[style=ieee, backend=biber]{biblatex}
 \addbibresource{../main.bib}
 \usepackage{booktabs}
-\usepackage{amsfonts}
 \usepackage{pdflscape}
 \usepackage{adjustbox}
 

thesis/sections/discussion.tex

@@ -7,11 +7,21 @@
 In this study, we investigated the performance of different machine learning architectures on the task of fertility prediction,
 with the aim to find a model that performs well for natural family planning and natural contraception on regular and irregular cycles.
 
-Our goal was to
 
 Based on an extensive real-world database and established model architectures for timeseries analysis,
-we expect to outperform both rule-based baselines and related studies.
-We think, that for regular cycles, the performance difference will be lower than
+we expect our models to outperform the rule-based baselines.
+We think, that for regular cycles, the performance difference will be lower than irregular cycles,
+as the baseline models have no way of adapting to irregularities.
+
+In general, we expect the transformer based model to outperform the LSTM basd models, as they have proven to be
+a more effective for time-series analysis tasks especially for longer sequences.
+We also expect to find similar performance on irregular cycles compared to regular cycles,
+if the temperature is a reliable predictive indicator for the ovulation.
+If the performance on irregular cycles is significantly worse, and the predicted fertility curves see no upward trend around
+the actual fertility curves, we have no reason to believe that there is any predictive value in the temperature as is.
+
+Results do not show any clear indication that the temperature contains any patterns useful for the prediction of
+fertility or the ovulation.
 
 
 % talk about whether bbt / temperature can be used for such a task, discuss bbt doubt papers
@@ -20,9 +30,48 @@ We think, that for regular cycles, the performance difference will be lower than
 % explain the need for further medical interpretation of the results of either model
 
 
-\section{Future Work}\label{sec:future_work}
+The findings do not show a clear indication, that the temperature can be used as a predictive target.
+The results of irregular cycles should be significantly better to infer, that there are usable patters in the
+temperature readings before the ovulation or the fertile days.
 
-% inclusion of markers
-% better labeling process
-% more advanced models
-%
+For a practical use case of any model's prediction, a medical interpretation should be performed.
+While the results themselves can give a clear indication of both the fertility probability and whether the ovulation
+of a given cycle is already over for any given day, there a variety of external factors that should be taken into
+consideration for a direct output to the user.
+
+
+\section{Future Work}\label{sec:future_work}
+There are several directions in which this study could be extended,
+most of which were omitted due to time and resource constraints but represent valuable areas for future exploration.
+
+One major area is feature selection.
+The dataset used includes additional user-entered markers such as physiological signs (e.g., bleeding, illness, stress)
+and external events (e.g., intercourse, pregnancy tests).
+These markers were not included in the present analysis but may carry predictive value and could meaningfully improve model performance.
+Similarly, the introduction of engineered or intermediate features, derived from raw inputs, may help models better
+capture relevant patterns and temporal dependencies.
+Additionally, the target features could be modelled in a better way, as, especially for long cycles,
+there is a large imbalance of value distribution.
+If a cycle has a length of 100 days with an ovulation at day 90, only 10\% of the ovulation-over targets are one.
+The same applies to the fertility target, which will be zero throughout almost the whole sequence,
+which will make it harder for the models to learn useful information.
+
+Alternative target formulations could also be explored to better reflect the structure of the fertile window and ovulation.
+For example, instead of predicting a daily fertility probability, models could aim to identify the absolute day of ovulation,
+or estimate the time until the next (or since the last) ovulation event.
+
+As noted in Section~\ref{subsubsec:data_labeling}, ovulation labels were assigned using a
+retrospective algorithm trained on expert-annotated data.
+Any inaccuracies in this algorithm propagate directly to the supervised learning labels.
+Thus, improving prediction quality may require a newly labeled dataset—ideally combining expert review with algorithmic assistance.
+
+Finally, while the models used in this study (LSTMs and Transformers) are well-established for time-series analysis,
+they were not extensively customized.
+Future work could involve tailoring architectures more specifically to the characteristics of menstrual cycle data.
+Recent transformer variants designed for time series, such as \emph{TimeXer}~\cite{wang_timexer_2024} or \emph{MAMBA}~\cite{wang_is_2024},
+may offer improved performance.
+Alternatively, a custom architecture could be developed to better reflect the domain-specific structure of biological temperature data.
+
+Future work may incorporate more advanced hyperparameter optimization techniques,
+such as Bayesian Optimization, Genetic Algorithms, or Neural Architecture Search (NAS),
+to better explore the joint parameter space in a more efficient and principled manner.

thesis/sections/methodology.tex

@@ -438,7 +438,8 @@ training conditions across models.
 The framework is designed to allow plug-and-play experimentation with model types (e.g., LSTM, Transformer),
 and it supports automated logging, early stopping, and checkpointing.
 
-Training was conducted on a GPU cluster equipped with NVIDIA A30 GPUs using the PyTorch framework~\cite{noauthor_pytorch_nodate}.
+The trainings were performed on the scientific compute cluster of Leipzig University on NVIDIA A30 GPUs,
+using the PyTorch framework~\cite{noauthor_pytorch_nodate}.
 All experiments were implemented in Python and run with mixed precision for improved memory efficiency.
 
 A detailed technical report on the training framework is planned for future work.
@@ -543,21 +544,25 @@ Note that the same settings were used for the hybrid models, as their architectu
 We acknowledge that interactions between input and model parameters may influence final model performance,
 and our two-stage tuning procedure may miss globally optimal combinations.
 
-Future work may incorporate more advanced hyperparameter optimization techniques, such as
-Bayesian Optimization, Genetic Algorithms, or Neural Architecture Search (NAS), to better explore the
-joint parameter space in a more efficient and principled manner.
-
 \subsubsection{Training Details}\label{subsubsec:training_details}
 
-The trainings were performed on the scientific compute cluster of Leipzig University on NVIDIA A30 GPUs.
-Due to the heterogeneity of both the models and the trainings, a dynamic batch size algorithm was used, that estimated
-the batch size dynamically during training to optimize the resource usage.
-The learning rate was scaled linearly with the batch size to allow for equivalent convergence behaviour~\cite{goyal_accurate_2018}
-The batch size was capped at 2048 to avoid OOM errors during data preparation.
+Due to the heterogeneity of both the models and training configurations,
+a dynamic batch size algorithm was used to optimize resource usage by estimating the batch size during training.
+The learning rate was scaled linearly with the batch size to maintain equivalent convergence behavior~\cite{goyal_accurate_2018}.
+The maximum batch size was capped at 2048 to avoid out-of-memory errors during data preparation.
+
+All training runs employed a \emph{One Cycle Scheduler}, which has proven effective for recurrent, convolutional,
+and transformer-based models.
+This schedule begins with a warm-up phase, followed by an annealing phase, allowing the model to converge more smoothly~\cite{smith_disciplined_2018}.
+
+Optimization was performed using \emph{AdamW}, a decoupled weight decay variant of the \emph{Adam} optimizer,
+designed to correct its flawed implementation of L2 regularization~\cite{loshchilov_decoupled_2019}.
+Each model was trained for up to 30 epochs, with early stopping based on validation performance.
 
 \subsection{Evaluation}\label{subsec:evaluation}
 
-To meaningfully compare model performance, we define a set of metrics that capture both overall accuracy and behavior at key points in the prediction sequence.
+To meaningfully compare model performance, we define a set of metrics according to the research objectives,
+that capture both overall accuracy and behavior at key points in the prediction sequence as well as cover the use cases introduced in~\ref{subsubsec:practical_use_cases}.
 This includes metrics for different temporal segments, enabling a more detailed understanding of model strengths and limitations.
 
 \subsubsection{Baseline Comparisons}\label{subsubsec:baseline_comparisons}
@@ -585,6 +590,8 @@ and is defined as:
     \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|
 \end{align}
 
+where \(y_i\) is the observed value and \(\hat{y}_i\) the predicted value.
+
 MAE was selected for its intuitive interpretability.
 In particular, the fertility probability target lends itself well to an absolute error interpretation,
 making MAE a natural choice for evaluating prediction accuracy.
@@ -593,15 +600,22 @@ To complement this, we also report the mean squared error (MSE):
 \begin{align}
     \text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2
 \end{align}
+where \(y_i\) is the observed value and \(\hat{y}_i\) the predicted value.
+
 MSE penalizes larger errors more heavily than MAE, making it useful for highlighting substantial deviations.
 This is particularly relevant for model comparison, where disproportionate errors can skew performance.
 Moreover, since the fertility probability target was trained using an MSE-based loss function,
 this metric directly reflects the optimization objective.
 
+We considered including the coefficient of determination (\(R^2\)) as an evaluation metric.
+However, we found that the target windows frequently exhibited very low variance,
+a condition under which \(R^2\) becomes unstable and potentially misleading.
+As a result, we decided to exclude it from our evaluation.
+
 To enable a more nuanced comparison of model performance,
 we complement the overall error metrics with targeted evaluations at biologically relevant subregions of the prediction sequence.
 
-This decomposition allows us to identify localized strengths and weaknesses in different models—
+This decomposition allows us to identify localized strengths and weaknesses in different models,
 for example, a model may accurately capture the onset of fertility but underperform closer to ovulation.
 
 It also facilitates a clearer comparison of false-positive tendencies,
@@ -674,7 +688,7 @@ the next ovulation day is likely to be identical or very close to the previous o
 To evaluate how different algorithms handle varying degrees of cycle variability,
 we compare model performance on two distinct user groups: one with highly regular cycles and another with highly irregular cycles.
 
-These groups are defined by the standard deviation of ovulation day across cycles.
+These groups are defined by the standard deviation of the ovulation day across cycles.
 Specifically, we select the 100 users with the lowest standard deviation (regular group)
 and the 100 users with the highest standard deviation (irregular group).
 This comparison allows us to assess not only overall performance,
@@ -691,7 +705,7 @@ enabling comparability between models and providing interpretable performance me
 
 \paragraph{Contraception Use-Case:}
 For evaluating contraceptive effectiveness, we developed an algorithm inspired by the classical \emph{Pearl Index},
-initially proposed by~\citeauthor{pearl_factors_1933} in~\citeyear{pearl_factors_1933}.
+initially proposed by~\citeauthor{pearl_factors_1933} in~\citeyear{pearl_factors_1933}\cite{pearl_factors_1933}.
 
 \begin{figure}[htbp]
     \centering
@@ -759,7 +773,7 @@ For each day in a full year we count occurrences of the following states:
 \end{itemize}
 
 This classification measures both fertility prediction accuracy and the impact of incorrect deferrals.
-A model overly predicting fertility would increase pregnancy rates but negatively affect usability due to misdirected efforts.
+A model overly predicting fertility would increase potential pregnancy rates but negatively affect usability and trust due to misdirected efforts.
 
 \subsection{Ethical Considerations}\label{subsec:ethical_considerations}
 

thesis/sections/results.tex

@@ -4,18 +4,29 @@
 
 \section{Results}\label{sec:results}
 
-The previous section detailed the design and implementation of our ovulation prediction pipeline,
+We begin by comparing the overall performance of all trained models and baselines using four key evaluation metrics:
+mean absolute error (MAE), mean squared error (MSE), and the coefficient of determination (\(R^2\)) for the regression target,
+as well as the metrics for their respective sub-intervals.
+Figure~\ref{fig:results_performance_overview_by_model_type} provides an overview of these metrics across all model types.
+
+Overall, transformer-based models consistently outperformed LSTM variants and baseline methods across most evaluation criteria.
+Among the baselines, [e.g., "the rule-based method"] showed the weakest performance,
+while the [e.g., "windowed logistic regression"] performed competitively in certain contexts.
+Differences across models were most pronounced in MSE and \(R^2\),
+indicating that advanced architectures better captured higher-order dynamics and reduced large prediction errors.
+
+The previous section detailed the design and implementation of our fertility prediction pipeline,
 including data preprocessing, feature engineering, input encoding, and the development of several deep learning architectures.
 We now present the results of our evaluation, focusing on the predictive accuracy of the proposed models across different temporal resolutions,
-cycle types, and user histories.
+cycle types and use cases.
 Model performance is assessed using both overall metrics and biologically targeted subintervals,
 allowing for a nuanced comparison of approaches and their practical relevance to real-time fertility forecasting.
+Additionally, model performance is compared to the three baseline models introduced.
 
-% provide information about the training behaviour and statistic of the different models
+% provide information about the training behaviour and statistic of the different models??
 
 \subsection{Overall Model Performance Across Architectures}\label{subsec:overall_model_performance_across_architectures}
 
-
 \begin{figure}
     \centering
     \includegraphics[width=0.9\textwidth]{resources/figures/results/model_performance_overview}
@@ -47,11 +58,10 @@ allowing for a nuanced comparison of approaches and their practical relevance to
 \end{landscape}
 
 
-
 % show why I selected the individual input configs for model config training
 % selected by best mse fertility, use 2nd best, as it provides basically the same performance, but more input data for more complex model configs
 
-\subsubsection{Performance Across Fertile Window}\label{subsubsec:fert_performance_across_fertile_window}
+\subsubsection{Fertility Probability Prediction}\label{subsubsec:fertility_probability_prediction}
 
 \subsubsection{Impact of Input Resolution}\label{subsubsec:fert_impact_of_input_resolution}
 
@@ -72,7 +82,7 @@ allowing for a nuanced comparison of approaches and their practical relevance to
 
 \subsubsection{Comparison with Baselines}\label{subsubsec:fert_comparison_with_baselines}
 
-\subsection{Ovulation-Over Prediction Accuracy}\label{subsec:ov_over_prediction_accuracy}
+\subsection{Ovulation-Over Prediction}\label{subsubsec:ov_over_prediction}
 
 \subsubsection{Performance around Ovulation}\label{subsubsec:ov_over_performance_around_ovulation}
 
@@ -94,5 +104,4 @@ allowing for a nuanced comparison of approaches and their practical relevance to
 
 \subsubsection{Pregnancy Use-Case Results}\label{subsubsec:use_case_pregnancy_results}
 
-
 \subsection{Summary of Key Findings}\label{subsec:summary_of_key_findings}