further work on methodology

thesis/sections/methodology.tex

@@ -564,18 +564,117 @@ The trainings were performed on the scientific compute cluster of Leipzig Univer
 Due to the heterogeneity of both the models and the trainings, a dynamic batch size algorithm was used, that estimated
 the batch size dynamically during training to optimize the resource usage.
 The learning rate was scaled linearly with the batch size to allow for equivalent convergence behaviour~\cite{goyal_accurate_2018}
-The batch size was capped at 2048 to avoid OOM errors on the system.
+The batch size was capped at 2048 to avoid OOM errors during data preparation.
 %TODO sources, for square rule
 
 \subsection{Evaluation}\label{subsec:evaluation}
 
-A variety of evaluation metrics have been defined, to best capture each aspect of the performance of an ovulation prediction.
+% TODO: review
-The evaluation is divided into two subgroups for the two prediction targets, fertility and past-ovulation indicator.
+To meaningfully compare model performance, we define a set of metrics that capture both overall accuracy and behavior at key points in the prediction sequence.
+This includes metrics for different temporal segments, enabling a more detailed understanding of model strengths and limitations.
 
 \subsubsection{Evaluation Metrics}\label{subsubsec:evaluation_metrics}
 
+The base metric used for all categories is the mean absolute error (MAE),
+which describes the average absolute deviation of the prediction from the target value,
+and is defined as:
+
+\begin{align}
+    \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|
+\end{align}
+
+MAE was selected for its intuitive interpretability.
+In particular, the fertility probability target lends itself well to an absolute error interpretation,
+making MAE a natural choice for evaluating prediction accuracy.
+
+To enable a more nuanced comparison of model performance,
+we complement the overall MAE with targeted evaluations at biologically relevant subregions of the prediction sequence.
+Rather than relying solely on the aggregate error across the full time series,
+we compute MAE for specific intervals—such as before, during, and after the fertile window,
+as well as for individual days relative to ovulation.
+
+As discussed in Section~\ref{subsubsec:physiological_signs},
+the five days leading up to ovulation are the most relevant for both contraception and natural family planning (NFP).
+These days are therefore evaluated individually.
+
+This decomposition allows us to identify localized strengths and weaknesses in different models—
+for example, a model may accurately capture the onset of fertility but underperform closer to ovulation.
+
+It also facilitates a clearer comparison of false-positive tendencies,
+i.e., days that are not fertile but are misclassified as such by the model.
+These insights are important for understanding practical limitations
+and choosing appropriate models depending on the intended clinical or user-facing application.
+
+In addition, we stratify each metric by the number of past cycles available as context,
+to assess how model performance varies with different amounts of historical information.
+This aspect is especially relevant for users with irregular cycles,
+where models may learn user-specific patterns to different degrees,
+potentially leading to divergent prediction behavior.
+
+Tables~\ref{tab:fertility_mae_metrics} and~\ref{tab:ov_over_mae_metrics} summarize the metrics used for evaluating the fertility probability and ovulation-over targets, respectively.
+
+\begin{table}[htbp]
+    \centering
+    \setlength{\tabcolsep}{10pt} % spacing between columns
+    \renewcommand{\arraystretch}{1.3} % spacing between rows
+    \begin{tabular}{@{}p{0.35\linewidth}p{0.60\linewidth}@{}}
+        \toprule
+        \textbf{Metric Name} & \textbf{Description}                                   \\
+        \midrule
+        Fertility Overall    & Mean absolute error (MAE) over the entire sequence.    \\
+        Pre-Fertility        & MAE before the fertile window.                         \\
+        Post-Fertility       & MAE after the fertile window.                          \\
+        During-Fertility     & MAE during the fertile phase.                          \\
+        First fertile day    & MAE on the first fertile day (start of fertile phase). \\
+        Second fertile day   & MAE on the second fertile day.                         \\
+        Third fertile day    & MAE on the third fertile day.                          \\
+        Fourth fertile day   & MAE on the fourth fertile day.                         \\
+        Fifth fertile day    & MAE on the fifth fertile day.                          \\
+        Sixth fertile day    & MAE on the sixth fertile day (end of fertile phase).   \\
+        \bottomrule
+    \end{tabular}
+    \caption{Evaluation metrics of the fertility probability target based on mean absolute error (MAE) at various points across the predicted fertility window.}
+    \label{tab:fertility_mae_metrics}
+\end{table}
+
+\begin{table}[htbp]
+    \centering
+    \setlength{\tabcolsep}{10pt} % spacing between columns
+    \renewcommand{\arraystretch}{1.3} % spacing between rows
+    \begin{tabular}{@{}p{0.35\linewidth}p{0.60\linewidth}@{}}
+        \toprule
+        \textbf{Metric Name} & \textbf{Description}                                             \\
+        \midrule
+        OV-Over Overall      & Mean absolute error (MAE) over the entire sequence.              \\
+        Pre-OV               & MAE before the ovulation.                                        \\
+        Post-OV              & MAE after the ovulation.                                         \\
+        Four days before     & MAE four days before the ovulation (beginning of fertile phase). \\
+        Three days before    & MAE three days before the ovulation.                             \\
+        Two days before      & MAE two days before the ovulation.                               \\
+        One day before       & MAE one day before the ovulation.                                \\
+        Day of OV            & MAE on the day of the ovulation.                                 \\
+        One day after        & MAE one day after the ovulation (end of fertile phase).          \\
+        \bottomrule
+    \end{tabular}
+    \caption{Evaluation metrics of the ovulation-over target based on mean absolute error (MAE) at various points across the predicted fertility window.}
+    \label{tab:ov_over_mae_metrics}
+\end{table}
+
 \subsubsection{Baseline Comparisons}\label{subsubsec:baseline_comparisons}
 
+To place model results into context, a set of simple baseline models was established.
+These models rely on minimal assumptions and serve to evaluate whether the more complex models actually learn meaningful patterns beyond basic cycle regularities.
+
+Three types of baselines are considered:
+\begin{itemize}
+    \item \textbf{Population-Mean OV Day}: predicts the mean ovulation day across the full population.
+    \item \textbf{User-Mean OV Day}: predicts the user’s average ovulation day; uses the population mean for the first cycle.
+    \item \textbf{Previous OV Day}: predicts the ovulation day from the previous cycle; uses the population mean for the first cycle.
+\end{itemize}
+
+These baselines also help assess performance on regular menstrual patterns, something often overlooked in related work (see Section~\ref{sec:related_work}).
+In many cases, it remains unclear whether proposed models genuinely outperform such simple heuristics.
+
 \subsection{Ethical Considerations}\label{subsec:ethical_considerations}
 
 This study was conducted using pseudonymized data collected in accordance with the terms of service and privacy policy of the data provider, VivoSensMedical GmbH (Leipzig, Germany).