further work on methodology
This commit is contained in:
@@ -564,18 +564,117 @@ The trainings were performed on the scientific compute cluster of Leipzig Univer
|
||||
Due to the heterogeneity of both the models and the trainings, a dynamic batch size algorithm was used, that estimated
|
||||
the batch size dynamically during training to optimize the resource usage.
|
||||
The learning rate was scaled linearly with the batch size to allow for equivalent convergence behaviour~\cite{goyal_accurate_2018}
|
||||
The batch size was capped at 2048 to avoid OOM errors on the system.
|
||||
The batch size was capped at 2048 to avoid OOM errors during data preparation.
|
||||
%TODO sources, for square rule
|
||||
|
||||
\subsection{Evaluation}\label{subsec:evaluation}
|
||||
|
||||
A variety of evaluation metrics have been defined, to best capture each aspect of the performance of an ovulation prediction.
|
||||
The evaluation is divided into two subgroups for the two prediction targets, fertility and past-ovulation indicator.
|
||||
% TODO: review
|
||||
To meaningfully compare model performance, we define a set of metrics that capture both overall accuracy and behavior at key points in the prediction sequence.
|
||||
This includes metrics for different temporal segments, enabling a more detailed understanding of model strengths and limitations.
|
||||
|
||||
\subsubsection{Evaluation Metrics}\label{subsubsec:evaluation_metrics}
|
||||
|
||||
The base metric used for all categories is the mean absolute error (MAE),
|
||||
which describes the average absolute deviation of the prediction from the target value,
|
||||
and is defined as:
|
||||
|
||||
\begin{align}
|
||||
\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right|
|
||||
\end{align}
|
||||
|
||||
MAE was selected for its intuitive interpretability.
|
||||
In particular, the fertility probability target lends itself well to an absolute error interpretation,
|
||||
making MAE a natural choice for evaluating prediction accuracy.
|
||||
|
||||
To enable a more nuanced comparison of model performance,
|
||||
we complement the overall MAE with targeted evaluations at biologically relevant subregions of the prediction sequence.
|
||||
Rather than relying solely on the aggregate error across the full time series,
|
||||
we compute MAE for specific intervals—such as before, during, and after the fertile window,
|
||||
as well as for individual days relative to ovulation.
|
||||
|
||||
As discussed in Section~\ref{subsubsec:physiological_signs},
|
||||
the five days leading up to ovulation are the most relevant for both contraception and natural family planning (NFP).
|
||||
These days are therefore evaluated individually.
|
||||
|
||||
This decomposition allows us to identify localized strengths and weaknesses in different models—
|
||||
for example, a model may accurately capture the onset of fertility but underperform closer to ovulation.
|
||||
|
||||
It also facilitates a clearer comparison of false-positive tendencies,
|
||||
i.e., days that are not fertile but are misclassified as such by the model.
|
||||
These insights are important for understanding practical limitations
|
||||
and choosing appropriate models depending on the intended clinical or user-facing application.
|
||||
|
||||
In addition, we stratify each metric by the number of past cycles available as context,
|
||||
to assess how model performance varies with different amounts of historical information.
|
||||
This aspect is especially relevant for users with irregular cycles,
|
||||
where models may learn user-specific patterns to different degrees,
|
||||
potentially leading to divergent prediction behavior.
|
||||
|
||||
Tables~\ref{tab:fertility_mae_metrics} and~\ref{tab:ov_over_mae_metrics} summarize the metrics used for evaluating the fertility probability and ovulation-over targets, respectively.
|
||||
|
||||
\begin{table}[htbp]
|
||||
\centering
|
||||
\setlength{\tabcolsep}{10pt} % spacing between columns
|
||||
\renewcommand{\arraystretch}{1.3} % spacing between rows
|
||||
\begin{tabular}{@{}p{0.35\linewidth}p{0.60\linewidth}@{}}
|
||||
\toprule
|
||||
\textbf{Metric Name} & \textbf{Description} \\
|
||||
\midrule
|
||||
Fertility Overall & Mean absolute error (MAE) over the entire sequence. \\
|
||||
Pre-Fertility & MAE before the fertile window. \\
|
||||
Post-Fertility & MAE after the fertile window. \\
|
||||
During-Fertility & MAE during the fertile phase. \\
|
||||
First fertile day & MAE on the first fertile day (start of fertile phase). \\
|
||||
Second fertile day & MAE on the second fertile day. \\
|
||||
Third fertile day & MAE on the third fertile day. \\
|
||||
Fourth fertile day & MAE on the fourth fertile day. \\
|
||||
Fifth fertile day & MAE on the fifth fertile day. \\
|
||||
Sixth fertile day & MAE on the sixth fertile day (end of fertile phase). \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Evaluation metrics of the fertility probability target based on mean absolute error (MAE) at various points across the predicted fertility window.}
|
||||
\label{tab:fertility_mae_metrics}
|
||||
\end{table}
|
||||
|
||||
\begin{table}[htbp]
|
||||
\centering
|
||||
\setlength{\tabcolsep}{10pt} % spacing between columns
|
||||
\renewcommand{\arraystretch}{1.3} % spacing between rows
|
||||
\begin{tabular}{@{}p{0.35\linewidth}p{0.60\linewidth}@{}}
|
||||
\toprule
|
||||
\textbf{Metric Name} & \textbf{Description} \\
|
||||
\midrule
|
||||
OV-Over Overall & Mean absolute error (MAE) over the entire sequence. \\
|
||||
Pre-OV & MAE before the ovulation. \\
|
||||
Post-OV & MAE after the ovulation. \\
|
||||
Four days before & MAE four days before the ovulation (beginning of fertile phase). \\
|
||||
Three days before & MAE three days before the ovulation. \\
|
||||
Two days before & MAE two days before the ovulation. \\
|
||||
One day before & MAE one day before the ovulation. \\
|
||||
Day of OV & MAE on the day of the ovulation. \\
|
||||
One day after & MAE one day after the ovulation (end of fertile phase). \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Evaluation metrics of the ovulation-over target based on mean absolute error (MAE) at various points across the predicted fertility window.}
|
||||
\label{tab:ov_over_mae_metrics}
|
||||
\end{table}
|
||||
|
||||
\subsubsection{Baseline Comparisons}\label{subsubsec:baseline_comparisons}
|
||||
|
||||
To place model results into context, a set of simple baseline models was established.
|
||||
These models rely on minimal assumptions and serve to evaluate whether the more complex models actually learn meaningful patterns beyond basic cycle regularities.
|
||||
|
||||
Three types of baselines are considered:
|
||||
\begin{itemize}
|
||||
\item \textbf{Population-Mean OV Day}: predicts the mean ovulation day across the full population.
|
||||
\item \textbf{User-Mean OV Day}: predicts the user’s average ovulation day; uses the population mean for the first cycle.
|
||||
\item \textbf{Previous OV Day}: predicts the ovulation day from the previous cycle; uses the population mean for the first cycle.
|
||||
\end{itemize}
|
||||
|
||||
These baselines also help assess performance on regular menstrual patterns, something often overlooked in related work (see Section~\ref{sec:related_work}).
|
||||
In many cases, it remains unclear whether proposed models genuinely outperform such simple heuristics.
|
||||
|
||||
\subsection{Ethical Considerations}\label{subsec:ethical_considerations}
|
||||
|
||||
This study was conducted using pseudonymized data collected in accordance with the terms of service and privacy policy of the data provider, VivoSensMedical GmbH (Leipzig, Germany).
|
||||
|
||||
Reference in New Issue
Block a user