From f9b1b63a128e97ecaccdf6439d9be7a6943e5ef3 Mon Sep 17 00:00:00 2001 From: Alex Blank <38751347+blankinator@users.noreply.github.com> Date: Tue, 12 Aug 2025 19:00:08 +0000 Subject: [PATCH] fixes --- ...tm_ov_over_results_by_input_resolution.png | Bin 397930 -> 397930 bytes thesis/sections/discussion.tex | 53 ++-- thesis/sections/results.tex | 238 ++++++++++-------- 3 files changed, 158 insertions(+), 133 deletions(-) diff --git a/thesis/resources/figures/results/lstm_ov_over_results_by_input_resolution.png b/thesis/resources/figures/results/lstm_ov_over_results_by_input_resolution.png index da1abc8f02f1e371ec128f40d191fa1b06600e60..4cdc5055065e3152ba4f0d72997617d7ea05db19 100644 GIT binary patch delta 61 zcmaF0MB>#Fi3y%^#yScaB_##LR{Hw6i6sR&`6W4-NqYH3>H2A^Te}+5Thkd^)0tY+ RnOoCYTGLs#rn4~%0RY~{75@ML delta 61 zcmaF0MB>#Fi3y%^2098EB_##LR{Hw6i6sR&`6W4-NqYH3>H1TOB8nQ*Thkd^)0tY+ RnOoCYTGLs#rn4~%0RZ2776JeO diff --git a/thesis/sections/discussion.tex b/thesis/sections/discussion.tex index 759daf0..d9b2cb7 100644 --- a/thesis/sections/discussion.tex +++ b/thesis/sections/discussion.tex @@ -4,24 +4,24 @@ \section{Discussion}\label{sec:discussion} -In this study, we investigated the performance of different machine learning architectures on the task of fertility prediction, -with the aim to find a model that performs well for natural family planning and natural contraception on regular and irregular cycles. - - -Based on an extensive real-world database and established model architectures for timeseries analysis, -we expect our models to outperform the rule-based baselines. -We think, that for regular cycles, the performance difference will be lower than irregular cycles, -as the baseline models have no way of adapting to irregularities. - -In general, we expect the transformer based model to outperform the LSTM basd models, as they have proven to be -a more effective for time-series analysis tasks especially for longer sequences. -We also expect to find similar performance on irregular cycles compared to regular cycles, -if the temperature is a reliable predictive indicator for the ovulation. -If the performance on irregular cycles is significantly worse, and the predicted fertility curves see no upward trend around -the actual fertility curves, we have no reason to believe that there is any predictive value in the temperature as is. - -Results do not show any clear indication that the temperature contains any patterns useful for the prediction of -fertility or the ovulation. +%In this study, we investigated the performance of different machine learning architectures on the task of fertility prediction, +%with the aim to find a model that performs well for natural family planning and natural contraception on regular and irregular cycles. +% +% +%Based on an extensive real-world database and established model architectures for timeseries analysis, +%we expect our models to outperform the rule-based baselines. +%We think, that for regular cycles, the performance difference will be lower than irregular cycles, +%as the baseline models have no way of adapting to irregularities. +% +%In general, we expect the transformer based model to outperform the LSTM basd models, as they have proven to be +%a more effective for time-series analysis tasks especially for longer sequences. +%We also expect to find similar performance on irregular cycles compared to regular cycles, +%if the temperature is a reliable predictive indicator for the ovulation. +%If the performance on irregular cycles is significantly worse, and the predicted fertility curves see no upward trend around +%the actual fertility curves, we have no reason to believe that there is any predictive value in the temperature as is. +% +%Results do not show any clear indication that the temperature contains any patterns useful for the prediction of +%fertility or the ovulation. % talk about whether bbt / temperature can be used for such a task, discuss bbt doubt papers @@ -30,14 +30,17 @@ fertility or the ovulation. % explain the need for further medical interpretation of the results of either model -The findings do not show a clear indication, that the temperature can be used as a predictive target. -The results of irregular cycles should be significantly better to infer, that there are usable patters in the -temperature readings before the ovulation or the fertile days. +% transformer can deal with more data and does also perform better for some cases +% lstm outperforms -For a practical use case of any model's prediction, a medical interpretation should be performed. -While the results themselves can give a clear indication of both the fertility probability and whether the ovulation -of a given cycle is already over for any given day, there a variety of external factors that should be taken into -consideration for a direct output to the user. +%The findings do not show a clear indication, that the temperature can be used as a predictive target. +%The results of irregular cycles should be significantly better to infer, that there are usable patters in the +%temperature readings before the ovulation or the fertile days. +% +%For a practical use case of any model's prediction, a medical interpretation should be performed. +%While the results themselves can give a clear indication of both the fertility probability and whether the ovulation +%of a given cycle is already over for any given day, there a variety of external factors that should be taken into +%consideration for a direct output to the user. \section{Future Work}\label{sec:future_work} diff --git a/thesis/sections/results.tex b/thesis/sections/results.tex index e98bed8..7a2c182 100644 --- a/thesis/sections/results.tex +++ b/thesis/sections/results.tex @@ -3,77 +3,39 @@ \section{Results}\label{sec:results} - -We begin by comparing the overall performance of all trained models and baselines using four key evaluation metrics: -mean absolute error (MAE), mean squared error (MSE), and the coefficient of determination (\(R^2\)) for the regression target, -as well as the metrics for their respective sub-intervals. -Figure~\ref{fig:results_performance_overview_by_model_type} provides an overview of these metrics across all model types. - -Overall, transformer-based models consistently outperformed LSTM variants and baseline methods across most evaluation criteria. -Among the baselines, [e.g., "the rule-based method"] showed the weakest performance, -while the [e.g., "windowed logistic regression"] performed competitively in certain contexts. -Differences across models were most pronounced in MSE and \(R^2\), -indicating that advanced architectures better captured higher-order dynamics and reduced large prediction errors. - -The previous section detailed the design and implementation of our fertility prediction pipeline, -including data preprocessing, feature engineering, input encoding, and the development of several deep learning architectures. -We now present the results of our evaluation, focusing on the predictive accuracy of the proposed models across different temporal resolutions, -cycle types and use cases. -Model performance is assessed using both overall metrics and biologically targeted subintervals, -allowing for a nuanced comparison of approaches and their practical relevance to real-time fertility forecasting. -Additionally, model performance is compared to the three baseline models introduced. - +% +%We begin by comparing the overall performance of all trained models and baselines using four key evaluation metrics: +%mean absolute error (MAE) and mean squared error (MSE), as well as the metrics for their respective sub-intervals. +% +%Overall, transformer-based models consistently outperformed LSTM variants and baseline methods across most evaluation criteria. +%Among the baselines, [e.g., "the rule-based method"] showed the weakest performance, +%while the [e.g., "windowed logistic regression"] performed competitively in certain contexts. +%Differences across models were most pronounced in MSE and \(R^2\), +%indicating that advanced architectures better captured higher-order dynamics and reduced large prediction errors. +% +%The previous section detailed the design and implementation of our fertility prediction pipeline, +%including data preprocessing, feature engineering, input encoding, and the development of several deep learning architectures. +%We now present the results of our evaluation, focusing on the predictive accuracy of the proposed models across different temporal resolutions, +%cycle types and use cases. +%Model performance is assessed using both overall metrics and biologically targeted subintervals, +%allowing for a nuanced comparison of approaches and their practical relevance to real-time fertility forecasting. +%Additionally, model performance is compared to the three baseline models introduced. +% % provide information about the training behaviour and statistic of the different models?? \subsection{Overall Model Performance Across Architectures}\label{subsec:overall_model_performance_across_architectures} -\begin{figure} - \centering - \includegraphics[width=0.9\textwidth]{resources/figures/results/model_performance_overview} - \caption{Overview of the performances of all model types including the baselines on 4 selected performance metrics.} - \label{fig:results_performance_overview_by_model_type} -\end{figure} - \subsection{Fertility Probability Prediction Accuracy}\label{subsec:fertility_probability_precition_accuracy} -\begin{landscape} - \begin{table}[ht] - \centering - \caption{Model comparison for fertility prediction using MAE and MSE} - \begin{adjustbox}{max width=\linewidth} - \begin{tabular}{lllcccccc} - \toprule - \textbf{Model} & \textbf{Window} & \textbf{Daily} & - \textbf{MAE$_{fert}$} & \textbf{MAE$_{fert,during}$} & \textbf{MAE$_{fert,non}$} & - \textbf{MSE$_{fert}$} & \textbf{MSE$_{fert,during}$} & \textbf{MSE$_{fert,non}$} \\ - \midrule - ModelA & 7 & Yes & 0.67 & 0.59 & 0.73 & 0.89 & 0.82 & 0.95 \\ - ModelB & 14 & No & 0.65 & 0.58 & 0.71 & 0.87 & 0.80 & 0.93 \\ -% More rows... - \bottomrule - \end{tabular} - \end{adjustbox} - \label{tab:fertility_comparison} - \end{table} -\end{landscape} - - % show why I selected the individual input configs for model config training % selected by best mse fertility, use 2nd best, as it provides basically the same performance, but more input data for more complex model configs \subsubsection{Fertility Probability Prediction}\label{subsubsec:fertility_probability_prediction} \subsubsection{Impact of Input Window Length}\label{subsubsec:fert_impact_of_historical_context} -\begin{figure} - \centering - \includegraphics[width=1.0\textwidth]{resources/figures/results/lstm_fertility_results_by_input_length} - \caption{Performance of LSTM models with different input window lengths and fixes input resolutions for the fertility probability target} - \label{fig:results_performance_lstm_fertility_input_window_length} -\end{figure} - \begin{landscape} \begin{table} -% \begin{tabular}{lrrrrrrr} + \small \begin{tabularx}{\linewidth}{l*{6}{X}} \toprule \multirow{2}{*}{Input-Length in Days} & \multicolumn{3}{c}{MAE} & \multicolumn{3}{c}{MSE} \\ @@ -95,9 +57,9 @@ Additionally, model performance is compared to the three baseline models introdu 160 & \underline{0.0411} & 0.0882 & \underline{0.0218} & 0.0045 & 0.0116 & \underline{0.0016} \\ \midrule \multicolumn{7}{c}{\textbf{Convolution-LSTM}} \\ \midrule - 10 & 0.0435 & \underline{0.0954} & 0.0224 & 0.0050 & 0.0133 & 0.0017 \\ - 20 & \underline{0.0414} & 0.0996 & \underline{0.0179} & 0.0050 & 0.0145 & \underline{\textbf{0.0013}} \\ - 40 & 0.0394 & 0.0911 & 0.0184 & \underline{0.0045} & \underline{0.0122} & 0.0014 \\ + 10 & 0.0435 & 0.0954 & 0.0224 & 0.0050 & 0.0133 & 0.0017 \\ + 20 & 0.0414 & 0.0996 & \underline{0.0179} & 0.0050 & 0.0145 & \underline{\textbf{0.0013}} \\ + 40 & \underline{0.0394} & \underline{0.0911} & 0.0184 & \underline{0.0045} & \underline{0.0122} & 0.0014 \\ 80 & 0.0481 & 0.1005 & 0.0278 & 0.0054 & 0.0146 & 0.0019 \\ 160 & 0.0505 & 0.1067 & 0.0290 & 0.0060 & 0.0166 & 0.0020 \\ \midrule @@ -108,26 +70,33 @@ Additionally, model performance is compared to the three baseline models introdu 80 & 0.0414 & 0.0932 & 0.0210 & 0.0048 & 0.0128 & 0.0018 \\ 160 & 0.0523 & 0.1018 & 0.0337 & 0.0059 & 0.0148 & 0.0026 \\ \bottomrule -% \end{tabular} \end{tabularx} - \caption{Evaluation Metrics for the Fertility-Probability target for each model architecture across different input lenghts. - Underlined values indicate the best value per metric for a model and bold and underlined indicate global best values for a metric.} - \label{tab:fertility_results} + \caption{Evaluation Metrics for the Fertility-Probability Target across Different Model Architectures and Input Lengths on a fixed Input Resolution of 12 Values per Day. + \underline{Underlined} values represent the best value for each metric within a model. + \textbf{\underline{Bold + Underlined}} values represent the global best values across all models for a given metric.} + \label{tab:fertility_results_by_window_length} \end{table} \end{landscape} -For the LSTM model, the error metrics show a small improvement with larger input window size. -Also, the MSE metric is numerically smaller than the MAE, which suggests, that the errors are rather small and tightly clustered with few large outliers. -For the MSE metric, the non-fertile phase seem to behave inversely proportional to the other both the fertile phase error and the overall error. -This is not the case for the MAE metric. -Overall, the improvement over a growing input window size is very small +Table~\ref{tab:fertility_results_by_window_length} presents the performance of different model architectures across various metrics and input window lengths. -\begin{figure} - \centering - \includegraphics[width=1.0\textwidth]{resources/figures/results/transformer_fertility_results_by_input_length} - \caption{Performance of Transformer models with different input window lengths and fixes input resolutions for the fertility probability target} - \label{fig:results_performance_transformer_fertility_input_window_length} -\end{figure} +It is evident that for both the LSTM and Transformer models, \textbf{longer input windows generally yield better performance}. +Specifically, the LSTM achieves the best results for four out of six metrics with the 160-day window. +The only exception is the error metrics for non-fertile days, where shorter input windows perform better. +Notably, the LSTM achieves the \textbf{global best performance} for four out of six metrics, including MAE and MSE on non-fertile and fertile days. + +For the Transformer model, the best performance is observed with the 160-day input window in three out of six metrics. +However, for non-fertile day metrics, performance improves with shorter input windows. +It is consistently outperformed by the LSTM model. + +The \textbf{convolutional models} tend to perform best with \textbf{medium-length input windows} (20 to 40 days). +The \textbf{convolutional LSTM} shows its best performance with a 40-day window for most metrics, +except for non-fertile day errors, where the 20-day window yields smaller errors. +It also achieves the \textbf{global best performance} for non-fertile day MSE with the 20-day window. +Similarly, the \textbf{convolutional Transformer} excels with 40-day windows for the overall and non-fertile-day metrics, +while the best fertile-day metrics are seen with 20-day windows. +The \textbf{convolutional Transformer} also achieves the \textbf{global best performance} for 40-day windows in both MAE and MSE\@. +In all but non-fertile day metrics, the \textbf{convolutional Transformer} outperforms the LSTM variant. \subsubsection{Impact of Input Resolution}\label{subsubsec:fert_impact_of_input_resolution} @@ -138,6 +107,44 @@ The plot shows the MSE values for three target categories: non-fertile days, ove Notably, the MSE for non-fertile days appears to behave inversely in relation to the other two metrics. However, there is no consistent upward or downward trend when changing the input window length with a fixed input resolution. +\begin{landscape} + \begin{table} + \small + \begin{tabularx}{\linewidth}{l*{6}{X}} + \toprule + \multirow{2}{*}{Values Per Day} & \multicolumn{3}{c}{MAE} & \multicolumn{3}{c}{MSE} \\ + \cmidrule(r){2-4} \cmidrule(r){5-7} + & Fertility Overall & Fertile Days & Non-Fertile Days & Fertility Overall & Fertile Days & Non-Fertile Days \\ + \midrule + \multicolumn{7}{c}{\textbf{LSTM}} \\ + \midrule + 1 & 0.0471 & 0.0987 & 0.0220 & 0.0062 & 0.0145 & 0.0022 \\ + 2 & 0.0457 & 0.0962 & 0.0231 & 0.0052 & 0.0133 & 0.0016 \\ + 4 & 0.0421 & 0.0898 & 0.0223 & \underline{\textbf{0.0046}}& 0.0116 & 0.0018 \\ + 12 & 0.0421 & 0.0994 & \underline{\textbf{0.0180}}& 0.0052 & 0.0144 & \underline{\textbf{0.0013}}\\ + 24 & 0.0419 & 0.0949 & 0.0201 & 0.0050 & 0.0132 & 0.0017 \\ + 48 & \underline{\textbf{0.0402}}& 0.0929 & 0.0182 & 0.0049 & 0.0126 & 0.0018 \\ + 72 & 0.0433 & 0.0972 & 0.0216 & 0.0052 & 0.0137 & 0.0018 \\ + 288 & 0.0410 & \underline{\textbf{0.0872}}& 0.0223 & 0.0049 & \underline{\textbf{0.0110}}& 0.0025 \\ + \midrule + \multicolumn{7}{c}{\textbf{Transformer}} \\ + \midrule + 1 & 0.0541 & 0.1015 & 0.0313 & 0.0063 & 0.0146 & 0.0023 \\ + 2 & 0.0468 & 0.0946 & 0.0259 & 0.0054 & 0.0128 & 0.0021 \\ + 4 & 0.0456 & \underline{0.0891}& 0.0277 & 0.0050 & \underline{0.0115}& 0.0023 \\ + 12 & 0.0472 & 0.0949 & 0.0274 & 0.0051 & 0.0133 & 0.0018 \\ + 24 & 0.0502 & 0.0960 & 0.0323 & 0.0052 & 0.0133 & 0.0021 \\ + 48 & 0.0447 & 0.0912 & \underline{0.0254}& \underline{0.0048}& 0.0122 & 0.0017 \\ + 72 & 0.0468 & 0.0975 & 0.0257 & 0.0051 & 0.0140 & \underline{0.0014} \\ + 288 & \underline{0.0449}& 0.0937 & 0.0257 & \underline{0.0048}& 0.0128 & 0.0017 \\ + \bottomrule + \end{tabularx} + \caption{Evaluation Metrics for the Fertility-Probability Target across Different Model Architectures and Input Resolutions on a fixed Input-Window-Length of 20 Days. + \underline{Underlined} values represent the best value for each metric within a model. + \textbf{\underline{Bold + Underlined}} values represent the global best values across all models for a given metric.} + \label{tab:fertility_results_by_window_resolution} + \end{table} +\end{landscape} \begin{figure} \centering \includegraphics[width=1.0\textwidth]{resources/figures/results/transformer_fertility_results_by_input_resolution} @@ -156,8 +163,14 @@ However, there is no consistent upward or downward trend when changing the input \subsubsection{Comparison with Baselines}\label{subsubsec:fert_comparison_with_baselines} \subsection{Ovulation-Over Prediction}\label{subsubsec:ov_over_prediction} + +\subsubsection{Performance around Ovulation}\label{subsubsec:ov_over_performance_around_ovulation} + +\subsubsection{Impact of Input Window Length}\label{subsubsec:ov_over_impact_of_historical_context} + \begin{landscape} \begin{table} + \small \begin{tabularx}{\linewidth}{l*{6}{X}} \toprule \multirow{2}{*}{Input-Length in Days} & \multicolumn{3}{c}{MAE} & \multicolumn{3}{c}{MSE} \\ @@ -197,45 +210,54 @@ However, there is no consistent upward or downward trend when changing the input 160 & 0.2440 & \underline{0.1051} & 0.3117 & 0.1403 & \underline{0.0286} & 0.1946 \\ \bottomrule \end{tabularx} - \caption{Evaluation Metrics for the Ovulation-Over target for each model architecture across different input lenghts. - Underlined values indicate the best value per metric for a model and bold and underlined indicate global best values for a metric.} - \label{tab:ov_over_results} + \caption{Evaluation Metrics for the Ovulation-Over Target across Different Model Architectures and Input Lengths on a fixed Input Resolution of 12 Values per Day. + \underline{Underlined} values represent the best value for each metric within a model. + \textbf{\underline{Bold + Underlined}} values represent the global best values across all models for a given metric.} + \label{tab:ov_over_results_by_window_length} \end{table} \end{landscape} -\subsubsection{Performance around Ovulation}\label{subsubsec:ov_over_performance_around_ovulation} - -\subsubsection{Impact of Input Window Length}\label{subsubsec:ov_over_impact_of_historical_context} - -\begin{figure} - \centering - \includegraphics[width=1.0\textwidth]{resources/figures/results/lstm_ov_over_results_by_input_length} - \caption{Performance of LSTM models with different input window lengths and fixes input resolutions for the ovulation-over target} - \label{fig:results_performance_lstm_ov_over_input_window_length} -\end{figure} - -\begin{figure} - \centering - \includegraphics[width=1.0\textwidth]{resources/figures/results/transformer_ov_over_results_by_input_length} - \caption{Performance of Transformer models with different input window lengths and fixes input resolutions for the ovulation-over target} - \label{fig:results_performance_transformer_ov_over_input_window_length} -\end{figure} - \subsubsection{Impact of Input Resolution}\label{subsubsec:ov_over_impact_of_input_resolution} -\begin{figure} - \centering - \includegraphics[width=1.00\textwidth]{resources/figures/results/lstm_ov_over_results_by_input_resolution} - \caption{Performance of LSTM models with different input resolutions and fixes input window lengths for the ovulation-over target} - \label{fig:results_performance_lstm_ov_over_input_resolution} -\end{figure} -\begin{figure} - \centering - \includegraphics[width=1.0\textwidth]{resources/figures/results/transformer_ov_over_results_by_input_resolution} - \caption{Performance of transformer models with different input window lengths and fixes input resolutions for the ovulation-over target} - \label{fig:results_performance_transformer_ov_over_input_resolution} -\end{figure} +\begin{landscape} + \begin{table} + \small + \begin{tabularx}{\linewidth}{l*{6}{X}} + \toprule + \multirow{2}{*}{Values Per Day} & \multicolumn{3}{c}{MAE} & \multicolumn{3}{c}{MSE} \\ + \cmidrule(r){2-4} \cmidrule(r){5-7} + & OV-Over Overall & OV-Over Before OV & OV-Over After OV & OV-Over Overall & OV-Over Before OV & OV-Over After OV \\ + \midrule + \multicolumn{7}{c}{\textbf{LSTM}} \\ + \midrule + 1 & 0.1691 & 0.1280 & 0.1819 & 0.0914 & 0.0411 & 0.1099 \\ + 2 & 0.1431 & 0.1239 & 0.1471 & 0.0747 & 0.0442 & 0.0849 \\ + 4 & 0.1344 & 0.1130 & 0.1417 & 0.0680 & 0.0371 & 0.0807 \\ + 12 & \underline{0.1153} & \underline{\textbf{0.0745}} & 0.1312 & 0.0641 & \underline{\textbf{0.0212}} & 0.0822 \\ + 24 & 0.1192 & 0.0975 & 0.1240 & \underline{0.0633} & 0.0288 & 0.0755 \\ + 48 & 0.1463 & 0.2493 & \underline{0.0973} & 0.0768 & 0.1203 & \underline{\textbf{0.0550}} \\ + 72 & 0.1587 & 0.1113 & 0.1844 & 0.0857 & 0.0341 & 0.1126 \\ + 288 & 0.1353 & 0.1801 & 0.1169 & 0.0726 & 0.0799 & 0.0713 \\ + \midrule + \multicolumn{7}{c}{\textbf{Transformer}} \\ + \midrule + 1 & 0.1687 & 0.1582 & 0.1709 & 0.0883 & 0.0578 & 0.0996 \\ + 2 & 0.1469 & 0.0895 & 0.1676 & 0.0823 & 0.0260 & 0.1044 \\ + 4 & 0.1282 & 0.0999 & 0.1393 & 0.0704 & 0.0329 & 0.0862 \\ + 12 & 0.1204 & \underline{0.0771}& 0.1366 & 0.0690 & \underline{0.0255}& 0.0864 \\ + 24 & 0.1952 & 0.1649 & 0.2102 & 0.0904 & 0.0638 & 0.1040 \\ + 48 & 0.1137 & 0.1159 & 0.1073 & 0.0610 & 0.0480 & 0.0629 \\ + 72 & \underline{\textbf{0.1041}}& 0.1180 & \underline{\textbf{0.0947}}& \underline{\textbf{0.0585}}& 0.0538 & 0.0581 \\ + 288 & 0.1319 & 0.1879 & 0.1097 & 0.0617 & 0.0769 & \underline{0.0578}\\ + \bottomrule + \end{tabularx} + \caption{Evaluation Metrics for the Ovulation-Over Target across Different Model Architectures and Input Resolutions on a fixed Input-Window-Length of 20 Days. + \underline{Underlined} values represent the best value for each metric within a model. + \textbf{\underline{Bold + Underlined}} values represent the global best values across all models for a given metric.} + \label{tab:ov_over_results_by_resolution} + \end{table} +\end{landscape} \subsubsection{Comparison with Baselines}\label{subsubsec:ov_over_comparison_with_baselines}