finished results and started discussion

thesis/sections/discussion.tex

@@ -5,19 +5,104 @@
 \section{Discussion}\label{sec:discussion}
 
 \paragraph{Limitations.}
-This analysis reports aggregate test-set metrics without uncertainty quantification.
-Because we did not retain per-user predictions, we cannot compute user-level bootstrap confidence intervals or perform paired significance testing.
-As a result, apparent performance differences—especially small ones—may reflect sampling variability.
-Future re-evaluation that stores per-user predictions will enable user-level bootstrapping, calibration assessment, and formal comparisons.
 
-%In this study, we investigated the performance of different machine learning architectures on the task of fertility prediction,
-%with the aim to find a model that performs well for natural family planning and natural contraception on regular and irregular cycles.
-%
-%
-%Based on an extensive real-world database and established model architectures for timeseries analysis,
-%we expect our models to outperform the rule-based baselines.
-%We think, that for regular cycles, the performance difference will be lower than irregular cycles,
-%as the baseline models have no way of adapting to irregularities.
+
+The model selection study based on the input window length, input resolution and model complexity parameters has shown,
+longer input windows and higher complexity seem to be favourable for the purposes of this study.
+The convolutional models show better performance for medium length input windows, which might be a limitation
+of the model or its convolutional layers.
+The LSTM and Transformer models also work better with medium resolution input, which suggests a sweet-spot of information.
+Too small a resolution might omit important information, while the opposite might introduce too much noise,
+that the models in the current configuration cannot handle.
+It could be worth investigating how more complex models handle more complex input data,
+as this grid-search was not possible in this study due to time and resource limitations.
+
+The study also showed, that the LSTM and Transformer base models outperform the convolutional models.
+However, this is only the case for longer input sequences.
+When isolating a 40-day input window, the convolutional models outperform the non-convolution models,
+which suggests, that there is an informational benefit in the convolution.
+However, the convolutional models seem to struggle with longer input windows,
+which clearly represent a benefit for the predictive tasks, as the normal LSTM and Transformer models perform better
+on longer input windows.
+Additional work might be necessary to investigate, whether the convolutional models need additional complexity
+or an optimized convolutional pre-processing to deal with longer input sequences.
+
+
+All prediction curve in this section have been generated with the best model selected in section~\ref{subsubsec:results_best_model_config_selection},
+i.e., the LSTM model.
+
+\begin{figure}[htbp]
+    \centering
+    \includegraphics[width=1.0\textwidth]{resources/figures/discussion/regular_cycle_fertility_prediction}
+    \caption{
+        Temperature rolling average and fertility-probability prediction for a user with a regular cycle pattern. (Values are scaled features)
+    }
+    \label{fig:discussion_regular_cycle_fertility_prediction}
+\end{figure}
+
+Figure~\ref{fig:discussion_regular_cycle_fertility_prediction} shows the prediction curve for the fertility-probability target
+for a user with a regular cycle pattern.
+It is clearly visible, that the predictions improve with each cycle, until they almost exactly match the targets.
+Another noteworthy observation is the correlation between a clear temperature drop preceding ovulation and
+the fertility rising.
+This can be seen more prominently in Figure~\ref{fig:discussion_temperature_drop_fertility_prediction}.
+The model seems to identify that this drop must be an indication for a heightened fertility.
+However, this decrease in temperature is not visible for all cycles, and thus significantly impacts the prediction qualities of the model.
+Figure~\ref{fig:discussion_spike_in_temperature_drop} shows an anomaly that results in a temperature spike during
+the pre-ovulatory temperature dip.
+This confuses the model into ending the fertile phase earlier.
+
+This suggests that the model may interpret the termination of the pre-ovulatory temperature decline as a
+signal that ovulation is imminent or has just occurred, thus marking the end of the fertile window.
+Another pattern that is visible for many users is a cycle with no clear temperature drop.
+Figure~\ref{fig:discussion_unclear_temperature_drop} shows such a curve.
+Here, potentially unrelated circumstances lead to a drop in temperature that the model incorrectly identifies as fertile days.
+The actual fertile days don't result in an equally visible temperature drop.
+This indicates, that a clear temperature drop is a precursor to the ovulation and comes with the fertile days,
+but its intensity varies between users and also between cycles of the same user.
+Such a pattern can turn out to be a useful predictor for ovulation / fertility, but the models we trained don't seem
+to be able to differentiate between ovulation-related and unrelated temperature drops.
+
+\begin{figure}[htbp]
+    \centering
+    \includegraphics[width=1.0\textwidth]{resources/figures/discussion/temperature_drop_fertility}
+    \caption{
+        Temperature rolling average and fertility-probability prediction with a clear correlation between the temperature drop
+        pre-ovulation and the fertility. (Values are scaled features)
+    }
+    \label{fig:discussion_temperature_drop_fertility_prediction}
+\end{figure}
+\begin{figure}[htbp]
+    \centering
+    \includegraphics[width=1.0\textwidth]{resources/figures/discussion/temperature_spike_in_temperature_dop}
+    \caption{
+        Temperature rolling average and fertility-probability prediction with a disruptive temperature spike during the pre-ovulatory phase,
+        leading to an early end of the predicted fertile window.
+    }
+    \label{fig:discussion_spike_in_temperature_drop}
+\end{figure}
+\begin{figure}[htbp]
+    \centering
+    \includegraphics[width=1.0\textwidth]{resources/figures/discussion/temperature_unclear_temperature_drop}
+    \caption{
+        Temperature rolling average and fertility-probability prediction with no clear temperature drop and a resulting
+        incorrect fertility-probability prediction. (Values are scaled features)
+    }
+    \label{fig:discussion_unclear_temperature_drop}
+\end{figure}
+
+The results of the irregular vs regular cycle study, as shown in~\ref{subsubsec:regular_vs_irregular_cycles},
+indicate, that the models seem to learn recurrent cycle pattern on a per-user basis.
+We conclude this based on the fact, that regular cycle performance is significantly better than the performance on the irregular cycle set.
+If there is a reliable indicator in the temperature data, or its ablations, then there would not be a large difference between the two subsets.
+The temperature patterns should show up in both regular and irregular cycle users.
+Thus, all models must learn or extract information beyond the actual temperature data, which they seem to rely on more than
+the temperature related features.
+This is likely information indicating some form of regularity, which the models then use to predict the targets for new cycles.
+It would be interesting to take a close look at how and in what intensity the models use certain features,
+and whether the performance changes upon omitting certain features.
+
+
 %
 %In general, we expect the transformer based model to outperform the LSTM basd models, as they have proven to be
 %a more effective for time-series analysis tasks especially for longer sequences.
@@ -38,6 +123,8 @@ Future re-evaluation that stores per-user predictions will enable user-level boo
 % explain the need for further medical interpretation of the results of either model
 
 
+% regular vs irregular cycles: mean baselines perform well on regular cycles as expected, as they get closer to the regularities with every past cycle
+
 % transformer can deal with more data and does also perform better for some cases
 % lstm outperforms
 
@@ -50,8 +137,13 @@ Future re-evaluation that stores per-user predictions will enable user-level boo
 %of a given cycle is already over for any given day, there a variety of external factors that should be taken into
 %consideration for a direct output to the user.
 
+\subsection{Limitations}\label{subsec:limiations}
+This analysis reports aggregate test-set metrics without uncertainty quantification.
+Because we did not retain per-user predictions, we cannot compute user-level bootstrap confidence intervals or perform paired significance testing.
+As a result, apparent performance differences, especially small ones, may reflect sampling variability.
+Future re-evaluation that stores per-user predictions will enable user-level bootstrapping, calibration assessment, and formal comparisons.
 
-\section{Future Work}\label{sec:future_work}
+\subsection{Future Work}\label{sec:future_work}
 There are several directions in which this study could be extended,
 most of which were omitted due to time and resource constraints but represent valuable areas for future exploration.