diff --git a/presentation/presentation_scads_template.odp b/presentation/presentation_scads_template.odp index b9fe36e..203db17 100644 Binary files a/presentation/presentation_scads_template.odp and b/presentation/presentation_scads_template.odp differ diff --git a/thesis/main.tex b/thesis/main.tex index d76805f..5418505 100644 --- a/thesis/main.tex +++ b/thesis/main.tex @@ -11,8 +11,8 @@ \usepackage{graphicx} \graphicspath{{resources/figures/}} -\usepackage[style=ieee, backend=biber]{biblatex} \usepackage{blindtext} +\usepackage[style=ieee, backend=biber]{biblatex} \addbibresource{../main.bib} \usepackage{booktabs} @@ -25,7 +25,7 @@ \includegraphics[width=7cm]{leipzig_university_logo}\\[1cm] % Adjust size as needed - {\huge \textbf{Ovulation Prediction with Machine Learning}}\\[1.5cm] + {\huge \textbf{Body-Core Temperature based Ovulation Prediction with Machine Learning}}\\[1.5cm] \textbf{Master’s Thesis}\\[1cm] @@ -38,7 +38,8 @@ \textbf{Date:} September 2025\\[2cm] \textbf{Master of Science: Data Science}\\ - \textbf{Leipzig University} + \textbf{Leipzig University}\\ + \textit{In collaboration with VivoSensMedical GmbH} \end{titlepage} \pagebreak @@ -47,6 +48,7 @@ \end{abstract} \pagebreak + \tableofcontents \include{sections/introduction} @@ -63,6 +65,19 @@ \include{sections/conclusion} + \section*{Declaration of Use of AI-Assisted Writing Tools} + + Parts of this thesis were prepared with the assistance of generative AI tools, including OpenAI’s ChatGPT . + These tools were used to improve clarity, coherence, and fluency of the written language, + to rephrase technical explanations for greater precision, and to support structural editing. + + At no point were generative models used to produce scientific content, generate results, or conduct analysis. + All AI-generated suggestions were critically reviewed, adapted, and verified by me to ensure their accuracy and relevance. + + The outputs served solely as editorial support and did not replace my own academic work or authorship. + Responsibility for the content, structure, and conclusions of this thesis remains entirely my own. + \pagebreak + \printbibliography \end{document} \ No newline at end of file diff --git a/thesis/resources/figures/methodology_convolution_architecture.png b/thesis/resources/figures/methodology_convolution_architecture.png new file mode 100644 index 0000000..0992cf9 Binary files /dev/null and b/thesis/resources/figures/methodology_convolution_architecture.png differ diff --git a/thesis/resources/figures/methodology_lstm_architecture.png b/thesis/resources/figures/methodology_lstm_architecture.png new file mode 100644 index 0000000..9c69f6c Binary files /dev/null and b/thesis/resources/figures/methodology_lstm_architecture.png differ diff --git a/thesis/resources/figures/methodology_transformer_architecture.png b/thesis/resources/figures/methodology_transformer_architecture.png new file mode 100644 index 0000000..60b0e18 Binary files /dev/null and b/thesis/resources/figures/methodology_transformer_architecture.png differ diff --git a/thesis/sections/methodology.tex b/thesis/sections/methodology.tex index 44accb2..95c55bc 100644 --- a/thesis/sections/methodology.tex +++ b/thesis/sections/methodology.tex @@ -165,6 +165,7 @@ The 1-day window length reflects the expected circadian cycle and strikes a bala Target features represent the outputs that the models are trained to predict. These can include exogenous biological outcomes or interpretable derivatives of input features. +For this study, the following exogenous variables are used: \begin{itemize} \item \textbf{Fertility / Pregnancy Probability} — The estimated probability of conception from unprotected intercourse on the current day. @@ -185,6 +186,7 @@ These can include exogenous biological outcomes or interpretable derivatives of The combination of these two targets is intended to provide the user with both physiological and practical insight: \textit{Fertility probability} conveys the likelihood of conception, but alone does not indicate whether ovulation is yet to come or has already passed. A fertility probability near zero could mean that ovulation is either in the past or still ahead—information the model alone cannot disambiguate. +Figure~\ref{fig:methodology_target_features} shows an example cycle where the target feature values during the course of a cycle can be seen. The \textit{ovulation-over indicator}, by contrast, explicitly marks the post-ovulatory phase, but does not describe conception risk. Together, the two outputs offer complementary information and improve interpretability for real-time user-facing applications. @@ -192,14 +194,50 @@ Together, the two outputs offer complementary information and improve interpreta As discussed in Section~\ref{sec:discussion}, all predictions are subject to further interpretation before presentation in the product interface. The model outputs represent data-driven estimates and do not constitute medical advice or diagnostic statements. -\subsection{Time-Series Modeling Approach}\label{subsec:time-series_modeling_approach} +\begin{table}[htbp] + \centering + \begin{tabular}{@{}lp{0.62\linewidth}@{}} + \toprule + \textbf{Feature Name} & \textbf{Description} \\ + \midrule + \multicolumn{2}{@{}l}{\textbf{Static Features}} \\ + \midrule + User age & Age in years; mean imputed if missing \\ + User height & Height in centimeters; mean imputed if missing \\ + User weight & Weight in kilograms; mean imputed if missing \\ + Average cycle length & Mean length of all previous cycles for this user \\ + Cycle length STD & Standard deviation of previous cycle lengths \\ + Ovulatory fraction & Proportion of prior cycles classified as ovulatory \\ + Cycle count & Number of previously completed cycles available \\ + Avg. pre-ovulatory temperature & Mean temperature in the follicular phase of previous cycles \\ + Avg. post-ovulatory temperature & Mean temperature in the luteal phase of previous cycles \\ + \midrule + \multicolumn{2}{@{}l}{\textbf{Known (Time-Based) Features}} \\ + \midrule + Time since cycle start & Hours since the start of the current cycle \\ + Hour of day & Hour of the day (0–23) \\ + Day of the week & Day of the week (0–6) \\ + Month of the year & Month of the year (0–11) \\ + \midrule + \multicolumn{2}{@{}l}{\textbf{Observable Features}} \\ + \midrule + Raw temperature & Temperature value recorded by the sensor \\ + Rolling average temperature & 1-day rolling average (288 measurements) \\ + Rolling window temp. minimum & Minimum temperature within 1-day window \\ + Rolling window temp. maximum & Maximum temperature within 1-day window \\ + \bottomrule + \end{tabular} + \caption{Overview of all features and their descriptions.} + \label{tab:feature_overview} +\end{table} -\subsubsection{Input \& Output Modeling}\label{subsubsec:input_output_modeling} -% build in the probability curve for the pregnancy chance +All features were normalized based on their empirical distributions. +A \textit{standard scaler} was applied to approximately normal features without outliers, +a \textit{robust scaler} was used for distributions with outliers, and a \textit{MinMax scaler} was used for all others. -\subsubsection{Sequence Representation and Sampling} -\label{subsubsec:input_sequence_construction} +\subsubsection{Time-Series Input Representation} +\label{subsubsec:time_series_input_representation} Due to the high temporal resolution of the temperature data (288 measurements per day), raw input sequences can become prohibitively long for most model types. @@ -207,19 +245,25 @@ To manage input size and evaluate the impact of temporal resolution on predictiv Consecutive time steps are aggregated into bins of configurable size, and each bin is reduced to a single value using a feature-specific aggregation function. For most continuous features, the \texttt{mean} is used. -For cyclic or categorical features—such as \textit{day of the week}— the \texttt{max} or \texttt{mode} is applied to avoid introducing -artifacts at bin boundaries, where values from distinct categories (e.g., hours 23 and 0) might otherwise be averaged into a nonexistent intermediate state. +For cyclic or categorical features—such as \textit{day of the week}—the \texttt{max} or \texttt{mode} is applied to avoid introducing artifacts at bin boundaries, +where values from distinct categories (e.g., hours 23 and 0) might otherwise be averaged into a nonexistent intermediate state. The effect of different sampling resolutions and aggregation strategies is evaluated in Section~\ref{sec:results}. To simulate real-time prediction rather than retrospective analysis, a sliding-window approach is employed. -This allows the model to make partial predictions based on the data available up to a certain point in the cycle. +This allows the model to make predictions based only on data available up to a specific point in the cycle. -Each cycle is split into overlapping input windows, where each window includes data from the start of the cycle up to a specific day. +Each cycle is split into overlapping input windows, where each window includes data from the cycle start up to a defined time step. The window length is fixed and configurable. As the cycle progresses, the window slides forward, allowing the model to incorporate increasing historical context over time. -This setup enables temporally resolved predictions at different stages of the cycle and supports evaluation of how predictive accuracy changes with increasing context. +For the model types used in this study, each window produces a single output vector. +By default, this corresponds to the predicted target values at the final time step of the window, though this can be offset depending on configuration. +While the architecture could be extended to produce output sequences (e.g., one prediction per input step), this study focuses on single-vector outputs. + +This setup enables temporally resolved predictions at different stages of the cycle and supports analysis of how predictive accuracy evolves with increasing context. +Depending on the configuration, downsampling and windowing can be skipped to allow the raw data to be processed by the models themselves. +This is used primarily in the convolutional flavours of the models, to allow them to learn the best way of reducing the input complexity based on the data itself. \begin{figure}[htbp] \centering @@ -234,20 +278,125 @@ This setup enables temporally resolved predictions at different stages of the cy Fixed-length input windows would normally prevent early-cycle predictions when insufficient data is available. To address this, left-padding is applied using masked values. -In this study, predictions are enabled once at least 4 days of data are available. +In this study, predictions are enabled once at least four days of data are available. A padding value of 0.0 is used for all features, and the padding length is adjusted accordingly. -This choice ensures that the model learns to ignore tokens consisting entirely of padding. +This design ensures that the model learns to ignore tokens consisting entirely of padding. -The feature \textit{hours since start}, which encodes the time elapsed since cycle onset, will be zero for all padded tokens—explicitly indicating -that these entries contain no meaningful information. +The feature \textit{hours since start}, which encodes the time elapsed since cycle onset, is also set to 0.0 for all padded tokens— +explicitly indicating that these entries contain no usable information. Figure~\ref{fig:methodology_padding_example} shows an example of such padding during early-cycle input preparation. +\subsection{Model Architecture and Selection}\label{subsec:model_architecture_and_selection} + +The primary objective of this study is to find models that accurately predict the features introduced in~\ref{fig:methodology_target_features}, +based on the historical data and context variables. +This task presents several modelling challenges: capturing temporal dependencies across varying cycle lengths, handling irregular menstrual patterns +and adapting to user-specific variability. + +Ultimately, the models used in this study were selected based on their ability to: +\begin{itemize} + \item Leverage sequential input efficiently across multiple time scales + \item Learn temporal patterns from partially observed data + \item Generalize across users while incorporating personalized cycle context +\end{itemize} + +\subsubsection{LSTM-Architecture}\label{subsubsec:lstm_architecture} +\begin{figure}[htbp] + \centering + \includegraphics[width=0.7\textwidth]{methodology_lstm_architecture} + \caption{ + Schematic overview of the LSTM architecture used in this study. + } + \label{fig:methodology_lstm_architecture} +\end{figure} + +Recurrent neural networks, particularly GRUs and LSTMs, were tested for their ability to model long-term dependencies in the time series. +Their sequential memory structure allows them to retain information across cycle days, but they may struggle with high-resolution input and longer sequences. +LSTMs, in particular, have a long history of strong performance in sequence prediction tasks. +For this study, a stack of LSTM layers was followed by a linear projection layer, mapping the hidden state at the final time +step to the two target variables: fertility probability and ovulation-over indicator. +Figure~\ref{fig:methodology_lstm_architecture} shows the overall architecture pipeline used for the LSTM-based model. +The stacked inputs and outputs denote the batch processing of the model. + +Model-specific architectural parameters are: +\begin{itemize} + \item \textbf{Input Length} — Number of time steps included in each input sequence. + \item \textbf{Hidden Size} — Dimensionality of the LSTM’s internal hidden state. + \item \textbf{Number of Layers} — Depth of the LSTM stack. +\end{itemize} +The specific values and tuning ranges for these parameters are discussed in Section~\ref{subsubsec:hyperparameter_tuning}. + +\subsubsection{Transformer Architecture}\label{subsubsec:transformer_architecture} + +\begin{figure}[htbp] + \centering + \includegraphics[width=0.7\textwidth]{methodology_transformer_architecture} + \caption{ + Schematic overview of the Transformer architecture used in this study. + } + \label{fig:methodology_transformer_architecture} +\end{figure} + +Self-attention models such as the Transformer were used to capture long-range dependencies in the sequence without relying on recurrence. +These models have demonstrated state-of-the-art performance in various sequential tasks and are more robust to vanishing gradients than RNN-based alternatives. + +An overview of the original Transformer design is provided in Section~\ref{subsubsec:transformer_models}. +For this study, the architecture was adapted for multivariate time-series prediction. + +As the input consists of continuous features rather than discrete tokens, no embedding layer is used. +However, positional encodings are still added to allow the model to interpret the relative positions of tokens—essential for effective attention computation. +While time-dependent features like \textit{hour-of-day} or \textit{time-since-cycle-start} carry positional information implicitly, +positional encoding was retained for architectural consistency. +A future direction could investigate the impact of omitting it in such naturally ordered domains. + +Since the task does not require sequence-to-sequence modeling, only the encoder part of the Transformer is used. +Its output—one vector per input token—is aggregated via 1D adaptive average pooling, resulting in a single vector representation per sequence. +This vector is then passed through a linear projection layer to produce the two target outputs: +fertility probability and ovulation-over indicator. + +Figure~\ref{fig:methodology_transformer_architecture} shows the overall architecture. +The stacked inputs and outputs indicate batch processing. + +\subsubsection{Temporal Convolution Layer} +\label{subsubsec:temporal_convolution_layer} + +\begin{figure}[htbp] + \centering + \includegraphics[width=0.7\textwidth]{methodology_convolution_architecture} + \caption{ + Schematic overview of the temporal convolution layer architecture used in this study. + } + \label{fig:methodology_convolution_architecture} +\end{figure} + +For both the LSTM and Transformer models, long input sequences can substantially increase model complexity and training time. +While input downsampling mitigates this, static resampling risks discarding relevant temporal patterns and reducing predictive quality. + +To address this, a learnable temporal downsampling module was introduced, based on 1D convolutional layers combined with adaptive pooling. +This allows the model to reduce sequence length in a data-driven manner while preserving important features. + +Specifically, a two-stage convolutional block is used to reduce the input resolution from 288 to 48 measurements per day. +Each stage consists of a 1D convolution followed by an adaptive average pooling layer. +The feature dimension remains unchanged throughout the downsampling process. +Figure~\ref{fig:methodology_convolution_architecture} shows the full pipeline of this convolutional preprocessing module. +The output is subsequently passed to the LSTM or Transformer model as described in Sections~\ref{subsubsec:lstm_architecture} and~\ref{subsubsec:transformer_architecture}. + +The convolutional layer is trained end-to-end with the main model, allowing gradient-based optimization +to adaptively learn which local patterns are most informative for the downstream prediction tasks. + +Further research may be necessary to identify optimal parameters for the convolutional downsampling process. +Both the number of convolutional stages and the final output resolution can be tuned to balance model capacity and temporal fidelity. +Future work could also explore alternative, potentially more interpretable downsampling strategies. +In particular, a qualitative analysis of which temporal motifs are preserved—or lost—through the convolutional layers +might offer valuable insight into the interpretability and robustness of learned representations. + \subsection{Model Training}\label{subsec:model_training} \subsubsection{Training Setup}\label{subsubsec:training_setup} \subsubsection{Hyperparameter Tuning}\label{subsubsec:hyperparameter_tuning} +% what name to use? -> fix experimental setup reference from earlier \subsubsection{Training Details}\label{subsubsec:training_details} diff --git a/thesis/sections/results.tex b/thesis/sections/results.tex index 6dc4ffa..8113a95 100644 --- a/thesis/sections/results.tex +++ b/thesis/sections/results.tex @@ -1,4 +1,6 @@ %! Author = alex %! Date = 3/6/25 -\section{Results}\label{sec:results} \ No newline at end of file +\section{Results}\label{sec:results} + +% provide information about the training behaviour and statistic of the different models \ No newline at end of file