alex/temperature-based-fertility-prediction-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new topic

Browse Source
This commit is contained in:
Alex
2025-02-24 11:50:03 +01:00
parent ff60eda011
commit 9ea10906aa
2 changed files with 672 additions and 266 deletions
Download Patch File Download Diff File Expand all files Collapse all files
expose/src/expose.tex
+6 -185
View File
@@ -13,213 +13,34 @@
% Document
\begin{document}
\title{Developing a Meta-Framework for Advanced Time-Series Analysis:
Integrating Transformer Architectures with Health Data}
\title{Finding Predictors for Human Ovulation with Attention Mechanisms}
\author{Alexander Blank}
\date{October 2024}
\date{February 2025}
\maketitle
\section{Introduction}\label{sec:introduction}
Time-series data is critical in fields like finance, health, and climate science, where understanding
temporal patterns and forecasting future trends, which rely on accurate modeling, are essential.
Recently, transformer-based architectures have demonstrated improved performance over traditional models like
LSTMs and GRUs, particularly in capturing long-term dependencies and complex temporal patterns.
Women's health is a topic that has been neglected for a long time.
The menstrual cycle and its corresponding ovulation are essential for the reproductive health,
family planning and natural contraception.
This exposé outlines a project that aims to develop a meta-framework that builds upon existing tools like
PyTorch-Forecasting, extending them to handle the unique challenges of high-frequency health data.
The framework will provide a configurable pipeline for dataset creation, model training and evaluation, allowing
easy adaptions fro various time-series tasks, and will introduce support for foundation models and fine-tuning.
A case study on ovulation cycle monitoring and analysis will demonstrate the framework's capabilities in
adaptability, performance, and interpretability.
v
\section{Related Work}\label{sec:related-work}
Traditional time-series models like LSTMs and GRUs have been widely used but struggle with capturing
long-term dependencies and complex temporal patterns.
Transformer-based architectures, such as the Temporal Fusion Transformer (TFT)\cite{lim_temporal_2020}
and PatchTST\cite{nie_time_2023}, offer significant improvements, particularly in scalability and interpretability.
These models have become key in addressing challenges inherent to time-series data,
but the field continues to evolve rapidly with new architectures being developed frequently.
Several existing frameworks, including PyTorch Lightning\cite{lightningai_pytorch_2024}, \
Amazon’s GluonTS\cite{alexandrov_gluonts_2019}, and Meta's Prophet\cite{taylor_forecasting_2017},
provide tools for scalable time-series analysis.
However, they remain low-level, requiring significant customization to incorporate new models or are limited to regression tasks.
This highlights the need for a flexible, modular framework that allows seamless integration of
transformer-based architectures for time-series data, particularly in health applications.
This project aims to bridge this gap by developing a modular, data-driven framework that
simplifies the integration and adaptation of emerging models, focusing on the unique challenges of
health-related time-series data.
\section{Research Questions and Objectives}\label{sec:research-questions-and-objectives}
The primary objective of this project is to create a modular framework designed to address the specific challenges
of high-frequency time-series data in health applications, such as temperature monitoring.
The framework will be adaptable to other types of physiological data, like heart rate and sleep patterns, and scalable across different model architectures.
The key research questions are:
\begin{itemize}
\item \textbf{How can a modular framework be designed to facilitate the integration and adaptation of
transformer-based models for time-series data?}
\item \textbf{What are the unique challenges of applying transformer architectures to health-related
time-series data, and how can the framework address these?}
\item \textbf{How can transformer-based architectures be optimized for high-frequency health data,
such as ovulation cycle monitoring, in terms of performance and interpretability?}
\end{itemize}
This project aims to provide a flexible and scalable solution for the health tech domain, addressing both
performance and integration challenges in time-series analysis.
If time allows, the project will also explore the following:
\begin{itemize}
\item \textbf{How can the framework be extended to allow for training of a foundation model on a large dataset
and fine-tuning on a smaller, domain-specific dataset?}
\item \textbf{What are the most effective evaluation metrics for assessing the performance of transformer models
in the context of health-related time-series data?}
\end{itemize}
\section{Methodology}\label{sec:methodology}
The proposed framework for this project is designed with two major components: dataset creation and model
training, with the potential for model evaluation and deployment depending on the outcomes of the case study.
The framework will be implemented in Python and structured to accommodate the specific requirements of
high-frequency health data analysis, particularly for ovulation cycle monitoring.
\subsection{Dataset Creation}
The dataset creation process will rely on an example MongoDB database as the primary data source, from which health
data will be extracted and prepared for model input.
This phase will involve preprocessing the raw data using \texttt{NumPy} and \texttt{Pandas} to generate
time-series datasets suitable for training transformer models.
The data will be curated to ensure it aligns with the high-frequency nature of ovulation cycle monitoring, and
the preprocessing will include handling missing values, normalizing features, and generating the necessary
time-series sequences.
\subsection{Model Training}
The model training component will focus on applying transformer-based architectures to the prepared dataset.
Two specific models will be used for the case study:
\begin{itemize}
\item \textbf{Temporal Fusion Transformer (TFT)}: Selected for its ability to provide interpretability while
modeling long-term dependencies in time-series data.
\item \textbf{PatchTST}: Chosen for its state-of-the-art performance in handling large-scale time-series
forecasting tasks.
\end{itemize}
Both models will be trained using \texttt{PyTorch}, with support from the \texttt{Transformers} library by
Hugging Face, allowing for efficient implementation and fine-tuning of the transformer architectures.
\subsection{Model Evaluation and Deployment}
The choice of evaluation metrics and deployment strategy will be determined based on the results of the case study.
These components will be incorporated into the framework once the models have been trained and evaluated for
performance and interpretability.
% - quickly introduce the two models I want to use (TFT and PatchTST)
% - framework will be build in python. dataset generation will be based on a example mongodb as data source with numpy and pandas.
% - model training will be done with pytorch with the help of the transformers library by huggingface.
% - choice of model evaluation and deployment will be made during the project, depending on the results of the case study.
\section{Case Study}
The case study will focus on analyzing high-frequency ovulation cycle data obtained from wearable devices,
which records a measurement every 5 minutes (288 measurements a day).
This dataset offers a unique opportunity to evaluate the ability of transformer-based models to handle
complex temporal dependencies in health-related data.
The Temporal Fusion Transformer (TFT) will be employed to explore model interpretability, particularly in terms of
identifying key features and trends that contribute to predictions, while PatchTST will be used to test the
performance and scalability of the framework in handling large-scale time-series forecasting.
The expected outcome of this case study is to demonstrate the adaptability of the framework across two dimensions:
interpretability and performance.
The results will provide insight into how these models can be optimized for real-world health applications and may
offer new directions for handling other physiological data, such as heart rate or sleep patterns.
\section{Timeline}\label{sec:timeline}
Given the exploratory nature of this project, the timeline will be flexible, focusing on key milestones rather than rigid deadlines.
The project will progress in phases, with time allocated for initial exploration, model development, and case study evaluation.
\begin{itemize}
\item \textbf{Exploration and Initial Testing:} Early project phases will involve exploring the transformer-based models (TFT and PatchTST), testing their suitability for high-frequency health data, and refining the framework design.
\item \textbf{Dataset Creation and Preprocessing:} Once the framework design is solidified, attention will turn to creating the dataset and ensuring data preprocessing is aligned with the needs of the transformer models.
\item \textbf{Model Training and Evaluation:} The core phase will focus on model training using PyTorch and evaluating both performance and interpretability, with adjustments made iteratively based on results.
\item \textbf{Final Evaluation and Analysis:} The project will conclude with a thorough analysis of model performance and adaptability, followed by preparing the framework for potential further deployment or case study extensions.
\end{itemize}
This flexible structure allows for adaptation as the project develops, with each phase being iterative and data-driven based on initial findings.
\section{Conclusion}
This project aims to develop a modular framework for integrating transformer-based models in time-series analysis,
with a focus on high-frequency health data.
By applying the framework in a case study involving ovulation cycle monitoring,
the project seeks to demonstrate its scalability, performance, and adaptability for real-world health applications.
The outcomes of this project could pave the way for further advancements in health-related time-series analysis,
extending to other physiological data types and domains.
% bibliography
\printbibliography
\end{document}
Title:
Developing a Meta-Framework for Configurable Time-Series Forecasting: Extending PyTorch Forecasting for Health Data
1. Introduction
Time-series data is critical in fields like finance, health, and climate science, where forecasting, anomaly detection, and classification tasks rely on accurate modeling. Recently, transformer-based architectures have demonstrated improved performance over traditional models like LSTMs in capturing long-term dependencies in time-series data.
This project proposes the development of a meta-framework that builds upon existing tools like PyTorch Forecasting, extending them to handle the unique challenges of high-frequency health data. The framework will provide a configurable pipeline for dataset creation, model training, and evaluation, allowing easy adaptation to various time-series tasks, and will introduce support for foundation models. A case study on ovulation cycle monitoring will demonstrate the framework's adaptability and scalability.
2. Related Work
Traditional models like LSTMs have been widely used but often struggle with long-term dependencies in time-series data. Transformer-based architectures like the Temporal Fusion Transformer (TFT) and PatchTST have emerged as more scalable and interpretable alternatives.
Frameworks such as PyTorch Lightning, Amazon’s GluonTS, and Meta’s Prophet provide tools for time-series analysis but often require significant customization and are limited in their flexibility for handling different data sources and tasks. PyTorch Forecasting, which includes implementations of models like TFT, is powerful but lacks a configurable pipeline for handling raw datasets, and it does not yet integrate with foundation models.
This project aims to bridge these gaps by building a meta-framework that integrates PyTorch Forecasting, supports foundation models, and provides a configurable data pipeline tailored for health data.
3. Research Questions and Objectives
The primary objective is to develop a meta-framework that extends PyTorch Forecasting with a configurable pipeline and support for foundation models, specifically for health-related time-series data. The framework will be designed to handle high-frequency data like ovulation cycle monitoring and be adaptable for future use cases.
Key research questions include:
How can a configurable pipeline be designed to integrate data processing, model training, and evaluation for time-series forecasting?
What challenges arise when applying transformer models to high-frequency health data, and how can these be addressed?
How can a foundation model be pre-trained on health data and fine-tuned for specific tasks like ovulation cycle monitoring?
4. Methodology
The proposed meta-framework will consist of three main components: dataset creation, model training, and evaluation/deployment. It will leverage PyTorch Forecasting while extending its capabilities with custom data preprocessing and support for foundation models.
4.1 Dataset Creation
The dataset creation phase will involve designing a pipeline that extracts data from sources like MongoDB, processes it using NumPy and Pandas, and outputs time-series datasets ready for forecasting models. The pipeline will handle the specific needs of high-frequency health data (e.g., handling missing values, feature engineering).
4.2 Model Training
Two models will be used for the case study:
Temporal Fusion Transformer (TFT) for interpretability.
PatchTST for high-performance time-series forecasting.
The training pipeline will be highly configurable, allowing users to define model parameters, preprocessing steps, and evaluation metrics via configuration files.
4.3 Foundation Model Integration
In addition to existing models, the framework will allow for the training and fine-tuning of a foundation model, pre-trained on health-related time-series data and adapted for specific tasks like ovulation cycle monitoring.
4.4 Model Evaluation and Deployment
Evaluation metrics will be chosen based on the case study’s results, focusing on both performance and interpretability. If successful, the system will include deployment options for real-world application.
5. Case Study
The case study will evaluate the framework using ovulation cycle data (recorded every 5 minutes, 288 measurements per day). The goal is to demonstrate how the framework can manage high-frequency health data, apply transformer-based models like TFT for interpretability, and optimize performance using PatchTST.
This case study will also show how a foundation model can be fine-tuned for specific health tasks, providing insights into how these models can generalize across different types of physiological data.
6. Timeline
Given the exploratory nature of the project, the timeline will focus on key milestones rather than rigid deadlines:
Initial Exploration: Refining the framework design and integrating PyTorch Forecasting (Month 1-2).
Dataset Creation: Developing the data pipeline and preprocessing health data (Month 3).
Model Training: Implementing and training TFT and PatchTST models (Month 4).
Foundation Model Integration: Pretraining and fine-tuning the foundation model (Month 5).
Evaluation and Conclusion: Assessing model performance and writing up results (Month 6).
7. Conclusion
This project will deliver a configurable meta-framework for time-series forecasting that extends PyTorch Forecasting by integrating health data preprocessing, flexible model training, and foundation model support. The case study on ovulation cycle monitoring will demonstrate its applicability and scalability, potentially leading to broader use in health-related time-series forecasting.