From ff60eda011f6c4b85a7c3cbb81be90f6e35aa423 Mon Sep 17 00:00:00 2001 From: Alex Blank Date: Tue, 15 Oct 2024 16:54:54 +0200 Subject: [PATCH] added new draft --- expose/src/expose.tex | 83 ++++++++++++--- main.bib | 233 ++++++++++++++++++++++-------------------- 2 files changed, 190 insertions(+), 126 deletions(-) diff --git a/expose/src/expose.tex b/expose/src/expose.tex index b85b44b..0663161 100644 --- a/expose/src/expose.tex +++ b/expose/src/expose.tex @@ -13,7 +13,7 @@ % Document \begin{document} - \title{Developing a Data-Driven Framework for Advanced Time-Series Analysis: + \title{Developing a Meta-Framework for Advanced Time-Series Analysis: Integrating Transformer Architectures with Health Data} \author{Alexander Blank} \date{October 2024} @@ -21,19 +21,17 @@ \section{Introduction}\label{sec:introduction} - Time-series data is a cornerstone of many fields, including finance, health, and climate science, - where it plays a crucial role in forecasting, anomaly detection, and classification tasks. - Predictive models based on time-series data are widely used in applications such as weather forecasting, - stock price prediction, health monitoring, and security systems. - In recent years, transformer-based architectures have shown significant improvements in time-series analysis, - outperforming traditional models like LSTMs in capturing complex temporal patterns. - As the field of time-series analysis rapidly evolves, new models and architectures are continually developed to - meet the growing demands of high-frequency data analysis. + Time-series data is critical in fields like finance, health, and climate science, where understanding + temporal patterns and forecasting future trends, which rely on accurate modeling, are essential. + Recently, transformer-based architectures have demonstrated improved performance over traditional models like + LSTMs and GRUs, particularly in capturing long-term dependencies and complex temporal patterns. - This exposé outlines the development of a modular, data-driven framework designed to integrate advanced transformer - architectures for time-series analysis. - To demonstrate its capabilities, the framework will be applied in a case study focused on high-frequency - ovulation cycle health data, showcasing its adaptability across various model architectures. + This exposé outlines a project that aims to develop a meta-framework that builds upon existing tools like + PyTorch-Forecasting, extending them to handle the unique challenges of high-frequency health data. + The framework will provide a configurable pipeline for dataset creation, model training and evaluation, allowing + easy adaptions fro various time-series tasks, and will introduce support for foundation models and fine-tuning. + A case study on ovulation cycle monitoring and analysis will demonstrate the framework's capabilities in + adaptability, performance, and interpretability. \section{Related Work}\label{sec:related-work} @@ -166,3 +164,62 @@ \end{document} + +Title: +Developing a Meta-Framework for Configurable Time-Series Forecasting: Extending PyTorch Forecasting for Health Data + +1. Introduction +Time-series data is critical in fields like finance, health, and climate science, where forecasting, anomaly detection, and classification tasks rely on accurate modeling. Recently, transformer-based architectures have demonstrated improved performance over traditional models like LSTMs in capturing long-term dependencies in time-series data. + +This project proposes the development of a meta-framework that builds upon existing tools like PyTorch Forecasting, extending them to handle the unique challenges of high-frequency health data. The framework will provide a configurable pipeline for dataset creation, model training, and evaluation, allowing easy adaptation to various time-series tasks, and will introduce support for foundation models. A case study on ovulation cycle monitoring will demonstrate the framework's adaptability and scalability. + +2. Related Work +Traditional models like LSTMs have been widely used but often struggle with long-term dependencies in time-series data. Transformer-based architectures like the Temporal Fusion Transformer (TFT) and PatchTST have emerged as more scalable and interpretable alternatives. + +Frameworks such as PyTorch Lightning, Amazon’s GluonTS, and Meta’s Prophet provide tools for time-series analysis but often require significant customization and are limited in their flexibility for handling different data sources and tasks. PyTorch Forecasting, which includes implementations of models like TFT, is powerful but lacks a configurable pipeline for handling raw datasets, and it does not yet integrate with foundation models. + +This project aims to bridge these gaps by building a meta-framework that integrates PyTorch Forecasting, supports foundation models, and provides a configurable data pipeline tailored for health data. + +3. Research Questions and Objectives +The primary objective is to develop a meta-framework that extends PyTorch Forecasting with a configurable pipeline and support for foundation models, specifically for health-related time-series data. The framework will be designed to handle high-frequency data like ovulation cycle monitoring and be adaptable for future use cases. + +Key research questions include: + +How can a configurable pipeline be designed to integrate data processing, model training, and evaluation for time-series forecasting? +What challenges arise when applying transformer models to high-frequency health data, and how can these be addressed? +How can a foundation model be pre-trained on health data and fine-tuned for specific tasks like ovulation cycle monitoring? +4. Methodology +The proposed meta-framework will consist of three main components: dataset creation, model training, and evaluation/deployment. It will leverage PyTorch Forecasting while extending its capabilities with custom data preprocessing and support for foundation models. + +4.1 Dataset Creation +The dataset creation phase will involve designing a pipeline that extracts data from sources like MongoDB, processes it using NumPy and Pandas, and outputs time-series datasets ready for forecasting models. The pipeline will handle the specific needs of high-frequency health data (e.g., handling missing values, feature engineering). + +4.2 Model Training +Two models will be used for the case study: + +Temporal Fusion Transformer (TFT) for interpretability. +PatchTST for high-performance time-series forecasting. +The training pipeline will be highly configurable, allowing users to define model parameters, preprocessing steps, and evaluation metrics via configuration files. + +4.3 Foundation Model Integration +In addition to existing models, the framework will allow for the training and fine-tuning of a foundation model, pre-trained on health-related time-series data and adapted for specific tasks like ovulation cycle monitoring. + +4.4 Model Evaluation and Deployment +Evaluation metrics will be chosen based on the case study’s results, focusing on both performance and interpretability. If successful, the system will include deployment options for real-world application. + +5. Case Study +The case study will evaluate the framework using ovulation cycle data (recorded every 5 minutes, 288 measurements per day). The goal is to demonstrate how the framework can manage high-frequency health data, apply transformer-based models like TFT for interpretability, and optimize performance using PatchTST. + +This case study will also show how a foundation model can be fine-tuned for specific health tasks, providing insights into how these models can generalize across different types of physiological data. + +6. Timeline +Given the exploratory nature of the project, the timeline will focus on key milestones rather than rigid deadlines: + +Initial Exploration: Refining the framework design and integrating PyTorch Forecasting (Month 1-2). +Dataset Creation: Developing the data pipeline and preprocessing health data (Month 3). +Model Training: Implementing and training TFT and PatchTST models (Month 4). +Foundation Model Integration: Pretraining and fine-tuning the foundation model (Month 5). +Evaluation and Conclusion: Assessing model performance and writing up results (Month 6). +7. Conclusion +This project will deliver a configurable meta-framework for time-series forecasting that extends PyTorch Forecasting by integrating health data preprocessing, flexible model training, and foundation model support. The case study on ovulation cycle monitoring will demonstrate its applicability and scalability, potentially leading to broader use in health-related time-series forecasting. + diff --git a/main.bib b/main.bib index 5b455d1..f709e98 100644 --- a/main.bib +++ b/main.bib @@ -1,87 +1,35 @@ -@article{zhang_crossformer_2023, - title = {{CROSSFORMER}: {TRANSFORMER} {UTILIZING} {CROSS}- {DIMENSION} {DEPENDENCY} {FOR} {MULTIVARIATE} {TIME} {SERIES} {FORECASTING}}, - abstract = {Recently many deep models have been proposed for multivariate time series (MTS) forecasting. In particular, Transformer-based models have shown great potential because they can capture long-term dependency. However, existing Transformerbased models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (crossdimension dependency), which is critical for MTS forecasting. To fill the gap, we propose Crossformer, a Transformer-based model utilizing cross-dimension dependency for MTS forecasting. In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information. Then the Two-Stage Attention (TSA) layer is proposed to efficiently capture the cross-time and cross-dimension dependency. Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the final forecasting. Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts.}, +@misc{taylor_forecasting_2017, + title = {Forecasting at scale}, + copyright = {http://creativecommons.org/licenses/by/4.0/}, + url = {https://peerj.com/preprints/3190v2}, + doi = {10.7287/peerj.preprints.3190v2}, + abstract = {Forecasting is a common data science task that helps organizations with capacity planning, goal setting, and anomaly detection. Despite its importance, there are serious challenges associated with producing reliable and high quality forecasts –especially when there are a variety of time series and analysts with expertise in time series modeling are relatively rare. To address these challenges, we describe a practical approach to forecasting “at scale” that combines configurable models with analyst-in-the-loop performance analysis. We propose a modular regression model with interpretable parameters that can be intuitively adjusted by analysts with domain knowledge about the time series. We describe performance analyses to compare and evaluate forecasting procedures, and automatically flag forecasts for manual review and adjustment. Tools that help analysts to use their expertise most effectively enable reliable, practical forecasting of business time series.}, language = {en}, - author = {Zhang, Yunhao and Yan, Junchi}, - year = {2023}, - file = {PDF:/home/alex/Zotero/storage/NM9CETJS/Zhang and Yan - 2023 - CROSSFORMER TRANSFORMER UTILIZING CROSS- DIMENSION DEPENDENCY FOR MULTIVARIATE TIME SERIES FORECAST.pdf:application/pdf}, -} - -@misc{shao_exploring_2023, - title = {Exploring {Progress} in {Multivariate} {Time} {Series} {Forecasting}: {Comprehensive} {Benchmarking} and {Heterogeneity} {Analysis}}, - shorttitle = {Exploring {Progress} in {Multivariate} {Time} {Series} {Forecasting}}, - url = {http://arxiv.org/abs/2310.06119}, - abstract = {Multivariate Time Series (MTS) widely exists in real-word complex systems, such as traffic and energy systems, making their forecasting crucial for understanding and influencing these systems. Recently, deep learning-based approaches have gained much popularity for effectively modeling temporal and spatial dependencies in MTS, specifically in Long-term Time Series Forecasting (LTSF) and Spatial-Temporal Forecasting (STF). However, the fair benchmarking issue and the choice of technical approaches have been hotly debated in related work. Such controversies significantly hinder our understanding of progress in this field. Thus, this paper aims to address these controversies to present insights into advancements achieved. To resolve benchmarking issues, we introduce BasicTS, a benchmark designed for fair comparisons in MTS forecasting. BasicTS establishes a unified training pipeline and reasonable evaluation settings, enabling an unbiased evaluation of over 30 popular MTS forecasting models on more than 18 datasets. Furthermore, we highlight the heterogeneity among MTS datasets and classify them based on temporal and spatial characteristics. We further prove that neglecting heterogeneity is the primary reason for generating controversies in technical approaches. Moreover, based on the proposed BasicTS and rich heterogeneous MTS datasets, we conduct an exhaustive and reproducible performance and efficiency comparison of popular models, providing insights for researchers in selecting and designing MTS forecasting models.}, - urldate = {2024-10-10}, - publisher = {arXiv}, - author = {Shao, Zezhi and Wang, Fei and Xu, Yongjun and Wei, Wei and Yu, Chengqing and Zhang, Zhao and Yao, Di and Jin, Guangyin and Cao, Xin and Cong, Gao and Jensen, Christian S. and Cheng, Xueqi}, - month = oct, - year = {2023}, - note = {arXiv:2310.06119}, - keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning}, - file = {Preprint PDF:/home/alex/Zotero/storage/7EFZ5IT6/Shao et al. - 2023 - Exploring Progress in Multivariate Time Series Forecasting Comprehensive Benchmarking and Heterogen.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/W6RNWBLM/2310.html:text/html}, -} - -@misc{nie_time_2023, - title = {A {Time} {Series} is {Worth} 64 {Words}: {Long}-term {Forecasting} with {Transformers}}, - shorttitle = {A {Time} {Series} is {Worth} 64 {Words}}, - url = {http://arxiv.org/abs/2211.14730}, - abstract = {We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.}, - urldate = {2024-10-10}, - publisher = {arXiv}, - author = {Nie, Yuqi and Nguyen, Nam H. and Sinthong, Phanwadee and Kalagnanam, Jayant}, - month = mar, - year = {2023}, - note = {arXiv:2211.14730}, - keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning}, - file = {Preprint PDF:/home/alex/Zotero/storage/DG4ZJCWV/Nie et al. - 2023 - A Time Series is Worth 64 Words Long-term Forecasting with Transformers.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/H6XGVBY6/2211.html:text/html}, -} - -@misc{lim_temporal_2020, - title = {Temporal {Fusion} {Transformers} for {Interpretable} {Multi}-horizon {Time} {Series} {Forecasting}}, - url = {http://arxiv.org/abs/1912.09363}, - abstract = {Multi-horizon forecasting problems often contain a complex mix of inputs -- including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed historically -- without any prior information on how they interact with the target. While several deep learning models have been proposed for multi-step prediction, they typically comprise black-box models which do not account for the full range of inputs present in common scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) -- a novel attention-based architecture which combines high-performance multi-horizon forecasting with interpretable insights into temporal dynamics. To learn temporal relationships at different scales, the TFT utilizes recurrent layers for local processing and interpretable self-attention layers for learning long-term dependencies. The TFT also uses specialized components for the judicious selection of relevant features and a series of gating layers to suppress unnecessary components, enabling high performance in a wide range of regimes. On a variety of real-world datasets, we demonstrate significant performance improvements over existing benchmarks, and showcase three practical interpretability use-cases of TFT.}, - urldate = {2024-10-10}, - publisher = {arXiv}, - author = {Lim, Bryan and Arik, Sercan O. and Loeff, Nicolas and Pfister, Tomas}, + urldate = {2024-10-14}, + publisher = {PeerJ Preprints}, + author = {Taylor, Sean J and Letham, Benjamin}, month = sep, - year = {2020}, - note = {arXiv:1912.09363}, - keywords = {Computer Science - Machine Learning, Statistics - Machine Learning}, - file = {Preprint PDF:/home/alex/Zotero/storage/2R2H34KB/Lim et al. - 2020 - Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/ETDAYW36/1912.html:text/html}, + year = {2017}, + file = {PDF:/home/alex/Zotero/storage/GK5AIG2V/Taylor and Letham - 2017 - Forecasting at scale.pdf:application/pdf}, } -@article{hochreiter_long_1997, - title = {Long {Short}-{Term} {Memory}}, - volume = {9}, - issn = {0899-7667, 1530-888X}, - url = {https://direct.mit.edu/neco/article/9/8/1735-1780/6109}, - doi = {10.1162/neco.1997.9.8.1735}, - abstract = {Learningtostoreinformationoverextendedtimeintervalsviarecurrentbackpropagation takesaverylongtime,mostlyduetoinsu cient,decayingerrorbackow.Webrieyreview Hochreiter's1991analysisofthisproblem,thenaddressitbyintroducinganovel,e cient, gradient-basedmethodcalled{\textbackslash}LongShort-TermMemory"(LSTM).Truncatingthegradient wherethisdoesnotdoharm,LSTMcanlearntobridgeminimaltimelagsinexcessof1000 discretetimestepsbyenforcingconstanterrorowthrough{\textbackslash}constanterrorcarrousels"within specialunits.Multiplicativegateunitslearntoopenandcloseaccesstotheconstanterror ow.LSTMislocalinspaceandtime;itscomputationalcomplexitypertimestepandweight isO(1).Ourexperimentswitharticialdatainvolvelocal,distributed,real-valued,andnoisy patternrepresentations.IncomparisonswithRTRL,BPTT,RecurrentCascade-Correlation, Elmannets,andNeuralSequenceChunking,LSTMleadstomanymoresuccessfulruns,and learnsmuchfaster.LSTMalsosolvescomplex,articiallongtimelagtasksthathavenever beensolvedbypreviousrecurrentnetworkalgorithms.}, +@book{hutter_machine_2021, + address = {Cham}, + series = {Lecture {Notes} in {Computer} {Science}}, + title = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}: {European} {Conference}, {ECML} {PKDD} 2020, {Ghent}, {Belgium}, {September} 14–18, 2020, {Proceedings}, {Part} {III}}, + volume = {12459}, + copyright = {https://www.springernature.com/gp/researchers/text-and-data-mining}, + isbn = {978-3-030-67663-6 978-3-030-67664-3}, + shorttitle = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}}, + url = {https://link.springer.com/10.1007/978-3-030-67664-3}, language = {en}, - number = {8}, - urldate = {2024-10-10}, - journal = {Neural Computation}, - author = {Hochreiter, Sepp and Schmidhuber, Jürgen}, - month = nov, - year = {1997}, - pages = {1735--1780}, - file = {PDF:/home/alex/Zotero/storage/CZSV2ASE/Hochreiter and Schmidhuber - 1997 - Long Short-Term Memory.pdf:application/pdf}, -} - -@misc{cho_learning_2014, - title = {Learning {Phrase} {Representations} using {RNN} {Encoder}-{Decoder} for {Statistical} {Machine} {Translation}}, - url = {http://arxiv.org/abs/1406.1078}, - abstract = {In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder-Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.}, - urldate = {2024-10-10}, - publisher = {arXiv}, - author = {Cho, Kyunghyun and Merrienboer, Bart van and Gulcehre, Caglar and Bahdanau, Dzmitry and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua}, - month = sep, - year = {2014}, - note = {arXiv:1406.1078}, - keywords = {Computer Science - Computation and Language, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing, Statistics - Machine Learning}, - file = {Preprint PDF:/home/alex/Zotero/storage/E8WMK2IN/Cho et al. - 2014 - Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/6PTCL8LW/1406.html:text/html}, + urldate = {2024-10-14}, + publisher = {Springer International Publishing}, + editor = {Hutter, Frank and Kersting, Kristian and Lijffijt, Jefrey and Valera, Isabel}, + year = {2021}, + doi = {10.1007/978-3-030-67664-3}, + file = {Submitted Version:/home/alex/Zotero/storage/JMVJMLJ5/Hutter et al. - 2021 - Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2020, Ghent, B.pdf:application/pdf}, } @incollection{hutter_general_2021, @@ -103,37 +51,12 @@ file = {Submitted Version:/home/alex/Zotero/storage/WQIHZ7IP/Bender et al. - 2021 - A General Machine Learning Framework for Survival Analysis.pdf:application/pdf}, } -@book{hutter_machine_2021, - address = {Cham}, - series = {Lecture {Notes} in {Computer} {Science}}, - title = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}: {European} {Conference}, {ECML} {PKDD} 2020, {Ghent}, {Belgium}, {September} 14–18, 2020, {Proceedings}, {Part} {III}}, - volume = {12459}, - copyright = {https://www.springernature.com/gp/researchers/text-and-data-mining}, - isbn = {978-3-030-67663-6 978-3-030-67664-3}, - shorttitle = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}}, - url = {https://link.springer.com/10.1007/978-3-030-67664-3}, - language = {en}, +@misc{lightningai_pytorch_2024, + title = {{PyTorch} {Lightning}}, + url = {https://www.pytorchlightning.ai}, urldate = {2024-10-14}, - publisher = {Springer International Publishing}, - editor = {Hutter, Frank and Kersting, Kristian and Lijffijt, Jefrey and Valera, Isabel}, - year = {2021}, - doi = {10.1007/978-3-030-67664-3}, - file = {Submitted Version:/home/alex/Zotero/storage/JMVJMLJ5/Hutter et al. - 2021 - Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2020, Ghent, B.pdf:application/pdf}, -} - -@misc{taylor_forecasting_2017, - title = {Forecasting at scale}, - copyright = {http://creativecommons.org/licenses/by/4.0/}, - url = {https://peerj.com/preprints/3190v2}, - doi = {10.7287/peerj.preprints.3190v2}, - abstract = {Forecasting is a common data science task that helps organizations with capacity planning, goal setting, and anomaly detection. Despite its importance, there are serious challenges associated with producing reliable and high quality forecasts –especially when there are a variety of time series and analysts with expertise in time series modeling are relatively rare. To address these challenges, we describe a practical approach to forecasting “at scale” that combines configurable models with analyst-in-the-loop performance analysis. We propose a modular regression model with interpretable parameters that can be intuitively adjusted by analysts with domain knowledge about the time series. We describe performance analyses to compare and evaluate forecasting procedures, and automatically flag forecasts for manual review and adjustment. Tools that help analysts to use their expertise most effectively enable reliable, practical forecasting of business time series.}, - language = {en}, - urldate = {2024-10-14}, - publisher = {PeerJ Preprints}, - author = {Taylor, Sean J and Letham, Benjamin}, - month = sep, - year = {2017}, - file = {PDF:/home/alex/Zotero/storage/GK5AIG2V/Taylor and Letham - 2017 - Forecasting at scale.pdf:application/pdf}, + author = {lightning.ai}, + year = {2024}, } @misc{alexandrov_gluonts_2019, @@ -151,10 +74,94 @@ file = {Preprint PDF:/home/alex/Zotero/storage/JP9K74A8/Alexandrov et al. - 2019 - GluonTS Probabilistic Time Series Models in Python.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/RJYSBT29/1906.html:text/html}, } -@misc{lightningai_pytorch_2024, - title = {{PyTorch} {Lightning}}, - url = {https://www.pytorchlightning.ai}, - urldate = {2024-10-14}, - author = {lightning.ai}, +@misc{cho_learning_2014, + title = {Learning {Phrase} {Representations} using {RNN} {Encoder}-{Decoder} for {Statistical} {Machine} {Translation}}, + url = {http://arxiv.org/abs/1406.1078}, + abstract = {In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder-Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Cho, Kyunghyun and Merrienboer, Bart van and Gulcehre, Caglar and Bahdanau, Dzmitry and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua}, + month = sep, + year = {2014}, + note = {arXiv:1406.1078}, + keywords = {Computer Science - Computation and Language, Computer Science - Machine Learning, Statistics - Machine Learning, Computer Science - Neural and Evolutionary Computing}, + file = {Preprint PDF:/home/alex/Zotero/storage/E8WMK2IN/Cho et al. - 2014 - Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/6PTCL8LW/1406.html:text/html}, +} + +@article{hochreiter_long_1997, + title = {Long {Short}-{Term} {Memory}}, + volume = {9}, + issn = {0899-7667, 1530-888X}, + url = {https://direct.mit.edu/neco/article/9/8/1735-1780/6109}, + doi = {10.1162/neco.1997.9.8.1735}, + abstract = {Learningtostoreinformationoverextendedtimeintervalsviarecurrentbackpropagation takesaverylongtime,mostlyduetoinsu cient,decayingerrorbackow.Webrieyreview Hochreiter's1991analysisofthisproblem,thenaddressitbyintroducinganovel,e cient, gradient-basedmethodcalled{\textbackslash}LongShort-TermMemory"(LSTM).Truncatingthegradient wherethisdoesnotdoharm,LSTMcanlearntobridgeminimaltimelagsinexcessof1000 discretetimestepsbyenforcingconstanterrorowthrough{\textbackslash}constanterrorcarrousels"within specialunits.Multiplicativegateunitslearntoopenandcloseaccesstotheconstanterror ow.LSTMislocalinspaceandtime;itscomputationalcomplexitypertimestepandweight isO(1).Ourexperimentswitharticialdatainvolvelocal,distributed,real-valued,andnoisy patternrepresentations.IncomparisonswithRTRL,BPTT,RecurrentCascade-Correlation, Elmannets,andNeuralSequenceChunking,LSTMleadstomanymoresuccessfulruns,and learnsmuchfaster.LSTMalsosolvescomplex,articiallongtimelagtasksthathavenever beensolvedbypreviousrecurrentnetworkalgorithms.}, + language = {en}, + number = {8}, + urldate = {2024-10-10}, + journal = {Neural Computation}, + author = {Hochreiter, Sepp and Schmidhuber, Jürgen}, + month = nov, + year = {1997}, + pages = {1735--1780}, + file = {PDF:/home/alex/Zotero/storage/CZSV2ASE/Hochreiter and Schmidhuber - 1997 - Long Short-Term Memory.pdf:application/pdf}, +} + +@misc{lim_temporal_2020, + title = {Temporal {Fusion} {Transformers} for {Interpretable} {Multi}-horizon {Time} {Series} {Forecasting}}, + url = {http://arxiv.org/abs/1912.09363}, + abstract = {Multi-horizon forecasting problems often contain a complex mix of inputs -- including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed historically -- without any prior information on how they interact with the target. While several deep learning models have been proposed for multi-step prediction, they typically comprise black-box models which do not account for the full range of inputs present in common scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) -- a novel attention-based architecture which combines high-performance multi-horizon forecasting with interpretable insights into temporal dynamics. To learn temporal relationships at different scales, the TFT utilizes recurrent layers for local processing and interpretable self-attention layers for learning long-term dependencies. The TFT also uses specialized components for the judicious selection of relevant features and a series of gating layers to suppress unnecessary components, enabling high performance in a wide range of regimes. On a variety of real-world datasets, we demonstrate significant performance improvements over existing benchmarks, and showcase three practical interpretability use-cases of TFT.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Lim, Bryan and Arik, Sercan O. and Loeff, Nicolas and Pfister, Tomas}, + month = sep, + year = {2020}, + note = {arXiv:1912.09363}, + keywords = {Computer Science - Machine Learning, Statistics - Machine Learning}, + file = {Preprint PDF:/home/alex/Zotero/storage/2R2H34KB/Lim et al. - 2020 - Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/ETDAYW36/1912.html:text/html}, +} + +@misc{nie_time_2023, + title = {A {Time} {Series} is {Worth} 64 {Words}: {Long}-term {Forecasting} with {Transformers}}, + shorttitle = {A {Time} {Series} is {Worth} 64 {Words}}, + url = {http://arxiv.org/abs/2211.14730}, + abstract = {We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Nie, Yuqi and Nguyen, Nam H. and Sinthong, Phanwadee and Kalagnanam, Jayant}, + month = mar, + year = {2023}, + note = {arXiv:2211.14730}, + keywords = {Computer Science - Machine Learning, Computer Science - Artificial Intelligence}, + file = {Preprint PDF:/home/alex/Zotero/storage/DG4ZJCWV/Nie et al. - 2023 - A Time Series is Worth 64 Words Long-term Forecasting with Transformers.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/H6XGVBY6/2211.html:text/html}, +} + +@misc{shao_exploring_2023, + title = {Exploring {Progress} in {Multivariate} {Time} {Series} {Forecasting}: {Comprehensive} {Benchmarking} and {Heterogeneity} {Analysis}}, + shorttitle = {Exploring {Progress} in {Multivariate} {Time} {Series} {Forecasting}}, + url = {http://arxiv.org/abs/2310.06119}, + abstract = {Multivariate Time Series (MTS) widely exists in real-word complex systems, such as traffic and energy systems, making their forecasting crucial for understanding and influencing these systems. Recently, deep learning-based approaches have gained much popularity for effectively modeling temporal and spatial dependencies in MTS, specifically in Long-term Time Series Forecasting (LTSF) and Spatial-Temporal Forecasting (STF). However, the fair benchmarking issue and the choice of technical approaches have been hotly debated in related work. Such controversies significantly hinder our understanding of progress in this field. Thus, this paper aims to address these controversies to present insights into advancements achieved. To resolve benchmarking issues, we introduce BasicTS, a benchmark designed for fair comparisons in MTS forecasting. BasicTS establishes a unified training pipeline and reasonable evaluation settings, enabling an unbiased evaluation of over 30 popular MTS forecasting models on more than 18 datasets. Furthermore, we highlight the heterogeneity among MTS datasets and classify them based on temporal and spatial characteristics. We further prove that neglecting heterogeneity is the primary reason for generating controversies in technical approaches. Moreover, based on the proposed BasicTS and rich heterogeneous MTS datasets, we conduct an exhaustive and reproducible performance and efficiency comparison of popular models, providing insights for researchers in selecting and designing MTS forecasting models.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Shao, Zezhi and Wang, Fei and Xu, Yongjun and Wei, Wei and Yu, Chengqing and Zhang, Zhao and Yao, Di and Jin, Guangyin and Cao, Xin and Cong, Gao and Jensen, Christian S. and Cheng, Xueqi}, + month = oct, + year = {2023}, + note = {arXiv:2310.06119}, + keywords = {Computer Science - Machine Learning, Computer Science - Artificial Intelligence}, + file = {Preprint PDF:/home/alex/Zotero/storage/7EFZ5IT6/Shao et al. - 2023 - Exploring Progress in Multivariate Time Series Forecasting Comprehensive Benchmarking and Heterogen.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/W6RNWBLM/2310.html:text/html}, +} + +@article{zhang_crossformer_2023, + title = {{CROSSFORMER}: {TRANSFORMER} {UTILIZING} {CROSS}- {DIMENSION} {DEPENDENCY} {FOR} {MULTIVARIATE} {TIME} {SERIES} {FORECASTING}}, + abstract = {Recently many deep models have been proposed for multivariate time series (MTS) forecasting. In particular, Transformer-based models have shown great potential because they can capture long-term dependency. However, existing Transformerbased models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (crossdimension dependency), which is critical for MTS forecasting. To fill the gap, we propose Crossformer, a Transformer-based model utilizing cross-dimension dependency for MTS forecasting. In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information. Then the Two-Stage Attention (TSA) layer is proposed to efficiently capture the cross-time and cross-dimension dependency. Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the final forecasting. Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts.}, + language = {en}, + author = {Zhang, Yunhao and Yan, Junchi}, + year = {2023}, + file = {PDF:/home/alex/Zotero/storage/NM9CETJS/Zhang and Yan - 2023 - CROSSFORMER TRANSFORMER UTILIZING CROSS- DIMENSION DEPENDENCY FOR MULTIVARIATE TIME SERIES FORECAST.pdf:application/pdf}, +} + +@misc{noauthor_pytorch-forecasting_2024, + title = {Pytorch-{Forecasting}}, + url = {https://pytorch-forecasting.readthedocs.io/en/stable/}, + urldate = {2024-10-15}, year = {2024}, }