diff --git a/expose/src/main.tex b/expose/src/main.tex index be634f2..37779e3 100644 --- a/expose/src/main.tex +++ b/expose/src/main.tex @@ -13,28 +13,153 @@ % Document \begin{document} - % title section - \title{Exposé: Creating a Generalized Transformer Framework for Predictive Analysis of Health Features from Time-Series Data} + \title{Developing a Data-Driven Framework for Advanced Time-Series Analysis: + Integrating Transformer Architectures with Health Data} \author{Alexander Blank} \date{September 2024} \maketitle - \section{Introduction} - \label{sec:introduction} - Test \cite{zhang2023crossformer} + \section{Introduction}\label{sec:introduction} + Time-series data is a cornerstone of many fields, including finance, health, and climate science, + where it plays a crucial role in forecasting, anomaly detection, and classification tasks. + Predictive models based on time-series data are widely used in applications such as weather forecasting, + stock price prediction, health monitoring, and security systems. + In recent years, transformer-based architectures have shown significant improvements in time-series analysis, + outperforming traditional models like LSTMs in capturing complex temporal patterns. + As the field of time-series analysis rapidly evolves, new models and architectures are continually developed to + meet the growing demands of high-frequency data analysis. - \section{Research Objectives and Questions} + This exposé outlines the development of a modular, data-driven framework designed to integrate advanced transformer + architectures for time-series analysis. + To demonstrate its capabilities, the framework will be applied in a case study focused on high-frequency + ovulation cycle health data, showcasing its adaptability across various model architectures. - \section{Related Work} - \section{Methodology} + \section{Related Work}\label{sec:related-work} + Traditional time-series models like LSTMs and GRUs have been widely used but struggle with capturing + long-term dependencies and complex temporal patterns. + Transformer-based architectures, such as the Temporal Fusion Transformer (TFT)\cite{lim_temporal_2020} + and PatchTST\cite{nie_time_2023}, offer significant improvements, particularly in scalability and interpretability. + These models have become key in addressing challenges inherent to time-series data, + but the field continues to evolve rapidly with new architectures being developed frequently. - \section{Expected Results} + Several existing frameworks, including PyTorch Lightning\cite{lightningai_pytorch_2024}, \ + Amazon’s GluonTS\cite{alexandrov_gluonts_2019}, and Meta's Prophet\cite{taylor_forecasting_2017}, + provide tools for scalable time-series analysis. + However, they remain low-level, requiring significant customization to incorporate new models or are limited to regression tasks. + This highlights the need for a flexible, modular framework that allows seamless integration of + transformer-based architectures for time-series data, particularly in health applications. + + This project aims to bridge this gap by developing a modular, data-driven framework that + simplifies the integration and adaptation of emerging models, focusing on the unique challenges of + health-related time-series data. + + + \section{Research Questions and Objectives}\label{sec:research-questions-and-objectives} + The primary objective of this project is to create a modular framework designed to address the specific challenges + of high-frequency time-series data in health applications, such as temperature monitoring. + The framework will be adaptable to other types of physiological data, like heart rate and sleep patterns, and scalable across different model architectures. + + The key research questions are: + \begin{itemize} + \item \textbf{How can a modular framework be designed to facilitate the integration and adaptation of + transformer-based models for time-series data?} + \item \textbf{What are the unique challenges of applying transformer architectures to health-related + time-series data, and how can the framework address these?} + \item \textbf{How can transformer-based architectures be optimized for high-frequency health data, + such as ovulation cycle monitoring, in terms of performance and interpretability?} + \end{itemize} + + This project aims to provide a flexible and scalable solution for the health tech domain, addressing both + performance and integration challenges in time-series analysis. + + If time allows, the project will also explore the following: + \begin{itemize} + \item \textbf{How can the framework be extended to allow for training of a foundation model on a large dataset + and fine-tuning on a smaller, domain-specific dataset?} + \item \textbf{What are the most effective evaluation metrics for assessing the performance of transformer models + in the context of health-related time-series data?} + \end{itemize} + + + \section{Methodology}\label{sec:methodology} + + The proposed framework for this project is designed with two major components: dataset creation and model + training, with the potential for model evaluation and deployment depending on the outcomes of the case study. + The framework will be implemented in Python and structured to accommodate the specific requirements of + high-frequency health data analysis, particularly for ovulation cycle monitoring. + + \subsection{Dataset Creation} + The dataset creation process will rely on an example MongoDB database as the primary data source, from which health + data will be extracted and prepared for model input. + This phase will involve preprocessing the raw data using \texttt{NumPy} and \texttt{Pandas} to generate + time-series datasets suitable for training transformer models. + The data will be curated to ensure it aligns with the high-frequency nature of ovulation cycle monitoring, and + the preprocessing will include handling missing values, normalizing features, and generating the necessary + time-series sequences. + + \subsection{Model Training} + The model training component will focus on applying transformer-based architectures to the prepared dataset. + Two specific models will be used for the case study: + \begin{itemize} + \item \textbf{Temporal Fusion Transformer (TFT)}: Selected for its ability to provide interpretability while + modeling long-term dependencies in time-series data. + \item \textbf{PatchTST}: Chosen for its state-of-the-art performance in handling large-scale time-series + forecasting tasks. + \end{itemize} + + Both models will be trained using \texttt{PyTorch}, with support from the \texttt{Transformers} library by + Hugging Face, allowing for efficient implementation and fine-tuning of the transformer architectures. + + \subsection{Model Evaluation and Deployment} + The choice of evaluation metrics and deployment strategy will be determined based on the results of the case study. + These components will be incorporated into the framework once the models have been trained and evaluated for + performance and interpretability. + + +% - quickly introduce the two models I want to use (TFT and PatchTST) +% - framework will be build in python. dataset generation will be based on a example mongodb as data source with numpy and pandas. +% - model training will be done with pytorch with the help of the transformers library by huggingface. +% - choice of model evaluation and deployment will be made during the project, depending on the results of the case study. + + + \section{Case Study} + The case study will focus on analyzing high-frequency ovulation cycle data obtained from wearable devices, + which records a measurement every 5 minutes (288 measurements a day). + This dataset offers a unique opportunity to evaluate the ability of transformer-based models to handle + complex temporal dependencies in health-related data. + The Temporal Fusion Transformer (TFT) will be employed to explore model interpretability, particularly in terms of + identifying key features and trends that contribute to predictions, while PatchTST will be used to test the + performance and scalability of the framework in handling large-scale time-series forecasting. + + The expected outcome of this case study is to demonstrate the adaptability of the framework across two dimensions: + interpretability and performance. + The results will provide insight into how these models can be optimized for real-world health applications and may + offer new directions for handling other physiological data, such as heart rate or sleep patterns. + + + \section{Timeline}\label{sec:timeline} + Given the exploratory nature of this project, the timeline will be flexible, focusing on key milestones rather than rigid deadlines. + The project will progress in phases, with time allocated for initial exploration, model development, and case study evaluation. + + \begin{itemize} + \item \textbf{Exploration and Initial Testing:} Early project phases will involve exploring the transformer-based models (TFT and PatchTST), testing their suitability for high-frequency health data, and refining the framework design. + \item \textbf{Dataset Creation and Preprocessing:} Once the framework design is solidified, attention will turn to creating the dataset and ensuring data preprocessing is aligned with the needs of the transformer models. + \item \textbf{Model Training and Evaluation:} The core phase will focus on model training using PyTorch and evaluating both performance and interpretability, with adjustments made iteratively based on results. + \item \textbf{Final Evaluation and Analysis:} The project will conclude with a thorough analysis of model performance and adaptability, followed by preparing the framework for potential further deployment or case study extensions. + \end{itemize} + + This flexible structure allows for adaptation as the project develops, with each phase being iterative and data-driven based on initial findings. - \section{Timeline} \section{Conclusion} + This project aims to develop a modular framework for integrating transformer-based models in time-series analysis, + with a focus on high-frequency health data. + By applying the framework in a case study involving ovulation cycle monitoring, + the project seeks to demonstrate its scalability, performance, and adaptability for real-world health applications. + The outcomes of this project could pave the way for further advancements in health-related time-series analysis, + extending to other physiological data types and domains. % bibliography \printbibliography diff --git a/main.bib b/main.bib index 752c1fc..5b455d1 100644 --- a/main.bib +++ b/main.bib @@ -1,12 +1,160 @@ -%! Author = alex -%! Date = 9/17/24 +@article{zhang_crossformer_2023, + title = {{CROSSFORMER}: {TRANSFORMER} {UTILIZING} {CROSS}- {DIMENSION} {DEPENDENCY} {FOR} {MULTIVARIATE} {TIME} {SERIES} {FORECASTING}}, + abstract = {Recently many deep models have been proposed for multivariate time series (MTS) forecasting. In particular, Transformer-based models have shown great potential because they can capture long-term dependency. However, existing Transformerbased models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (crossdimension dependency), which is critical for MTS forecasting. To fill the gap, we propose Crossformer, a Transformer-based model utilizing cross-dimension dependency for MTS forecasting. In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information. Then the Two-Stage Attention (TSA) layer is proposed to efficiently capture the cross-time and cross-dimension dependency. Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the final forecasting. Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts.}, + language = {en}, + author = {Zhang, Yunhao and Yan, Junchi}, + year = {2023}, + file = {PDF:/home/alex/Zotero/storage/NM9CETJS/Zhang and Yan - 2023 - CROSSFORMER TRANSFORMER UTILIZING CROSS- DIMENSION DEPENDENCY FOR MULTIVARIATE TIME SERIES FORECAST.pdf:application/pdf}, +} -@inproceedings{ - zhang2023crossformer, - title = {Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting}, - author = {Yunhao Zhang and Junchi Yan}, - booktitle = {The Eleventh International Conference on Learning Representations }, - year = {2023}, - url = {https://openreview.net/forum?id=vSVLM2j9eie} -} \ No newline at end of file +@misc{shao_exploring_2023, + title = {Exploring {Progress} in {Multivariate} {Time} {Series} {Forecasting}: {Comprehensive} {Benchmarking} and {Heterogeneity} {Analysis}}, + shorttitle = {Exploring {Progress} in {Multivariate} {Time} {Series} {Forecasting}}, + url = {http://arxiv.org/abs/2310.06119}, + abstract = {Multivariate Time Series (MTS) widely exists in real-word complex systems, such as traffic and energy systems, making their forecasting crucial for understanding and influencing these systems. Recently, deep learning-based approaches have gained much popularity for effectively modeling temporal and spatial dependencies in MTS, specifically in Long-term Time Series Forecasting (LTSF) and Spatial-Temporal Forecasting (STF). However, the fair benchmarking issue and the choice of technical approaches have been hotly debated in related work. Such controversies significantly hinder our understanding of progress in this field. Thus, this paper aims to address these controversies to present insights into advancements achieved. To resolve benchmarking issues, we introduce BasicTS, a benchmark designed for fair comparisons in MTS forecasting. BasicTS establishes a unified training pipeline and reasonable evaluation settings, enabling an unbiased evaluation of over 30 popular MTS forecasting models on more than 18 datasets. Furthermore, we highlight the heterogeneity among MTS datasets and classify them based on temporal and spatial characteristics. We further prove that neglecting heterogeneity is the primary reason for generating controversies in technical approaches. Moreover, based on the proposed BasicTS and rich heterogeneous MTS datasets, we conduct an exhaustive and reproducible performance and efficiency comparison of popular models, providing insights for researchers in selecting and designing MTS forecasting models.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Shao, Zezhi and Wang, Fei and Xu, Yongjun and Wei, Wei and Yu, Chengqing and Zhang, Zhao and Yao, Di and Jin, Guangyin and Cao, Xin and Cong, Gao and Jensen, Christian S. and Cheng, Xueqi}, + month = oct, + year = {2023}, + note = {arXiv:2310.06119}, + keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning}, + file = {Preprint PDF:/home/alex/Zotero/storage/7EFZ5IT6/Shao et al. - 2023 - Exploring Progress in Multivariate Time Series Forecasting Comprehensive Benchmarking and Heterogen.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/W6RNWBLM/2310.html:text/html}, +} + +@misc{nie_time_2023, + title = {A {Time} {Series} is {Worth} 64 {Words}: {Long}-term {Forecasting} with {Transformers}}, + shorttitle = {A {Time} {Series} is {Worth} 64 {Words}}, + url = {http://arxiv.org/abs/2211.14730}, + abstract = {We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Nie, Yuqi and Nguyen, Nam H. and Sinthong, Phanwadee and Kalagnanam, Jayant}, + month = mar, + year = {2023}, + note = {arXiv:2211.14730}, + keywords = {Computer Science - Artificial Intelligence, Computer Science - Machine Learning}, + file = {Preprint PDF:/home/alex/Zotero/storage/DG4ZJCWV/Nie et al. - 2023 - A Time Series is Worth 64 Words Long-term Forecasting with Transformers.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/H6XGVBY6/2211.html:text/html}, +} + +@misc{lim_temporal_2020, + title = {Temporal {Fusion} {Transformers} for {Interpretable} {Multi}-horizon {Time} {Series} {Forecasting}}, + url = {http://arxiv.org/abs/1912.09363}, + abstract = {Multi-horizon forecasting problems often contain a complex mix of inputs -- including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed historically -- without any prior information on how they interact with the target. While several deep learning models have been proposed for multi-step prediction, they typically comprise black-box models which do not account for the full range of inputs present in common scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) -- a novel attention-based architecture which combines high-performance multi-horizon forecasting with interpretable insights into temporal dynamics. To learn temporal relationships at different scales, the TFT utilizes recurrent layers for local processing and interpretable self-attention layers for learning long-term dependencies. The TFT also uses specialized components for the judicious selection of relevant features and a series of gating layers to suppress unnecessary components, enabling high performance in a wide range of regimes. On a variety of real-world datasets, we demonstrate significant performance improvements over existing benchmarks, and showcase three practical interpretability use-cases of TFT.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Lim, Bryan and Arik, Sercan O. and Loeff, Nicolas and Pfister, Tomas}, + month = sep, + year = {2020}, + note = {arXiv:1912.09363}, + keywords = {Computer Science - Machine Learning, Statistics - Machine Learning}, + file = {Preprint PDF:/home/alex/Zotero/storage/2R2H34KB/Lim et al. - 2020 - Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/ETDAYW36/1912.html:text/html}, +} + +@article{hochreiter_long_1997, + title = {Long {Short}-{Term} {Memory}}, + volume = {9}, + issn = {0899-7667, 1530-888X}, + url = {https://direct.mit.edu/neco/article/9/8/1735-1780/6109}, + doi = {10.1162/neco.1997.9.8.1735}, + abstract = {Learningtostoreinformationoverextendedtimeintervalsviarecurrentbackpropagation takesaverylongtime,mostlyduetoinsu cient,decayingerrorbackow.Webrieyreview Hochreiter's1991analysisofthisproblem,thenaddressitbyintroducinganovel,e cient, gradient-basedmethodcalled{\textbackslash}LongShort-TermMemory"(LSTM).Truncatingthegradient wherethisdoesnotdoharm,LSTMcanlearntobridgeminimaltimelagsinexcessof1000 discretetimestepsbyenforcingconstanterrorowthrough{\textbackslash}constanterrorcarrousels"within specialunits.Multiplicativegateunitslearntoopenandcloseaccesstotheconstanterror ow.LSTMislocalinspaceandtime;itscomputationalcomplexitypertimestepandweight isO(1).Ourexperimentswitharticialdatainvolvelocal,distributed,real-valued,andnoisy patternrepresentations.IncomparisonswithRTRL,BPTT,RecurrentCascade-Correlation, Elmannets,andNeuralSequenceChunking,LSTMleadstomanymoresuccessfulruns,and learnsmuchfaster.LSTMalsosolvescomplex,articiallongtimelagtasksthathavenever beensolvedbypreviousrecurrentnetworkalgorithms.}, + language = {en}, + number = {8}, + urldate = {2024-10-10}, + journal = {Neural Computation}, + author = {Hochreiter, Sepp and Schmidhuber, Jürgen}, + month = nov, + year = {1997}, + pages = {1735--1780}, + file = {PDF:/home/alex/Zotero/storage/CZSV2ASE/Hochreiter and Schmidhuber - 1997 - Long Short-Term Memory.pdf:application/pdf}, +} + +@misc{cho_learning_2014, + title = {Learning {Phrase} {Representations} using {RNN} {Encoder}-{Decoder} for {Statistical} {Machine} {Translation}}, + url = {http://arxiv.org/abs/1406.1078}, + abstract = {In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder-Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.}, + urldate = {2024-10-10}, + publisher = {arXiv}, + author = {Cho, Kyunghyun and Merrienboer, Bart van and Gulcehre, Caglar and Bahdanau, Dzmitry and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua}, + month = sep, + year = {2014}, + note = {arXiv:1406.1078}, + keywords = {Computer Science - Computation and Language, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing, Statistics - Machine Learning}, + file = {Preprint PDF:/home/alex/Zotero/storage/E8WMK2IN/Cho et al. - 2014 - Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/6PTCL8LW/1406.html:text/html}, +} + +@incollection{hutter_general_2021, + address = {Cham}, + title = {A {General} {Machine} {Learning} {Framework} for {Survival} {Analysis}}, + volume = {12459}, + isbn = {978-3-030-67663-6 978-3-030-67664-3}, + url = {https://link.springer.com/10.1007/978-3-030-67664-3_10}, + language = {en}, + urldate = {2024-10-14}, + booktitle = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}}, + publisher = {Springer International Publishing}, + author = {Bender, Andreas and Rügamer, David and Scheipl, Fabian and Bischl, Bernd}, + editor = {Hutter, Frank and Kersting, Kristian and Lijffijt, Jefrey and Valera, Isabel}, + year = {2021}, + doi = {10.1007/978-3-030-67664-3_10}, + note = {Series Title: Lecture Notes in Computer Science}, + pages = {158--173}, + file = {Submitted Version:/home/alex/Zotero/storage/WQIHZ7IP/Bender et al. - 2021 - A General Machine Learning Framework for Survival Analysis.pdf:application/pdf}, +} + +@book{hutter_machine_2021, + address = {Cham}, + series = {Lecture {Notes} in {Computer} {Science}}, + title = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}: {European} {Conference}, {ECML} {PKDD} 2020, {Ghent}, {Belgium}, {September} 14–18, 2020, {Proceedings}, {Part} {III}}, + volume = {12459}, + copyright = {https://www.springernature.com/gp/researchers/text-and-data-mining}, + isbn = {978-3-030-67663-6 978-3-030-67664-3}, + shorttitle = {Machine {Learning} and {Knowledge} {Discovery} in {Databases}}, + url = {https://link.springer.com/10.1007/978-3-030-67664-3}, + language = {en}, + urldate = {2024-10-14}, + publisher = {Springer International Publishing}, + editor = {Hutter, Frank and Kersting, Kristian and Lijffijt, Jefrey and Valera, Isabel}, + year = {2021}, + doi = {10.1007/978-3-030-67664-3}, + file = {Submitted Version:/home/alex/Zotero/storage/JMVJMLJ5/Hutter et al. - 2021 - Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD 2020, Ghent, B.pdf:application/pdf}, +} + +@misc{taylor_forecasting_2017, + title = {Forecasting at scale}, + copyright = {http://creativecommons.org/licenses/by/4.0/}, + url = {https://peerj.com/preprints/3190v2}, + doi = {10.7287/peerj.preprints.3190v2}, + abstract = {Forecasting is a common data science task that helps organizations with capacity planning, goal setting, and anomaly detection. Despite its importance, there are serious challenges associated with producing reliable and high quality forecasts –especially when there are a variety of time series and analysts with expertise in time series modeling are relatively rare. To address these challenges, we describe a practical approach to forecasting “at scale” that combines configurable models with analyst-in-the-loop performance analysis. We propose a modular regression model with interpretable parameters that can be intuitively adjusted by analysts with domain knowledge about the time series. We describe performance analyses to compare and evaluate forecasting procedures, and automatically flag forecasts for manual review and adjustment. Tools that help analysts to use their expertise most effectively enable reliable, practical forecasting of business time series.}, + language = {en}, + urldate = {2024-10-14}, + publisher = {PeerJ Preprints}, + author = {Taylor, Sean J and Letham, Benjamin}, + month = sep, + year = {2017}, + file = {PDF:/home/alex/Zotero/storage/GK5AIG2V/Taylor and Letham - 2017 - Forecasting at scale.pdf:application/pdf}, +} + +@misc{alexandrov_gluonts_2019, + title = {{GluonTS}: {Probabilistic} {Time} {Series} {Models} in {Python}}, + shorttitle = {{GluonTS}}, + url = {http://arxiv.org/abs/1906.05264}, + abstract = {We introduce Gluon Time Series (GluonTS, available at https://gluon-ts.mxnet.io), a library for deep-learning-based time series modeling. GluonTS simplifies the development of and experimentation with time series models for common tasks such as forecasting or anomaly detection. It provides all necessary components and tools that scientists need for quickly building new models, for efficiently running and analyzing experiments and for evaluating model accuracy.}, + urldate = {2024-10-14}, + publisher = {arXiv}, + author = {Alexandrov, Alexander and Benidis, Konstantinos and Bohlke-Schneider, Michael and Flunkert, Valentin and Gasthaus, Jan and Januschowski, Tim and Maddix, Danielle C. and Rangapuram, Syama and Salinas, David and Schulz, Jasper and Stella, Lorenzo and Türkmen, Ali Caner and Wang, Yuyang}, + month = jun, + year = {2019}, + note = {arXiv:1906.05264}, + keywords = {Computer Science - Machine Learning, Statistics - Machine Learning}, + file = {Preprint PDF:/home/alex/Zotero/storage/JP9K74A8/Alexandrov et al. - 2019 - GluonTS Probabilistic Time Series Models in Python.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/RJYSBT29/1906.html:text/html}, +} + +@misc{lightningai_pytorch_2024, + title = {{PyTorch} {Lightning}}, + url = {https://www.pytorchlightning.ai}, + urldate = {2024-10-14}, + author = {lightning.ai}, + year = {2024}, +} diff --git a/notes.txt b/notes.txt new file mode 100644 index 0000000..525d58f --- /dev/null +++ b/notes.txt @@ -0,0 +1,3 @@ +- introduce that the goal is not to cover the models, but to use the two "types" of models as an example covering the two +example cases of a next token prediction and a classification task +- highlight the modular design of the framework and its adaptability to new and future models and data structures as input \ No newline at end of file diff --git a/sources/1912.09363v3.pdf b/sources/1912.09363v3.pdf new file mode 100644 index 0000000..815644a Binary files /dev/null and b/sources/1912.09363v3.pdf differ