new expose mk1
This commit is contained in:
+73
-5
@@ -7,7 +7,7 @@
|
||||
% Packages
|
||||
\usepackage{amsmath}
|
||||
% biber bibliography
|
||||
\usepackage[backend=biber]{biblatex}
|
||||
\usepackage[style=ieee, backend=biber]{biblatex}
|
||||
\addbibresource{../../main.bib}
|
||||
|
||||
% Document
|
||||
@@ -20,24 +20,92 @@
|
||||
|
||||
|
||||
\section{Introduction}\label{sec:introduction}
|
||||
Women's health is a topic that has been neglected for a long time.
|
||||
The menstrual cycle and its corresponding ovulation are essential for the reproductive health,
|
||||
Women's health is a topic that has been neglected for a long time, even though it has direct implications for
|
||||
more than half of the world's population.
|
||||
The menstrual cycle and its corresponding ovulation are essential for reproductive health,
|
||||
family planning and natural contraception.
|
||||
|
||||
v
|
||||
Part of the challenge lies in the lack of comprehensive datasets, but even more critically,
|
||||
in the incomplete understanding of the underlying physiological mechanisms.
|
||||
New advances in machine learning have shown, that there is potential in predicting the menstrual cycle and ovulation
|
||||
with high accuracy based on physiological data, such as basal body temperature,
|
||||
heart rate or cervicovaginal fluid\cite{noauthor_cervicovaginal_2005, luo_detection_2020, masuda_machine_2025, yu_tracking_2022}.
|
||||
A problem with the current landscape of research is often the lack of interpretability of the models as well as insufficient
|
||||
data to train the models.
|
||||
This lack of understanding is compounded by the opacity of many machine learning models, making their predictions difficult to trust and validate in clinical settings.
|
||||
In female health, a wrong prediction can have severe consequences, such as unwanted pregnancies, health risks or even death.
|
||||
This thesis aims to improve upon that by using a large dataset of body core temperature and markers to train an
|
||||
interpretable model based on attention mechanisms to predict ovulation and to identify the most important predictors for ovulation.
|
||||
|
||||
|
||||
\section{Related Work}\label{sec:related-work}
|
||||
Research on fertility prediction goes back as far as the 1990s, where the first purely statistical models were
|
||||
developed to predict both the ovulation and the fertility over time\cite{royston_identifying_1991}.
|
||||
Since then, there has been a long pause in research until the rise of machine learning in the late 2010s.
|
||||
Both data quality and quantity have improved significantly, which has led to more accurate models\cite{maman_prediction_2023, noauthor_pdf_2024}.
|
||||
However, since the female menstrual cycle is a biological process, prone to noise and irregularities, the models
|
||||
often lack generalizability and interpretability, especially for women with irregular cycles.
|
||||
There is a variety of biological features that can be used to predict ovulation, but most of it have shown to be
|
||||
impractical for large scale use, as they require daily blood samples or ultrasound measurements.
|
||||
The most promising are models based on temperature data, as it is a quantity that can be recorded on a large scale
|
||||
without intrusive methods (ear-piece or vaginal sensor monitoring)\cite{alexander_fertilitatsmonitoring_2014, luo_detection_2020}.
|
||||
However, body temperature alone is an imperfect predictor of ovulation,
|
||||
as it is influenced by external factors such as stress, sport, illness, and sleep disturbances.
|
||||
While external measurement (e.g.\ skin or earpiece sensors) are even less reliable,
|
||||
even vaginal temperature measurements can be influenced by these factors.
|
||||
This is where this research aims to improve upon, by incorporating more predictors, such as markers for illness, sport,
|
||||
stress or similar things.
|
||||
|
||||
|
||||
\section{Research Questions and Objectives}\label{sec:research-questions-and-objectives}
|
||||
The main research objective is to create a model that can predict ovulation and fertile days with high accuracy
|
||||
while offering a high level of interpretability in its underlying attention mechanisms.
|
||||
|
||||
\begin{itemize}
|
||||
\item Can attention-based models improve the prediction of ovulation?
|
||||
\item Can attention-based models improve the interpretability of the female menstrual cycle?
|
||||
\item What are the most important predictors for ovulation, both in space and time?
|
||||
\item How does an attention-based model compare to other models in terms of accuracy and interpretability?
|
||||
\item Can we account for external influences (e.g. stress, illness, sport) to improve prediction robustness?
|
||||
\item Optional: Use a non-interpretable transformer model to see impact of interpretability on prediction accuracy.
|
||||
\end{itemize}
|
||||
|
||||
|
||||
\section{Methodology}\label{sec:methodology}
|
||||
The research will be based on an extensive dataset, generously provided by the FemTech company\cite{noauthor_vivosens_nodate},
|
||||
which incorporates 50,000 cycles with temperature measurements every 5 minutes by a intravaginal sensor ring.
|
||||
The dataset contains a wide variety of cycle types and predispositions, including regular and irregular cycles,
|
||||
PCOS, endometriosis, and other conditions.
|
||||
Additionally, the dataset contains both user generated and automatically generated markers for cervical fluid, menstruation,
|
||||
mood and other relevant factors.
|
||||
|
||||
The model will be based on a transformer architecture, which has shown to be very effective in time series prediction\cite{vaswani_attention_2017}.
|
||||
This architecture is particularly suited for ovulation prediction because it can capture dependencies across different time points—identifying,
|
||||
for instance, how temperature fluctuations in the follicular phase influence predictions in the luteal phase.
|
||||
Transformer-based models have been further improved for specific use cases.
|
||||
Here we will use a model that has shown to be effective in time-series analysis with the addition of an
|
||||
interpretable attention mechanism, called the \emph{Temporal Fusion Transformer}(TFT)\cite{lim_temporal_2020}.
|
||||
TFT is specifically designed for multivariate time-series forecasting, allowing it to incorporate static covariates
|
||||
(e.g., age) alongside dynamic features (e.g., temperature, stress markers) to improve prediction robustness.
|
||||
By training an interpretable attention mechanism, we will identify the most crucial factors for ovulation and
|
||||
fertility prediction.
|
||||
These factors will then be presented to a reproductive health expert to validate and interpret the findings.
|
||||
Given the sensitivity of fertility prediction, particular attention
|
||||
will be paid to data privacy and ethical considerations in model deployment.
|
||||
|
||||
The model will be evaluated on its accuracy and interpretability and compared to other models in the field such as
|
||||
OvulaRing's previous methods\cite{alexander_fertilitatsmonitoring_2014} other machine learning models (LSTM, GRU, etc.)
|
||||
and simpler statistical models ((S)ARIMA, etc.).
|
||||
Next to standard metrics, such as accuracy, precision, recall, F1-score, we will also use fertility-specific metrics,
|
||||
that incorporate the probability curve of the fertile window according to \citeauthor{dunson_day-specific_1999}\cite{dunson_day-specific_1999},
|
||||
and the severity of the consequences of a wrong prediction.
|
||||
|
||||
|
||||
\section{Conclusion}
|
||||
The research aims to improve the prediction of ovulation and fertility by using an interpretable attention-based model.
|
||||
The model will be trained on a large dataset of intravaginal temperature measurements and set of time-varying and constant features.
|
||||
The model will be evaluated on its accuracy and interpretability and compared to other models in the field.
|
||||
The most important predictors for ovulation will be identified and presented to a reproductive health expert for validation.
|
||||
|
||||
% bibliography
|
||||
\printbibliography
|
||||
|
||||
|
||||
@@ -750,3 +750,271 @@ Publisher: {JMIR} Publications Inc., Toronto, Canada},
|
||||
note = {Publisher: Elsevier},
|
||||
file = {Snapshot:/home/alex/Zotero/storage/FYI9GPUM/S0015028216484360.html:text/html},
|
||||
}
|
||||
|
||||
@inproceedings{regidor_identifizierung_2018,
|
||||
title = {Identifizierung und Vorhersage des fertilen Fensters des weiblichen Zyklus mit einem neuen web basierten Medizinprodukt ({OvulaRing}®).},
|
||||
volume = {78},
|
||||
rights = {Georg Thieme Verlag {KG} Stuttgart · New York},
|
||||
url = {https://www.thieme-connect.com/products/ejournals/html/10.1055/s-0038-1671278},
|
||||
doi = {10.1055/s-0038-1671278},
|
||||
abstract = {Thieme E-Books \& E-Journals},
|
||||
pages = {P 23},
|
||||
booktitle = {Geburtshilfe und Frauenheilkunde},
|
||||
publisher = {Georg Thieme Verlag {KG}},
|
||||
author = {Regidor, P. A. and Alexander, H.},
|
||||
urldate = {2025-02-21},
|
||||
date = {2018-09-20},
|
||||
langid = {german},
|
||||
note = {{ISSN}: 0016-5751},
|
||||
keywords = {Präsidentin der {DGGG} e.V.: Prof. Dr. Birgit Seelbach-Göbel{\textless}/conf-president{\textgreater}{\textless}/conference{\textgreater}},
|
||||
file = {Snapshot:/home/alex/Zotero/storage/DW6578ZM/s-0038-1671278.html:text/html},
|
||||
}
|
||||
|
||||
@article{alexander_fertilitatsmonitoring_2014,
|
||||
title = {Fertilitätsmonitoring mit vaginalem Biosensor ({OvulaRing}©)},
|
||||
volume = {74},
|
||||
issn = {0016-5751},
|
||||
url = {https://www.thieme-connect.com/products/ejournals/abstract/10.1055/s-0034-1388603},
|
||||
doi = {10.1055/s-0034-1388603},
|
||||
abstract = {Thieme E-Books \& E-Journals},
|
||||
pages = {FV\_08\_05},
|
||||
journaltitle = {Geburtshilfe und Frauenheilkunde},
|
||||
author = {Alexander, H. and Kaczmarczyk, M. and Pretzsch, G. and Kersken, T. and Puschmann, D. and Schiwek, E. and Goeckenjan, M.},
|
||||
urldate = {2025-02-21},
|
||||
date = {2014-09-05},
|
||||
langid = {german},
|
||||
keywords = {60. Kongress der Deutschen Gesellschaft für Gynäkologie und Geburtshilfe},
|
||||
file = {Snapshot:/home/alex/Zotero/storage/HPL6XYJW/s-0034-1388603.html:text/html},
|
||||
}
|
||||
|
||||
@article{regidor_identification_2018,
|
||||
title = {Identification and prediction of the fertile window with a new web-based medical device using a vaginal biosensor for measuring the circadian and circamensual core body temperature},
|
||||
volume = {34},
|
||||
issn = {0951-3590},
|
||||
url = {https://doi.org/10.1080/09513590.2017.1390737},
|
||||
doi = {10.1080/09513590.2017.1390737},
|
||||
abstract = {Fertility awareness-based ({FAB}) methods represent a term that includes all family planning methods that are based on the identification of the fertile window. They are based on the woman’s observation of physiological signs of the fertile and infertile phases of the menstrual cycle. The first approach consists basically in symptothermal methods accompanied by cervical mucus measurements and clinical menstrual cycling data recording. The second most often used methods are the urinary measurement of E3G and luteinizing hormone ({LH}) with a personalized computer system. Hence these systems lack the efficacy of the continuous circadian and circamensual measurement of the core body temperature. Only this approach enables the accurate detection of the ovulation during the fertile window. A new medical device called {OvulaRing} has been developed to fill this gap. In the present study, the system and its first clinical results are presented. {OvulaRing} is a medical device used just like a tampon. The device is a vaginal ring of evatane that contains an integrated biosensor. This sensor measures continuously every 5 min the core body temperature throughout the entire cycle. This device allows a circadian and circamensual intravaginal exact measurement. With this system, 288 measurements are created per day. The system can detect retrospectively and predict prospectively the fertile window of the users. One hundred and fifty eight women aged between 18 and 45 years used this medical device in an open non-randomized clinical study for 15 months. A total of 470 cycles could be recorded and were able for analysis. By the same time in a subgroup of patients, hormonal assessments of {LH}, follicle-stimulating hormone, estradiol and progesterone as well as vaginal ultrasound were performed in parallel between the 9th and the 36th day of the cycle. The validation error due to software errors was 0.89\% for the retrospective analysis; that means that the accuracy for the detection of the ovulation was 99.11\%. Accuracy of 88.8\% for a window of 3 days before ovulation, the day of ovulation and the 3 days after ovulation was achieved for the prospective analysis. In the subgroup of woman with recorded pregnancies, it could be shown that after 3.79 months of use (median) pregnancies were observed. In 67.72\% in up to 3 months, in 16.36\% between 3 and 6 months of use, in 7.27\% between 7 and 9 months, in 5.45\% between 10 and 12 months and in 1.82\% between 13 and 15 months of use of the system. With this new web-based system, a precise determination of the fertile window even in women with ultralong cycles ({\textgreater}35 days) could be detected independently of their personal live circumstances. Exact determination of the fertile window is herewith possible so that {OvulaRing} represents an evolution in the {FAB} method for the cycle diagnosis of women with regular, irregular or anovulatory menstrual cycles.},
|
||||
pages = {256--260},
|
||||
number = {3},
|
||||
journaltitle = {Gynecological Endocrinology},
|
||||
author = {Regidor, Pedro-Antonio and Kaczmarczyk, Marta and Schiweck, Esther and Goeckenjan-Festag, Maren and Alexander, Henry},
|
||||
urldate = {2025-02-21},
|
||||
date = {2018-03-04},
|
||||
pmid = {29082805},
|
||||
note = {Publisher: Taylor \& Francis
|
||||
\_eprint: https://doi.org/10.1080/09513590.2017.1390737},
|
||||
keywords = {central nervous system, circadian rhythm, circamensual rhythm, core body temperature, fertile window, Infertility, vagina},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/ITD68HTW/Regidor et al. - 2018 - Identification and prediction of the fertile window with a new web-based medical device using a vagi.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@article{goeckenjan_continuous_2020,
|
||||
title = {Continuous Body Temperature Monitoring to Improve the Diagnosis of Female Infertility},
|
||||
volume = {80},
|
||||
rights = {Georg Thieme Verlag {KG} Stuttgart · New York},
|
||||
issn = {0016-5751},
|
||||
url = {https://www.thieme-connect.com/products/ejournals/html/10.1055/a-1191-7888},
|
||||
doi = {10.1055/a-1191-7888},
|
||||
abstract = {Introduction Ovulatory dysfunction is a major cause of female infertility. We evaluated the use of continuous body temperature monitoring with a vaginal biosensor to improve
|
||||
standard diagnostic procedures for determining ovulatory dysfunction.
|
||||
|
||||
Material and Methods This prospective interventional study was performed in a reproductive medicine department of a university hospital. The menstrual cycles of 51 women with
|
||||
infertility were monitored and analysed using three different strategies: sonographic and hormonal assessment (standard approach), continuous core body temperature measurement and analysis
|
||||
using the algorithm of {OvulaRing}, and lowest daily body temperature measurement monitored with a vaginal biosensor and analysed based on the body temperature curves used in natural family
|
||||
planning.
|
||||
|
||||
Results Statistically significant differences were found in the temperature curves of women with luteal phase deficiency and polycystic ovary syndrome compared to women with normal
|
||||
menstrual cycles. The analysis of individual cyclofertilograms can be used to detect cycle phases and estimate the date of ovulation.
|
||||
|
||||
Conclusions Continuous body temperature monitoring with a vaginal biosensor can improve the standard diagnostic procedures used to determine ovulatory dysfunction, especially if
|
||||
dysfunction is due to luteal phase deficiency and polycystic ovary syndrome. Analysis of the lowest daily body temperature combined with the basal body temperature measurements used in
|
||||
fertility awareness methods may be equieffective to continuous body temperature measurements with {OvulaRing}. The results of this study show that a revised diagnostic approach using fewer
|
||||
hormonal assessments combined with continuous body temperature monitoring can reduce the number of appointments in an infertility clinic as well as the costs.},
|
||||
pages = {702--712},
|
||||
journaltitle = {Geburtshilfe und Frauenheilkunde},
|
||||
author = {Goeckenjan, Maren and Schiwek, Esther and Wimberger, Pauline},
|
||||
urldate = {2025-02-21},
|
||||
date = {2020-07-14},
|
||||
langid = {english},
|
||||
note = {Publisher: Georg Thieme Verlag {KG}},
|
||||
keywords = {infertility, Key words
|
||||
fertility awareness, luteal phase deficiency, polycystic ovary syndrome, vaginal biosensor},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/QKPIJD23/Goeckenjan et al. - 2020 - Continuous Body Temperature Monitoring to Improve the Diagnosis of Female Infertility.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@online{noauthor_zyklus-apps_nodate,
|
||||
title = {Zyklus-Apps zur Verhütung – sicher oder Gesellschaftsspiel? - {ProQuest}},
|
||||
url = {https://www.proquest.com/openview/739071fff0941b30f3a5d33b56259c60/1?pq-origsite=gscholar&cbl=6629261},
|
||||
shorttitle = {Zyklus-Apps zur Verhütung – sicher oder Gesellschaftsspiel?},
|
||||
abstract = {Explore millions of resources from scholarly journals, books, newspapers, videos and more, on the {ProQuest} Platform.},
|
||||
urldate = {2025-02-21},
|
||||
langid = {english},
|
||||
file = {Snapshot:/home/alex/Zotero/storage/QHYUJU9G/1.html:text/html},
|
||||
}
|
||||
|
||||
@article{lyzwinski_innovative_2024,
|
||||
title = {Innovative Approaches to Menstruation and Fertility Tracking Using Wearable Reproductive Health Technology: Systematic Review},
|
||||
volume = {26},
|
||||
url = {https://www.jmir.org/2024/1/e45139},
|
||||
doi = {10.2196/45139},
|
||||
shorttitle = {Innovative Approaches to Menstruation and Fertility Tracking Using Wearable Reproductive Health Technology},
|
||||
abstract = {Background: Emerging digital health technology has moved into the reproductive health market for female individuals. In the past, mobile health apps have been used to monitor the menstrual cycle using manual entry. New technological trends involve the use of wearable devices to track fertility by assessing physiological changes such as temperature, heart rate, and respiratory rate.
|
||||
Objective: The primary aims of this study are to review the types of wearables that have been developed and evaluated for menstrual cycle tracking and to examine whether they may detect changes in the menstrual cycle in female individuals. Another aim is to review whether these devices are effective for tracking various stages in the menstrual cycle including ovulation and menstruation. Finally, the secondary aim is to assess whether the studies have validated their findings by reporting accuracy and sensitivity.
|
||||
Methods: A review of {PubMed} or {MEDLINE} was undertaken to evaluate wearable devices for their effectiveness in predicting fertility and differentiating between the different stages of the menstrual cycle.
|
||||
Results: Fertility cycle–tracking wearables include devices that can be worn on the wrists, on the fingers, intravaginally, and inside the ear. Wearable devices hold promise for predicting different stages of the menstrual cycle including the fertile window and may be used by female individuals as part of their reproductive health. Most devices had high accuracy for detecting fertility and were able to differentiate between the luteal phase (early and late), fertile window, and menstruation by assessing changes in heart rate, heart rate variability, temperature, and respiratory rate.
|
||||
Conclusions: More research is needed to evaluate consumer perspectives on reproductive technology for monitoring fertility, and ethical issues around the privacy of digital data need to be addressed. Additionally, there is also a need for more studies to validate and confirm this research, given its scarcity, especially in relation to changes in respiratory rate as a proxy for reproductive cycle staging.},
|
||||
pages = {e45139},
|
||||
number = {1},
|
||||
journaltitle = {Journal of Medical Internet Research},
|
||||
author = {Lyzwinski, Lynnette and Elgendi, Mohamed and Menon, Carlo},
|
||||
urldate = {2025-02-21},
|
||||
date = {2024-02-15},
|
||||
note = {Company: Journal of Medical Internet Research
|
||||
Distributor: Journal of Medical Internet Research
|
||||
Institution: Journal of Medical Internet Research
|
||||
Label: Journal of Medical Internet Research
|
||||
Publisher: {JMIR} Publications Inc., Toronto, Canada},
|
||||
file = {Full Text:/home/alex/Zotero/storage/IP23WZLE/Lyzwinski et al. - 2024 - Innovative Approaches to Menstruation and Fertility Tracking Using Wearable Reproductive Health Tech.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/TJN3WV3I/e45139.html:text/html},
|
||||
}
|
||||
|
||||
@misc{serrano_is_2019-1,
|
||||
title = {Is Attention Interpretable?},
|
||||
url = {http://arxiv.org/abs/1906.03731},
|
||||
doi = {10.48550/arXiv.1906.03731},
|
||||
abstract = {Attention mechanisms have recently boosted performance on a range of {NLP} tasks. Because attention layers explicitly weight input components' representations, it is also often assumed that attention can be used to identify information that models found important (e.g., specific contextualized word tokens). We test whether that assumption holds by manipulating attention weights in already-trained text classification models and analyzing the resulting differences in their predictions. While we observe some ways in which higher attention weights correlate with greater impact on model predictions, we also find many ways in which this does not hold, i.e., where gradient-based rankings of attention weights better predict their effects than their magnitudes. We conclude that while attention noisily predicts input components' overall importance to a model, it is by no means a fail-safe indicator.},
|
||||
number = {{arXiv}:1906.03731},
|
||||
publisher = {{arXiv}},
|
||||
author = {Serrano, Sofia and Smith, Noah A.},
|
||||
urldate = {2025-02-21},
|
||||
date = {2019-06-09},
|
||||
eprinttype = {arxiv},
|
||||
eprint = {1906.03731 [cs]},
|
||||
keywords = {Computer Science - Computation and Language},
|
||||
file = {Preprint PDF:/home/alex/Zotero/storage/DFZ28RG8/Serrano and Smith - 2019 - Is Attention Interpretable.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/63B9XS2Y/1906.html:text/html},
|
||||
}
|
||||
|
||||
@inproceedings{choi_retain_2016,
|
||||
title = {{RETAIN}: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism},
|
||||
volume = {29},
|
||||
url = {https://proceedings.neurips.cc/paper/2016/hash/231141b34c82aa95e48810a9d1b33a79-Abstract.html},
|
||||
shorttitle = {{RETAIN}},
|
||||
abstract = {Accuracy and interpretability are two dominant features of successful predictive models. Typically, a choice must be made in favor of complex black box models such as recurrent neural networks ({RNN}) for accuracy versus less accurate but more interpretable traditional models such as logistic regression. This tradeoff poses challenges in medicine where both accuracy and interpretability are important. We addressed this challenge by developing the {REverse} Time {AttentIoN} model ({RETAIN}) for application to Electronic Health Records ({EHR}) data. {RETAIN} achieves high accuracy while remaining clinically interpretable and is based on a two-level neural attention model that detects influential past visits and significant clinical variables within those visits (e.g. key diagnoses). {RETAIN} mimics physician practice by attending the {EHR} data in a reverse time order so that recent clinical visits are likely to receive higher attention. {RETAIN} was tested on a large health system {EHR} dataset with 14 million visits completed by 263K patients over an 8 year period and demonstrated predictive accuracy and computational scalability comparable to state-of-the-art methods such as {RNN}, and ease of interpretability comparable to traditional models.},
|
||||
booktitle = {Advances in Neural Information Processing Systems},
|
||||
publisher = {Curran Associates, Inc.},
|
||||
author = {Choi, Edward and Bahadori, Mohammad Taha and Sun, Jimeng and Kulas, Joshua and Schuetz, Andy and Stewart, Walter},
|
||||
urldate = {2025-02-21},
|
||||
date = {2016},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/XQLMYHUU/Choi et al. - 2016 - RETAIN An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@article{kitada_attention_2021,
|
||||
title = {Attention Meets Perturbations: Robust and Interpretable Attention With Adversarial Training},
|
||||
volume = {9},
|
||||
issn = {2169-3536},
|
||||
url = {https://ieeexplore.ieee.org/abstract/document/9467291},
|
||||
doi = {10.1109/ACCESS.2021.3093456},
|
||||
shorttitle = {Attention Meets Perturbations},
|
||||
abstract = {Although attention mechanisms have been applied to a variety of deep learning models and have been shown to improve the prediction performance, it has been reported to be vulnerable to perturbations to the mechanism. To overcome the vulnerability to perturbations in the mechanism, we are inspired by adversarial training ({AT}), which is a powerful regularization technique for enhancing the robustness of the models. In this paper, we propose a general training technique for natural language processing tasks, including {AT} for attention (Attention {AT}) and more interpretable {AT} for attention (Attention {iAT}). The proposed techniques improved the prediction performance and the model interpretability by exploiting the mechanisms with {AT}. In particular, Attention {iAT} boosts those advantages by introducing adversarial perturbation, which enhances the difference in the attention of the sentences. Evaluation experiments with ten open datasets revealed that {AT} for attention mechanisms, especially Attention {iAT}, demonstrated (1) the best performance in nine out of ten tasks and (2) more interpretable attention (i.e., the resulting attention correlated more strongly with gradient-based word importance) for all tasks. Additionally, the proposed techniques are (3) much less dependent on perturbation size in {AT}.},
|
||||
pages = {92974--92985},
|
||||
journaltitle = {{IEEE} Access},
|
||||
author = {Kitada, Shunsuke and Iyatomi, Hitoshi},
|
||||
urldate = {2025-02-21},
|
||||
date = {2021},
|
||||
note = {Conference Name: {IEEE} Access},
|
||||
keywords = {adversarial training, attention mechanism, binary classification, interpretability, Knowledge discovery, natural language inference, Natural language processing, Perturbation methods, Predictive models, question answering, Robustness, Solid modeling, Task analysis, Training},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/BUXFCXRU/Kitada and Iyatomi - 2021 - Attention Meets Perturbations Robust and Interpretable Attention With Adversarial Training.pdf:application/pdf;IEEE Xplore Abstract Record:/home/alex/Zotero/storage/ZNQD3FRW/9467291.html:text/html},
|
||||
}
|
||||
|
||||
@inproceedings{chefer_transformer_2021,
|
||||
title = {Transformer Interpretability Beyond Attention Visualization},
|
||||
url = {https://openaccess.thecvf.com/content/CVPR2021/html/Chefer_Transformer_Interpretability_Beyond_Attention_Visualization_CVPR_2021_paper.html},
|
||||
eventtitle = {Proceedings of the {IEEE}/{CVF} Conference on Computer Vision and Pattern Recognition},
|
||||
pages = {782--791},
|
||||
author = {Chefer, Hila and Gur, Shir and Wolf, Lior},
|
||||
urldate = {2025-02-21},
|
||||
date = {2021},
|
||||
langid = {english},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/AB22TAFK/Chefer et al. - 2021 - Transformer Interpretability Beyond Attention Visualization.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@inproceedings{rigotti_attention-based_2021,
|
||||
title = {Attention-based Interpretability with Concept Transformers},
|
||||
url = {https://openreview.net/forum?id=kAa9eDS0RdO},
|
||||
abstract = {Attention is a mechanism that has been instrumental in driving remarkable performance gains of deep neural network models in a host of visual, {NLP} and multimodal tasks. One additional notable aspect of attention is that it conveniently exposes the ``reasoning'' behind each particular output generated by the model. Specifically, attention scores over input regions or intermediate features have been interpreted as a measure of the contribution of the attended element to the model inference. While the debate in regard to the interpretability of attention is still not settled, researchers have pointed out the existence of architectures and scenarios that afford a meaningful interpretation of the attention mechanism. Here we propose the generalization of attention from low-level input features to high-level concepts as a mechanism to ensure the interpretability of attention scores within a given application domain. In particular, we design the {ConceptTransformer}, a deep learning module that exposes explanations of the output of a model in which it is embedded in terms of attention over user-defined high-level concepts. Such explanations are {\textbackslash}emph\{plausible\} (i.e.{\textbackslash} convincing to the human user) and {\textbackslash}emph\{faithful\} (i.e.{\textbackslash} truly reflective of the reasoning process of the model). Plausibility of such explanations is obtained by construction by training the attention heads to conform with known relations between inputs, concepts and outputs dictated by domain knowledge. Faithfulness is achieved by design by enforcing a linear relation between the transformer value vectors that represent the concepts and their contribution to the classification log-probabilities. We validate our {ConceptTransformer} module on established explainability benchmarks and show how it can be used to infuse domain knowledge into classifiers to improve accuracy, and conversely to extract concept-based explanations of classification outputs. Code to reproduce our results is available at: {\textbackslash}url\{https://github.com/ibm/concept\_transformer\}.},
|
||||
eventtitle = {International Conference on Learning Representations},
|
||||
author = {Rigotti, Mattia and Miksovic, Christoph and Giurgiu, Ioana and Gschwind, Thomas and Scotton, Paolo},
|
||||
urldate = {2025-02-21},
|
||||
date = {2021-10-06},
|
||||
langid = {english},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/U2FUGVF6/Rigotti et al. - 2021 - Attention-based Interpretability with Concept Transformers.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@article{royston_identifying_1991-1,
|
||||
title = {Identifying the fertile phase of the human menstrual cycle},
|
||||
volume = {10},
|
||||
issn = {1097-0258},
|
||||
url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.4780100207},
|
||||
doi = {10.1002/sim.4780100207},
|
||||
abstract = {The identification of the human fertile phase as the time during which a woman or a couple may conceive is elusive. The fertile time depends on many factors in each individual menstrual cycle and may be said to be more of a statistical than a physiological entity. This paper reviews the application of statistical methods to three areas related to conception and the fertile phase. The first is the prediction and detection of ovulation from serial measurements, such as hormones, basal body temperature and cervical mucus, throughout the menstrual cycle. Typically, such variables increase from some baseline level to a peak around ovulation (the most fertile time), then subside to low levels in the postovulatory phase. The statistical challenge is to detect the rise (signalling the onset of potential fertility) and subsequent fall. Analytic methods considered include thresholds, Bayesian change-point models and particularly the cumulative sum (cusum) technique which is both simple to apply and understand, and effective. The second area comprises appropriate methods of analysing and interpreting data from clinical studies of the fertile phase, especially in so-alled natural family planning ({NFP}) where it is usual for women to observe several indices of potential fertility. Such studies usually try to establish the temporal relationships between markers of the fertile phase and examine the success of different combinations of markers in delineating the fertile time in comparison with a standard ‘defined’ phase, for example, the interval from three days before to two days after the peak of luteinizing hormone. The third area is the assessment of the probability of conception on certain days of the cycle, which is vital to the understanding of the fertile phase and its application to {NFP}. Direct estimation of such probabilities is impractical; instead, resort must be made to estimation by maximum likelihood of the parameters of specially constructed models. Suitable models are described. Finally, the need for a new prospective study of the probability of conception in relation to the markers of the fertile phase used in the symptothermal method of {NFP} is discussed.},
|
||||
pages = {221--240},
|
||||
number = {2},
|
||||
journaltitle = {Statistics in Medicine},
|
||||
author = {Royston, Patrick},
|
||||
urldate = {2025-02-24},
|
||||
date = {1991},
|
||||
langid = {english},
|
||||
note = {\_eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sim.4780100207},
|
||||
file = {PDF:/home/alex/Zotero/storage/CFS45CFD/Royston - 1991 - Identifying the fertile phase of the human menstrual cycle.pdf:application/pdf;Snapshot:/home/alex/Zotero/storage/LCN63EBN/sim.html:text/html},
|
||||
}
|
||||
|
||||
@online{noauthor_vivosens_nodate,
|
||||
title = {vivosens medical gmbh},
|
||||
url = {https://www.vivosensmedical.com/},
|
||||
urldate = {2025-02-24},
|
||||
file = {vivosensmedical.com:/home/alex/Zotero/storage/KSM7HHBJ/www.vivosensmedical.com.html:text/html},
|
||||
}
|
||||
|
||||
@inproceedings{vaswani_attention_2017,
|
||||
title = {Attention is All you Need},
|
||||
volume = {30},
|
||||
url = {https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html},
|
||||
abstract = {The dominant sequence transduction models are based on complex recurrent orconvolutional neural networks in an encoder and decoder configuration. The best performing such models also connect the encoder and decoder through an attentionm echanisms. We propose a novel, simple network architecture based solely onan attention mechanism, dispensing with recurrence and convolutions entirely.Experiments on two machine translation tasks show these models to be superiorin quality while being more parallelizable and requiring significantly less timeto train. Our single model with 165 million parameters, achieves 27.5 {BLEU} {onEnglish}-to-German translation, improving over the existing best ensemble result by over 1 {BLEU}. On English-to-French translation, we outperform the previoussingle state-of-the-art with model by 0.7 {BLEU}, achieving a {BLEU} score of 41.1.},
|
||||
booktitle = {Advances in Neural Information Processing Systems},
|
||||
publisher = {Curran Associates, Inc.},
|
||||
author = {Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Ł ukasz and Polosukhin, Illia},
|
||||
urldate = {2025-02-24},
|
||||
date = {2017},
|
||||
file = {Full Text PDF:/home/alex/Zotero/storage/MU7NU9LR/Vaswani et al. - 2017 - Attention is All you Need.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@article{noauthor_temporal_2021,
|
||||
title = {Temporal Fusion Transformers for interpretable multi-horizon time series forecasting},
|
||||
volume = {37},
|
||||
issn = {0169-2070},
|
||||
url = {https://www.sciencedirect.com/science/article/pii/S0169207021000637},
|
||||
doi = {10.1016/j.ijforecast.2021.03.012},
|
||||
abstract = {Multi-horizon forecasting often contains a complex mix of inputs – including static (i.e. time-invariant) covariates, known future inputs, and other e…},
|
||||
pages = {1748--1764},
|
||||
number = {4},
|
||||
journaltitle = {International Journal of Forecasting},
|
||||
urldate = {2025-02-24},
|
||||
date = {2021-10-01},
|
||||
langid = {american},
|
||||
note = {Publisher: Elsevier},
|
||||
file = {Snapshot:/home/alex/Zotero/storage/SFYESIWK/S0169207021000637.html:text/html;Submitted Version:/home/alex/Zotero/storage/A9AYS5UI/2021 - Temporal Fusion Transformers for interpretable multi-horizon time series forecasting.pdf:application/pdf},
|
||||
}
|
||||
|
||||
@article{dunson_day-specific_1999,
|
||||
title = {Day-specific probabilities of clinical pregnancy based on two studies with imperfect measures of ovulation},
|
||||
volume = {14},
|
||||
issn = {0268-1161},
|
||||
url = {https://doi.org/10.1093/humrep/14.7.1835},
|
||||
doi = {10.1093/humrep/14.7.1835},
|
||||
abstract = {Two studies have related the timing of sexual intercourse (relative to ovulation) to day-specific fecundability. The first was a study of Catholic couples practising natural family planning in London in the 1950s and 1960s and the second was of North Carolina couples attempting to become pregnant in the early 1980s. The former identified ovulation based on the ovulatory shift in the basal body temperature, while the latter used urinary assays of hormones. We use a statistical model to correct for error in identifying ovulation and to re-estimate the length of the fertile window and day-specific fecundabilities. We estimate the same 6-day fertile interval in both studies after controlling for error. After adjusting for error both data sets showed the highest estimate of the probability of pregnancy on the day prior to ovulation and both fell close to zero after ovulation. Given that the fertile interval is before ovulation, methods that anticipate ovulation by several days (such as the assessment of cervical mucus) would be particularly useful for couples who want to time their intercourse either to avoid or facilitate conception.},
|
||||
pages = {1835--1839},
|
||||
number = {7},
|
||||
journaltitle = {Human Reproduction},
|
||||
author = {Dunson, D.B. and Baird, D.D. and Wilcox, A.J. and Weinberg, C.R.},
|
||||
urldate = {2025-02-24},
|
||||
date = {1999-07-01},
|
||||
file = {Snapshot:/home/alex/Zotero/storage/WZEWDS8D/2913179.html:text/html},
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user