diff --git a/thesis/main.tex b/thesis/main.tex index 68982e6..1c05dbf 100644 --- a/thesis/main.tex +++ b/thesis/main.tex @@ -14,6 +14,7 @@ \usepackage{pdflscape} \usepackage{adjustbox} \usepackage{multirow} +\usepackage{subcaption} \usepackage{blindtext} \usepackage[a4paper, margin=1in]{geometry} \usepackage[toc,page]{appendix} diff --git a/thesis/sections/discussion.tex b/thesis/sections/discussion.tex index 43a206b..276785b 100644 --- a/thesis/sections/discussion.tex +++ b/thesis/sections/discussion.tex @@ -42,7 +42,7 @@ i.e., the LSTM model. Figure~\ref{fig:discussion_regular_cycle_fertility_prediction} shows the prediction curve for the fertility-probability target for a user with a regular cycle pattern. -It is clearly visible, that the predictions improve with each cycle, until they almost exactly match the targets. +For such a regular cycle pattern, the predictions almost exactly match the targets. Another noteworthy observation is the correlation between a clear temperature drop preceding ovulation and the fertility rising. This can be seen more prominently in Figure~\ref{fig:discussion_temperature_drop_fertility_prediction}. @@ -63,6 +63,16 @@ but its intensity varies between users and also between cycles of the same user. Such a pattern can turn out to be a useful predictor for ovulation / fertility, but the models we trained don't seem to be able to differentiate between ovulation-related and unrelated temperature drops. +This also points to a potential flaw in our workflow. +We don't yet have clinically accurate labels for the cycles that were used for the training of our models. +In fact, we cannot guarantee an ovulation, not even for the cycles with a clear temperature rise after the apparent ovulation. +It is not uncommon for women to have a clear temperature rise without an ovulation and vice versa, +to have an ovulation but no clear temperature rise. +The identified temperature drop and the fertility that seems to come with it could be a base for further research +with more accurate ovulation labeling. +It might be an indicator for a successful upcoming ovulation. + + \begin{figure}[htbp] \centering \includegraphics[width=1.0\textwidth]{resources/figures/discussion/temperature_drop_fertility} @@ -86,7 +96,7 @@ to be able to differentiate between ovulation-related and unrelated temperature \includegraphics[width=1.0\textwidth]{resources/figures/discussion/temperature_unclear_temperature_drop} \caption{ Temperature rolling average and fertility-probability prediction with no clear temperature drop and a resulting - incorrect fertility-probability prediction. (Values are scaled features) + incorrect prediction. (Values are scaled features) } \label{fig:discussion_unclear_temperature_drop} \end{figure} @@ -102,6 +112,25 @@ This is likely information indicating some form of regularity, which the models It would be interesting to take a close look at how and in what intensity the models use certain features, and whether the performance changes upon omitting certain features. +\paragraph{Use-Case Study.} + +The threshold has a large effect on the overall effectiveness of the different use cases. +For the contraception use case, changing the threshold doubles and even quadruples the pregnancy rate. +It has to be noted, that the contraception use case is naive use-case, where the woman does not take any other +measures next to the prediction of our models. +In a real-world scenario, this is largely not the case, and thus the pregnancy rates should be even lower. +A measure often taken is abstinence during the first cycle phase (luteal phase), which should significantly lower +unwanted pregnancies. +Even without additional measures, a threshold of 0.05 leads to an approximate \emph{Pearl-Index} (pregnancy rate over 1 year for 100 women) +of 4--5, which is significantly better than methods such as the contraceptive pill (7) or the condom (13) for a typical use case. +However, these numbers have to be taken with caution, as this is not an actual study, but a naive theoretical projection. +Further research is necessary to find more reliable results. + +The pregnancy use case is harder to contextualize, as there are no comparable results for other methods, +and the actual pregnancy probability is subject to many more factors, we could not take into consideration for this study. + + + % %In general, we expect the transformer based model to outperform the LSTM basd models, as they have proven to be diff --git a/thesis/sections/results.tex b/thesis/sections/results.tex index 3fae997..6bd0046 100644 --- a/thesis/sections/results.tex +++ b/thesis/sections/results.tex @@ -582,24 +582,6 @@ Each evaluation used a test set of 100 users (100 user-years) and was repeated f we report means and 95\% confidence intervals (CIs). \subsubsection{Contraception Use-Case Results}\label{subsubsec:use_case_contraception_results} -\begin{table} - \scriptsize - \begin{tabularx}{\linewidth}{l*{5}{X}} - \toprule - Model & No. of Inter. Events & Pregnancies & Correct Denials & Incorrect Denials \\ - \midrule - Convolutional LSTM & 1154 (1149-1158) & 34.9 (34.1-35.7) & 1615 (1611-1620) & 4963 (4954-4971) \\ - Transformer & 3200 (3194-3207) & 4.4 (4.1-4.7) & 1817 (1812-1822) & 2717 (2710-2724) \\ - Convolutional Transformer & 1156 (1151-1160) & 34.3 (33.5-35.0) & 1613 (1608-1619) & 4974 (4965-4983) \\ - LSTM & 3206 (3199-3213) & 4.6 (4.3-4.9) & 1832 (1827-1838) & 2711 (2705-2717) \\ - Last-Cycle Baseline & 6066 (6057-6075) & 127.5 (125.9-129.1) & 918 (915-922) & 762 (759-766) \\ - Population-Mean Baseline & 5874 (5865-5883) & 153.7 (151.9-155.5) & 749 (746-752) & 1121 (1116-1125) \\ - User-Mean Baseline & 5933 (5924-5942) & 105.8 (104.4-107.2) & 1030 (1026-1034) & 776 (772-779) \\ - \bottomrule - \end{tabularx} - \caption{Contraception metrics at threshold \textbf{0.01} for all models. Values are means over 200 iterations; 95\% CIs in parentheses.} - \label{tab:results_contraception_use_case_0_01} -\end{table} \paragraph{Threshold 0.01 (Table~\ref{tab:results_contraception_use_case_0_01}).} At the strictest threshold of 0.01, Transformer and LSTM models achieve the lowest pregnancy rates, only 4.4 to 4.6 @@ -611,22 +593,74 @@ leading to roughly 34–35 pregnancies, or about 30 per 1,000 events—far less All baseline models perform substantially worse, with pregnancy counts exceeding 100 in all cases, confirming the value of personalized predictions. \begin{table} - \scriptsize - \begin{tabularx}{\linewidth}{l*{5}{X}} - \toprule - Model & No. of Inter. Events & Pregnancies & Correct Denials & Incorrect Denials \\ - \midrule - Convolutional LSTM & 2509 (2502-2515) & 55.0 (54.0-56.0) & 1448 (1444-1452) & 3768 (3760-3775) \\ - Transformer & 3801 (3793-3810) & 8.2 (7.8-8.6) & 1768 (1763-1773) & 2162 (2157-2168) \\ - Convolutional Transformer & 2510 (2503-2516) & 55.6 (54.5-56.6) & 1448 (1443-1453) & 3776 (3769-3783) \\ - LSTM & 3801 (3794-3809) & 8.2 (7.8-8.7) & 1772 (1766-1777) & 2160 (2154-2165) \\ - Last-Cycle Baseline & 6106 (6096-6116) & 130.2 (128.7-131.7) & 888 (885-892) & 750 (746-753) \\ - Population-Mean Baseline & 5924 (5914-5933) & 156.1 (154.3-157.8) & 716 (713-720) & 1100 (1096-1104) \\ - User-Mean Baseline & 5988 (5978-5997) & 108.8 (107.3-110.3) & 998 (994-1002) & 764 (761-768) \\ - \bottomrule - \end{tabularx} - \caption{Contraception metrics at threshold \textbf{0.05} for all models. Values are means over 200 iterations; 95\% CIs in parentheses.} - \label{tab:results_contraception_use_case_0_05} + \centering + + \begin{subtable}{\textwidth} + \centering + \scriptsize + \begin{tabularx}{\linewidth}{l*{5}{X}} + \toprule + Model & No. of Intercourse Events & Pregnancies & Correct Denials & Incorrect Denials \\ + \midrule + Convolutional LSTM & 1154 (1149-1158) & 34.9 (34.1-35.7) & 1615 (1611-1620) & 4963 (4954-4971) \\ + Transformer & 3200 (3194-3207) & 4.4 (4.1-4.7) & 1817 (1812-1822) & 2717 (2710-2724) \\ + Convolutional Transformer & 1156 (1151-1160) & 34.3 (33.5-35.0) & 1613 (1608-1619) & 4974 (4965-4983) \\ + LSTM & 3206 (3199-3213) & 4.6 (4.3-4.9) & 1832 (1827-1838) & 2711 (2705-2717) \\ + Last-Cycle Baseline & 6066 (6057-6075) & 127.5 (125.9-129.1) & 918 (915-922) & 762 (759-766) \\ + Population-Mean Baseline & 5874 (5865-5883) & 153.7 (151.9-155.5) & 749 (746-752) & 1121 (1116-1125) \\ + User-Mean Baseline & 5933 (5924-5942) & 105.8 (104.4-107.2) & 1030 (1026-1034) & 776 (772-779) \\ + \bottomrule + \end{tabularx} + \caption{Contraception metrics at threshold \textbf{0.01} for all models.} + \label{tab:results_contraception_use_case_0_01} + \end{subtable} + + \vspace{1.5em} + + \begin{subtable}{\textwidth} + \centering + \scriptsize + \begin{tabularx}{\linewidth}{l*{5}{X}} + \toprule + Model & No. of Intercourse Events & Pregnancies & Correct Denials & Incorrect Denials \\ + \midrule + Convolutional LSTM & 2509 (2502-2515) & 55.0 (54.0-56.0) & 1448 (1444-1452) & 3768 (3760-3775) \\ + Transformer & 3801 (3793-3810) & 8.2 (7.8-8.6) & 1768 (1763-1773) & 2162 (2157-2168) \\ + Convolutional Transformer & 2510 (2503-2516) & 55.6 (54.5-56.6) & 1448 (1443-1453) & 3776 (3769-3783) \\ + LSTM & 3801 (3794-3809) & 8.2 (7.8-8.7) & 1772 (1766-1777) & 2160 (2154-2165) \\ + Last-Cycle Baseline & 6106 (6096-6116) & 130.2 (128.7-131.7) & 888 (885-892) & 750 (746-753) \\ + Population-Mean Baseline & 5924 (5914-5933) & 156.1 (154.3-157.8) & 716 (713-720) & 1100 (1096-1104) \\ + User-Mean Baseline & 5988 (5978-5997) & 108.8 (107.3-110.3) & 998 (994-1002) & 764 (761-768) \\ + \bottomrule + \end{tabularx} + \caption{Contraception metrics at threshold \textbf{0.05} for all models.} + \label{tab:results_contraception_use_case_0_05} + \end{subtable} + + \vspace{1.5em} + + \begin{subtable}{\textwidth} + \centering + \scriptsize + \begin{tabularx}{\linewidth}{l*{5}{X}} + \toprule + Model & No. of Intercourse Events & Pregnancies & Correct Denials & Incorrect Denials \\ + \midrule + Convolutional LSTM & 3596 (3588-3603) & 75.2 (74.0-76.4) & 1292 (1288-1297) & 2847 (2840-2853) \\ + Transformer & 4226 (4217-4234) & 13.3 (12.8-13.7) & 1692 (1687-1696) & 1822 (1817-1827) \\ + Convolutional Transformer & 3592 (3584-3599) & 73.8 (72.7-75.0) & 1296 (1292-1301) & 2846 (2840-2852) \\ + LSTM & 4222 (4214-4230) & 13.3 (12.8-13.8) & 1694 (1688-1699) & 1818 (1812-1823) \\ + Last-Cycle Baseline & 6145 (6135-6155) & 134.8 (133.3-136.3) & 865 (861-868) & 744 (740-747) \\ + Population-Mean Baseline & 5960 (5951-5969) & 160.3 (158.5-162.1) & 694 (691-697) & 1088 (1084-1093) \\ + User-Mean Baseline & 6014 (6005-6022) & 111.8 (110.3-113.3) & 969 (965-973) & 758 (755-762) \\ + \bottomrule + \end{tabularx} + \caption{Contraception metrics at threshold \textbf{0.10} for all models.} + \label{tab:results_contraception_use_case_0_10} + \end{subtable} + + \caption{grouped contraception metrics at thresholds 0.01, 0.05, and 0.10. values are means over 200 iterations; 95\% confidence intervals in parentheses.} + \label{tab:results_contraception_grouped} \end{table} \paragraph{Threshold 0.05 (Table~\ref{tab:results_contraception_use_case_0_05}).} @@ -638,25 +672,6 @@ Convolutional models also allow more events (~2,510) but continue to produce sig yielding a less favorable risk-benefit profile. Baselines remain underperforming. -\begin{table} - \scriptsize - \begin{tabularx}{\linewidth}{l*{5}{X}} - \toprule - Model & No. of Inter. Events & Pregnancies & Correct Denials & Incorrect Denials \\ - \midrule - Convolutional LSTM & 3596 (3588-3603) & 75.2 (74.0-76.4) & 1292 (1288-1297) & 2847 (2840-2853) \\ - Transformer & 4226 (4217-4234) & 13.3 (12.8-13.7) & 1692 (1687-1696) & 1822 (1817-1827) \\ - Convolutional Transformer & 3592 (3584-3599) & 73.8 (72.7-75.0) & 1296 (1292-1301) & 2846 (2840-2852) \\ - LSTM & 4222 (4214-4230) & 13.3 (12.8-13.8) & 1694 (1688-1699) & 1818 (1812-1823) \\ - Last-Cycle Baseline & 6145 (6135-6155) & 134.8 (133.3-136.3) & 865 (861-868) & 744 (740-747) \\ - Population-Mean Baseline & 5960 (5951-5969) & 160.3 (158.5-162.1) & 694 (691-697) & 1088 (1084-1093) \\ - User-Mean Baseline & 6014 (6005-6022) & 111.8 (110.3-113.3) & 969 (965-973) & 758 (755-762) \\ - \bottomrule - \end{tabularx} - \caption{Contraception metrics at threshold \textbf{0.10} for all models. Values are means over 200 iterations; 95\% CIs in parentheses.} - \label{tab:results_contraception_use_case_0_10} -\end{table} - \paragraph{Threshold 0.10 (Table~\ref{tab:results_contraception_use_case_0_10}).} A further increase to 0.10 raises Transformer/LSTM events to ~4,220, but also raises pregnancies to ~13.3 (3.1 per 1,000). This is a ~11\% gain in access compared to 0.05, but the pregnancy count increases by ~62\%. @@ -669,24 +684,77 @@ Transformer and LSTM models perform best, allowing a relatively high number of i This threshold offers the best compromise and is selected as the most promising setting for contraceptive use. \subsubsection{Pregnancy Use-Case Results}\label{subsubsec:use_case_pregnancy_results} + \begin{table} - \scriptsize - \begin{tabularx}{\linewidth}{l*{5}{X}} - \toprule - Model & No. of Inter. Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\ - \midrule - Convolutional LSTM & 4899 (4890-4908) & 216.9 (214.8-219.1) & 10038 & 1734 \\ - Transformer & 3685 (3678-3692) & 264.9 (262.7-267.0) & 17489 & 350 \\ - Convolutional Transformer & 4903 (4894-4912) & 214.3 (212.2-216.4) & 10038 & 1734 \\ - LSTM & 3688 (3680-3696) & 266.9 (264.7-269.1) & 17489 & 350 \\ - Last-Cycle Baseline & 1536 (1531-1541) & 138.7 (137.1-140.2) & 24593 & 4038 \\ - Population-Mean Baseline & 1704 (1699-1709) & 113.5 (112.1-114.9) & 22956 & 4841 \\ - User-Mean Baseline & 1652 (1647-1656) & 158.4 (156.8-160.0) & 24528 & 3523 \\ - \bottomrule - \end{tabularx} - \caption{Pregnancy metrics at threshold \textbf{0.01} for all models. Values are means over 200 iterations; 95\% CIs in parentheses. + \centering + + \begin{subtable}{\textwidth} + \centering + \scriptsize + \begin{tabularx}{\linewidth}{l*{5}{X}} + \toprule + Model & No. of Intercourse Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\ + \midrule + Convolutional LSTM & 4899 (4890-4908) & 216.9 (214.8-219.1) & 10038 & 1734 \\ + Transformer & 3685 (3678-3692) & 264.9 (262.7-267.0) & 17489 & 350 \\ + Convolutional Transformer & 4903 (4894-4912) & 214.3 (212.2-216.4) & 10038 & 1734 \\ + LSTM & 3688 (3680-3696) & 266.9 (264.7-269.1) & 17489 & 350 \\ + Last-Cycle Baseline & 1536 (1531-1541) & 138.7 (137.1-140.2) & 24593 & 4038 \\ + Population-Mean Baseline & 1704 (1699-1709) & 113.5 (112.1-114.9) & 22956 & 4841 \\ + User-Mean Baseline & 1652 (1647-1656) & 158.4 (156.8-160.0) & 24528 & 3523 \\ + \bottomrule + \end{tabularx} + \caption{Pregnancy metrics at threshold \textbf{0.01} for all models. } + \label{tab:results_pregnancy_use_case_0_01} + \end{subtable} + + \vspace{1.5em} + + \begin{subtable}{\textwidth} + \centering + \scriptsize + \begin{tabularx}{\linewidth}{l*{5}{X}} + \toprule + Model & No. of Intercourse Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\ + \midrule + Convolutional LSTM & 6174 (6164-6184) & 236.1 (234.1-238.2) & 4332 & 1076 \\ + Transformer & 4253 (4245-4261) & 270.9 (268.6-273.1) & 14825 & 190 \\ + Convolutional Transformer & 6173 (6164-6182) & 235.7 (233.6-237.9) & 4332 & 1076 \\ + LSTM & 4255 (4247-4262) & 273.0 (270.6-275.3) & 14825 & 190 \\ + Last-Cycle Baseline & 1581 (1576-1586) & 139.8 (138.2-141.3) & 24327 & 4091 \\ + Population-Mean Baseline & 1752 (1747-1757) & 117.1 (115.6-118.6) & 22655 & 4888 \\ + User-Mean Baseline & 1695 (1690-1701) & 158.3 (156.6-160.1) & 24271 & 3567 \\ + \bottomrule + \end{tabularx} + \caption{Pregnancy metrics at threshold \textbf{0.05} for all models.} + \label{tab:results_pregnancy_use_case_0_05} + \end{subtable} + + \vspace{1.5em} + + \begin{subtable}{\textwidth} + \centering + \scriptsize + \begin{tabularx}{\linewidth}{l*{5}{X}} + \toprule + Model & No. of Intercourse Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\ + \midrule + Convolutional LSTM & 3887 (3880-3895) & 198.0 (196.1-199.9) & 14544 & 2296 \\ + Transformer & 3291 (3284-3298) & 260.7 (258.6-262.7) & 19261 & 551 \\ + Convolutional Transformer & 3883 (3875-3891) & 198.4 (196.6-200.3) & 14544 & 2296 \\ + LSTM & 3296 (3289-3303) & 261.1 (259.1-263.2) & 19261 & 551 \\ + Last-Cycle Baseline & 1509 (1504-1513) & 135.2 (133.6-136.8) & 24778 & 4010 \\ + Population-Mean Baseline & 1668 (1663-1673) & 109.5 (108.1-110.9) & 23156 & 4811 \\ + User-Mean Baseline & 1619 (1614-1624) & 154.0 (152.4-155.6) & 24717 & 3501 \\ + \bottomrule + \end{tabularx} + \caption{Pregnancy metrics at threshold \textbf{0.10} for all models.} + \label{tab:results_pregnancy_use_case_0_10} + \end{subtable} + + \caption{grouped pregnancy metrics at thresholds 0.01, 0.05, and 0.10. values are means over 200 iterations; 95\% confidence intervals in parentheses. Under our simulation at a fixed threshold, correct/incorrect deferrals are deterministic; CIs are therefore omitted for these columns.} - \label{tab:results_pregnancy_use_case_0_01} + \label{tab:results_pregnancy_grouped} \end{table} \paragraph{Threshold 0.01 (Table~\ref{tab:results_pregnancy_use_case_0_01}).} @@ -696,25 +764,6 @@ Convolutional models yield fewer pregnancies (~214–217) but allow ~4,900 inter Baselines underperform on both metrics, allowing fewer events and achieving lower pregnancy counts, suggesting they are overly conservative without yielding benefits in effectiveness. -\begin{table} - \scriptsize - \begin{tabularx}{\linewidth}{l*{5}{X}} - \toprule - Model & No. of Inter. Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\ - \midrule - Convolutional LSTM & 6174 (6164-6184) & 236.1 (234.1-238.2) & 4332 & 1076 \\ - Transformer & 4253 (4245-4261) & 270.9 (268.6-273.1) & 14825 & 190 \\ - Convolutional Transformer & 6173 (6164-6182) & 235.7 (233.6-237.9) & 4332 & 1076 \\ - LSTM & 4255 (4247-4262) & 273.0 (270.6-275.3) & 14825 & 190 \\ - Last-Cycle Baseline & 1581 (1576-1586) & 139.8 (138.2-141.3) & 24327 & 4091 \\ - Population-Mean Baseline & 1752 (1747-1757) & 117.1 (115.6-118.6) & 22655 & 4888 \\ - User-Mean Baseline & 1695 (1690-1701) & 158.3 (156.6-160.1) & 24271 & 3567 \\ - \bottomrule - \end{tabularx} - \caption{Pregnancy metrics at threshold \textbf{0.05} for all models. Values are means over 200 iterations; 95\% CIs in parentheses.} - \label{tab:results_pregnancy_use_case_0_05} -\end{table} - \paragraph{Threshold 0.05 (Table~\ref{tab:results_pregnancy_use_case_0_05}).} At threshold 0.05, Transformer/LSTM models slightly increase intercourse access (~4,250 events) with pregnancies rising to ~271–273 (64 per 1,000). Notably, these models also achieve very low incorrect deferral counts (~190), indicating they rarely block opportunities for conception when they shouldn’t. @@ -723,26 +772,6 @@ Convolutional models allow substantially more intercourse (~6,170) with lower pr but at the cost of higher incorrect deferrals (~1,076). This suggests they are more permissive but less selective. - -\begin{table} - \scriptsize - \begin{tabularx}{\linewidth}{l*{5}{X}} - \toprule - Model & No. of Inter. Events & Pregnancies & Correct Deferrals & Incorrect Deferrals \\ - \midrule - Convolutional LSTM & 3887 (3880-3895) & 198.0 (196.1-199.9) & 14544 & 2296 \\ - Transformer & 3291 (3284-3298) & 260.7 (258.6-262.7) & 19261 & 551 \\ - Convolutional Transformer & 3883 (3875-3891) & 198.4 (196.6-200.3) & 14544 & 2296 \\ - LSTM & 3296 (3289-3303) & 261.1 (259.1-263.2) & 19261 & 551 \\ - Last-Cycle Baseline & 1509 (1504-1513) & 135.2 (133.6-136.8) & 24778 & 4010 \\ - Population-Mean Baseline & 1668 (1663-1673) & 109.5 (108.1-110.9) & 23156 & 4811 \\ - User-Mean Baseline & 1619 (1614-1624) & 154.0 (152.4-155.6) & 24717 & 3501 \\ - \bottomrule - \end{tabularx} - \caption{Pregnancy metrics at threshold \textbf{0.10} for all models. Values are means over 200 iterations; 95\% CIs in parentheses.} - \label{tab:results_pregnancy_use_case_0_10} -\end{table} - \paragraph{Threshold 0.10 (Table~\ref{tab:results_pregnancy_use_case_0_10}).} At the highest threshold, Transformer/LSTM models see a drop in access (~3,290 events) and in pregnancies (~261), but with an increase in incorrect deferrals (~551).