Reaching Pandemic Milestones with Country Primary and Secondary Vaccination Inflection Points: An Assessment of Foundational and Hybrid Forecasting Methodologies”

I. INTRODUCTION

COVID-19 is an infectious disease, caused by the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) characterized by high morbidity and mortality, and a significant burden on hospital systems and country economies. Over the last three years, the COVID-19 virus infected over 300 million people, caused death for approximately seven million people [1], and had a negative impact of $3.8 trillion on economies around the world. At the end of 2023, COVID-19 is still present with different virus mutations continuing to cause infections and deaths across the world [2, 3, 4].

The experience with the COVID-19 pandemic demonstrated the inadequate levels of preparedness across countries. Two thirds of world countries have a good capacity for public health threat surveillance and analytics in order to drive policy and planning. However, half of the countries have a limited capacity to systematically monitor care, including the impact of vaccination $[5]$. Both surveillance and monitoring are needed to adequately plan and prepare for possible infectious disease outbreaks with novel viruses [6, 7, 8, 9, 10, 11, 12, 13].

Vaccination is an effective way to obtain individual and herd immunity $[14]$. When the herd immunity threshold is reached, naturally or through vaccination, it creates an environment that is sufficient to control large outbreaks, reduce the number of infected individuals and possible deaths, protect vulnerable individuals in the society, and relax other public health measures $[15]$. There are many factors that influence a success of a vaccination campaign, such as availability of vaccine supply (e.g. speed of development, level of demand, difficulties with production and distribution of vaccines worldwide), vaccination strategy outlining priority groups for vaccination, and population acceptance of vaccination (e.g. anti-vaccination movement).

The vaccination efforts for COVID-19 started in December of 2020 for most of the countries in the world. There were several types of vaccines that were available: genetically engineered messenger RNA, viral vector, and protein subunit vaccines. The initial vaccinations from 2020 were followed with booster doses in 2021, 2022 and 2023, for a total of four booster doses, specifically in developed countries [16]. Understanding the impact of vaccination campaigns, the correlation of vaccination rates, incidence of COVID-19, and mortality rates was researched over the last two years. The results confirmed that successful vaccination efforts (e.g. availability of vaccines, public acceptance, strong government programs, etc.) can significantly reduce the negative effects of the COVID-19 pandemic, with a sharp decrease in the fatality rate [17, 18, 19]. Some researchers were able to define the vaccination threshold, identifying that a mean level of administering about 80 doses of vaccines per 100 inhabitants can sustain a reduction of confirmed cases and number of deaths [11], or when the mean cumulative vaccination rate reaches 29.06 doses per 100 people and 7.88 doses per 100 people, respectively, for spread and mortality [19]. Many researchers also looked at the sentiment around vaccination. Attitudes toward COVID-19 and vaccination, conspiracy beliefs, misconceptions, and complaints about COVID-19 control, were documented as dominant sentiments $[21,22,23]$. Researchers used data from different sources (local, national, and global registries) and different time frames (e.g., periods of 3 or 6 months post initial vaccination).

Diverse research methodologies were applied to increase sensitivity of analyses and achieve more accurate results, such as neural networks with cut effect $[17]$, Augmented Artificial Neural Network Model for the COVID-19 Mortality Prediction relative to the vaccination rates $[24]$; Deep Learning Sequence Models for Forecasting COVID-19 Spread and Vaccinations with two recurrent neural network-based approaches, LSTM and GRU $[25]$; amalgamation of neural network with two powerful optimization algorithms, firefly algorithm and artificial bee colony based feed-forward neural networks to look at the effect of vaccinated population on the COVID-19 prediction $[26]$; and a multi-path long short term memory (LSTM) neural network for COVID-19 forecasting of new viral variants and vaccination $[27]$. Other researchers explored other models, structured and unstructured machine learning (ML) models $[22]$, structural topic modeling $[23]$, Latent Dirichlet Allocation (LDA) $[28]$, deep learning and NLP $[29,30]$. Cheng applied newly developed ARIMA models to improve the accuracy of weekly COVID-19 case growth rates and forecast COVID-19 spread according to protective behavior and vaccination $[31]$. Dhamodharavadhani and colleagues used hybrid models to forecast the vaccination rate, such as HARIMA, a hybrid of ARIMA and HGRNN, a hybrid of Generalized Regression Neural Network and the Gaussian Process Regression model $[32]$. Yi-Tui Chen and colleagues explored the effect of vaccination patterns and vaccination rates on the spread and mortality of the COVID-19 pandemic $[19]$, and Kumar utilized the recurrent neural network (RNN) Convolutional Residual Network (RNNCON-Res) $[33]$.

Nicholson and colleagues used both supervised and unsupervised methodologies to identify the critical county-level factors for studying COVID-

19 propagation prior to the widespread availability of a vaccine [40].

Published research has increased collective knowledge and has answered many questions. With limitations of every research, availability of more data and novel methodologies, there is a need and a responsibility to continue to expand the knowledge around pandemic vulnerability that can allow for better understanding of the dynamics of vaccination, infection rates and mortality.

This research was conducted to identify the vaccination inflection points and the time needed to reach the critical cumulative vaccination rate thresholds to observe continuous decrease of the case fatality rates. It was conducted both at an aggregate and at the country level. COVID-19 historical data was utilized to develop models that can be used for future pandemics. Applying advanced AI methodologies to forecast time to country specific vaccination inflection points, and assessing the vaccination rates relative to the case fatality rates, can provide another useful tool to guide countries in their pandemic risk preparedness.

II. MATERIALS AND METHODS

2.1 Data

This research utilized data from the Oxford University Our World in Data Covid 19 Dataset. This dataset contains data points collected on an ongoing basis from Johns Hopkins University, Center for Systems Science and Engineering COVID-19 data, European Centre for Disease Control, and OXFORD COVID-19 Government Response Tracker, from January 2020 to the present. The original dataset contains data from 207 countries and territories from which 26 countries were selected for this research: United States, Canada, Italy, Ireland, Finland, Iceland, Denmark, Belgium, Sweden, United Kingdom, Switzerland, Slovenia, Austria, Portugal, France, Netherlands, Luxembourg, Spain, Romania, Latvia, Cyprus, Estonia, Czechia, Slovakia, Serbia, and Bulgaria. Data for this research paper was accessed and downloaded on Dec 30, 2022 [35], and this longitudinal dataset was used from the period of December 2020, when most of the countries in the research dataset started vaccinating their population, to December 30, 2022.

The analyses in this research used 16 variables. Table 1 presents the 14 variables that represent the actual values from the research dataset. Two additional variables, case fatality rate and vaccination rate, were derived. The case fatality rate (CFR), an epidemiologic metric defined as the proportion of deaths within an observed population of interest [34], was calculated by dividing the respective values in the total deaths column by the total cases column of the dataset, for each of the 26 countries. The vaccination rate was calculated by dividing the number of people vaccinated (with at least one dose) by the total population of each country, for each of the 26 countries.

For a more meaningful interpretation of the data variables used to assess the correlation with the vaccination and CFR rates, data variables were organized into novel public health

indices, the Population Health Index, PHI [35], and Pandemic Sensitivity Index, PSI (Table 1). The PHI contains the parameters that describe the health of the population such as: cardiovascular death rate, diabetes prevalence, female smokers, male smokers, life expectancy, age 65 and older, and median age. The PSI Index represents variables that are directly impacted by the pandemic, such as total COVID-19 cases and deaths, number of COVID-19 hospital and ICU admissions, Government response stringency index (a composite measure based on nine response indicators including school and workplace closures, and travel bans), reproduction rate of transmission of COVID-19, and positivity rate of COVID-19.

In this research, it was assumed that all vaccines produced by different technologies and manufacturers have the same effectiveness. It was also assumed that distribution of different vaccines in different countries includes a combination of initial two-dose and single-dose vaccines and single dose booster vaccines over the two-year period (Dec 2020-Dec 2022). Since all vaccines require approximately two weeks to produce immunity, the effect of performance of vaccines on CFR was examined two weeks after the start of vaccination.

Several types of vaccines were available at the time of the initial vaccination: genetically engineered messenger RNA Pfizer/BioNTech and Moderna, viral vector vaccines (Janssen/Johnson

& Johnson and University of Oxford/AstraZeneca, Sputnik V), protein subunit vaccine (Novavax, Sinovac). The initial vaccinations in 2020 were delivered, in most cases, in sets of 2-doses, with a 3-week period in between (Pfizer/BioNTech, Moderna, Sinovac, Sputnik V). Some initial vaccines were delivered as a single dose vaccine (J&J, AZ/Oxford). Consequently, booster doses were delivered as single dose vaccines, starting in the third quarter of 2021 (Sep 2021 in the US, Oct/Nov 2021 in the EU) and continuing in 2022 (approved boosters in Mar and Sep 2022 in the US) and 2023 (approved in Sep 2023 in US and EU), for a total of four booster doses [16]. Today there are approximately 40 COVID-19 vaccines that were approved by regulatory agencies for full emergency use authorization. Of those 40, 16 have full authorization in only one country, 12 in ten or fewer countries, and 12 in more than 10 countries [36]. Emergence of new variants may be a challenge for the vaccines, reducing their protective power with the transmissibility of new variants substantially higher than the pre-existing SARS-CoV-2 variants. Booster dose vaccines were introduced to boost the protection power of vaccines and help the individuals with weakened immune systems. Efficacy of most vaccines range from 70-95%, mainly against symptomatic disease [37, 38]. All countries from this dataset (26 countries) are classified in three categories relative to their GDP per capita (>$50,000,$35,000-35,000) [20]. Table 3 summarizes the distribution of countries. This research was solely conducted by using publicly available data.

2.2 Methodologies

Data utilized in this research was pre-processed by assigning the original time series dataset to training and testing datasets temporally. For each country, the training set included data from the beginning of the pandemic (March 1, 2020) until a few weeks post vaccination start. The testing set included the remaining data post vaccination until the end of the dataset (December 30, 2022).

Data cleaning was conducted by resolving the problem of missing and duplicate values, resolving data inconsistencies, removing outliers, and smoothing variables used for forecasting (vaccination_rate and case_fatality_rate), including all exogenous variables (stringency_index, aged_65_older, life_expectancy, and positive_rate). Smoothing was conducted by using a window of seven days to remove all noisy data. The current day value was calculated using the mean of the previous seven days for each variable. In this type of dataset, it is common that some data is missing, both at random and not at random. For this research, it was important that the data on the total number of cases and deaths was complete since it was used to derive the case fatality rates. This missing data was resolved by taking the mean values of the total number of cases and deaths from the previous day and the next day. Other missing data was managed in a similar manner. Data quality assessments (completeness, reliability, consistency, validity, and no redundancy) were also completed. Exploratory Data Analysis was conducted by exploring graphs and visuals in order to observe trends over time of the vaccination and case fatality rates for each country.

Three foundational forecasting methodologies were applied: Autoregressive Integrated Moving Average (ARIMA), Prophet, and Long-Short Term Memory (LSTM) models. These models were then enhanced and combined to develop novel double and triple hybrids, SARIMA-Bidirectional LSTM and SARIMA-Prophet-Bidirectional LSTM models. They were used to forecast the primary and secondary vaccination inflection points (PVIP and SVIP) relative to the case fatality rates, for each of the 26 countries. All machine learning and deep learning analyses were done using Python version 3.10.1 and the scikit-learn library version 1.2.0 [39]. In addition, the novel Vaccination Inflection Point Score was developed, and countries were classified according to the score.

2.2.1 Correlation Analysis

The correlation analysis was performed using Ordinary Least Squares Multifactor Regression Methodology to identify the top four variables that correlate the most with vaccination and case fatality rates for implementation into forecasting methodologies. These analyses were performed as an aggregate analysis of 14 variables that were assessed for correlation with vaccination and case fatality rates. All variables were used for the correlation assessment with the vaccination rate. Two variables, total_cases and total_deaths were not used in the assessment of the case fatality correlation since the CFR is a ratio of these two variables. In order to derive the list of the top four variables most correlated with both vaccination and case fatality rates together, the ranking order was assessed across both target variables (vaccination and case fatality rates).

2.2.2 Foundational Forecasting Methodologies

Baseline forecasting methodologies were selected based on literature search, model strengths and limitations.

A. Autoregressive Integrated Moving Average (Arima)

ARIMA (Autoregressive Integrated Moving Average) model is selected for its characteristics of being well-suited for forecasting time series data that exhibits trends and seasonality. It is deemed to be effective in forecasting a variety of real-world phenomena, which has good applicability for COVID-19, showing greater flexibility, accuracy, interpretability, and robustness. The parameters of the ARIMA model are defined as follows: p is the lag order, which represents the number of lag observations incorporated in the model, d is the degree of differencing, which denotes the number of times raw observations undergo differencing, and q is the order of the moving average, which indicates the size of the moving average window [41].

B. Prophet

The Facebook Prophet algorithm is an open-source software developed by Facebook's core Data Science Team. If the time series data has strong seasonal effects, this model works the best. It is a regression model for forecasting, specifically designed to forecast time series data that exhibits trends, seasonality, and coverage for holidays. It is also fast and scalable, and similar to ARIMA, this model is interpretable, robust, flexible, and accurate [42].

C. Long Short-Term Memory (LSTM)

LSTM Model is a neural network model that can learn long-term dependencies in time series data, handle nonstationary and noisy data, as well as leverage additional features. It is also accurate, flexible, and scalable $[43,83]$.

1) Double Hybrid Forecasting Model: Sarima-Bidirectional LSTM

Review of published literature showcases the use of different forecast models and enhancements in COVID-19 research, demonstrating better accuracy and performance in forecasting by hybrid models. For example, ARIMA-LSTM hybrid model was used to predict future COVID-19 transmissions in China where ARIMA-LSTM model was paralleled by weight of regression coefficient performing better than ARIMA alone [45]; the same group also looked at COVID-19 prediction using data from Germany and Japan and utilized three enhanced hybrid models: PSO-LSTM-ARIMA, MLR-LSTM-ARIMA, and BPNN-LSTM-ARIMA. The research showed that BPNN-LSTM-ARIMA had the best prediction accuracy [46]. Priya and colleagues compared time series forecasting models utilizing ARIMA, Facebook Prophet, Holt-Winters Model, and Hybrid ARIMA-ANN (to take advantage of the unique characteristics of ARIMA and ANN models in linear and nonlinear modeling). The Hybrid model showed better accuracy and root mean square error [47]; Morais looked at forecasting daily Covid-19 cases with a hybrid

ARIMA and neural network model to capture the linear and non-linear structures of daily Covid-19 cases (MLP-ARIMA) [48]; and Nawi researched a hybrid ARIMA-SVM model [49]. Borges looked at COVID-19 ICU demand forecasting utilizing Prophet-LSTM approach vs a stand-alone approach in Brazil, confirming better performance of the hybrid model [50], and Long researched an efficient forecasting tool for Monkeypox outbreak in the US using ARIMA, Prophet, Neural Prophet, stacking model, and LSTM models. NeuralProphet achieved the optimal performance [51]. In addition, Guha, in his paper, presented two recurrent neural network-based approaches to predict the daily confirmed COVID-19 cases, daily total positive tests and total individuals vaccinated using LSTM and gated recurrent unit (GRU) [25]; Shastri looked at time series forecasting of Covid-19 using deep learning models: the recurrent neural network based variants of long-short term memory (LSTM) such as Stacked LSTM, Bi-directional LSTM and Convolutional [52]; Devaray utilized ARIMA, LSTM, Stacked LSTM (SLSTM) and Prophet approaches [53]; Zhenyu Li researched convolutional neural network combined with the stacked long-short-term-memory network model (CNN-Stack BiLSTM) [54]. The Stacked LSTM (SLSTM) model was also researched by Maaliw [55] and Ali, who also use the bidirectional enhancement to create a stacked Bi-directional long short-term memory (Stacked Bi-LSTM) network that forecasts COVID-19 more accurately [56]. Sah compared different COVID-19 forecasting models, Prophet, ARIMA, LSTM, and stacked LSTM-GRU models demonstrating better prediction results with the hybrid stacked LSTM-GRU model [57]. Other researchers looked at the Ensemble Empirical Mode Decomposition and Deep Learning creating an EEMD-LSTM hybrid model [58] and EEMD method with the Autoregressive Integrated Moving Average Exogenous inputs (ARIMAX) method, which they called EEMD-ARIMAX [59].

Hybrid models for this research were selected based on the literature search, strengths, and limitations of the individual components for forecasting performance, available enhancements to address limitations, and for their specific complementary characteristics that land them well for hybrid application. SARIMA-Bidirectional LSTM hybrid model combines the strengths of two powerful forecasting techniques, ARIMA enhanced with a seasonality component (the S) in SARIMA and enhancing the LSTM model to analyze data in both directions (Bidirectional component). This hybrid combines a linear and non-linear model, benefits from forecasting time series data that exhibits trends and seasonality and at the same time, an ability to learn long-term dependencies in time series data, as well as capture both forward and backward dependencies. SARIMA-Bidirectional LSTM complements the strength of each model and is expected to achieve better forecasting accuracy than either model individually [44].

2) Triple Hybrid Forecasting Model: Sarima-Prophet-Bidirectional LSTM

The triple hybrid SARIMA-Prophet-Bidirectional LSTM forecasting model enhances the previously mentioned hybrid model with a Facebook Prophet forecasting model that is specifically designed to forecast time series data that exhibits trends, seasonality, and holidays. The new triple hybrid combines the strengths of all three forecasting techniques with an ability to capture short-, medium-, and long-term dependencies, handle non-stationary and noisy data, and leverage additional features. Due to the complementary nature of the hybrid model components and a better fit for the data being researched, it would be expected that the new models would achieve better forecasting accuracy than either model individually.

3) Accuracy and Performance Assessment

Accuracy and performance assessment was conducted across all the models (foundational and hybrid models) evaluating vaccination and case fatality rates: Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and Entropy, relative to the actual data. In addition, the accuracy of the forecasting results of each model was compared with actual historical data from the Our World in Data dataset, specifically, to the actual time needed to reach the vaccination inflection points for each country.

4) Anomaly and Volatility Analyses

Anomaly and Volatility analysis and assessments were conducted across all-time series analysis and forecasting models utilizing Isolation Forest and Generalized Autoregressive Conditional Heteroskedasticity (GARCH) models, both well-studied in this field. These methodologies were selected based on the review of published literature that showcase their good performance as well as being valuable algorithms for anomaly and volatility detection in the context of COVID-19 vaccination forecasting [60]. The results obtained upon performing anomaly and volatility detection were used to select the best performing model for forecasting the time to COVID-19 vaccination inflection point for each country.

A. Isolation Forest

The last part of the research was focused on the assessment of anomaly and volatility detection analysis across the time series analysis models. These analyses were conducted to identify unusual or unexpected patterns in data, to prevent overfitting, improve the accuracy, performance, and reliability of machine learning models and complex systems. It is often used in Systems Engineering to detect unusual activity in system logs, performance bottlenecks in systems, and anomalous patterns in system data and to improve overall reliability, efficiency, and security of complex systems. The first algorithm used in this research is Isolation Forest.

Isolation Forest can detect anomalies in an unsupervised manner. This model is used to compare the accuracy of different forecasting models and considered to be efficient, scalable, and robust to outliers. It works by randomly selecting features and splitting values to create partitions of the data. This process is repeated until isolation of the anomalies. It is particularly well-suited for high-dimensional data, which is the case with COVID-19 vaccination data, which includes features such as vaccination rate, case fatality rate, population density, and socio-economic factors. It is also relatively insensitive to outliers, which can be a problem for other anomaly detection algorithms. Isolation Forest can be used to detect anomalies in the vaccination and case fatality rates. This can be useful for identifying periods where the vaccination and CFR rate are significantly higher or lower than expected, adjusting, or improving the forecasts for the vaccination inflection point [61].

Isolation Forest measured three parameters: Precision, Recall, and F1-score. Precision measures the proportion of detected anomalies that are actually true anomalies, where high precision (closer to 1) is very accurate in its anomaly detections, with few false positives. A good threshold for Isolation Forest is 0.7 or higher. Recall measures the proportion of true anomalies that are correctly identified by the model, high recall (closer to 1) means the model is sensitive and can capture most anomalies. A good threshold for Isolation Forest is 0.7 or higher. F1-score combines precision and recall into a single metric, balancing their trade-off. A high F1-score (closer to 1) indicates a good balance between precision and recall, suggesting a reliable anomaly detector. Isolation Forest results at 0.7 or higher for all parameters are considered to be good results [61].

B. Generalized Autoregressive Conditional Heteroskedasticity (Garch)

The second algorithm used to compare the accuracy of different forecasting models is the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model. The GARCH model is a powerful tool employed to capture and model volatility patterns in the residuals. This model considers the conditional variance and accounts for the time-varying volatility and is especially well suited for time-series analysis, which is the case with COVID-19 vaccination and case fatality rate data.

The GARCH model was used to forecast the volatility of the COVID-19 vaccination and CFR rate. This helped to identify periods where the vaccination and CFR rates are likely to increase or decrease more rapidly than expected. If the model detects anomalies, this could indicate that the vaccination and CFR rates are not following the expected patterns $[62]$.

Isolation Forest and GARCH models are both well-suited for anomaly and volatility detection, respectively, in the context of COVID-19 vaccination inflection point forecasting. They are both efficient, important for anomaly and volatility detection in large datasets, and robust to outliers. This can be a problem in COVID-19 vaccination data due to factors such as data entry errors and reporting delays. These models are also flexible, due to ease of adaptation to a variety of different anomaly detection tasks. The GARCH model also has several limitations, such as sensitivity to the choice of parameters, less robust performance for very short time series datasets, and the inability to capture all types of anomalies.

The GARCH model measures three parameters: Volatility Persistence, Relative Importance of ARCH Term, and Relative Importance of GARCH Term. Volatility Persistence represents the degree to which shocks to volatility persist over time, with an acceptable range between 0.7 and 1. Values below 1 are considered acceptable, ensuring stationarity of the volatility process. However, values closer to or exceeding 1, indicate stronger persistence, meaning shocks have longer-lasting impacts on volatility and might suggest issues like integrated volatility or model misspecification. The range that is typical and acceptable for Relative Importance of ARCH Term is 0 to 0.4. Relative Importance of GARCH Term captures the persistence of volatility shocks over time with an acceptable range of 0.3 to 0.9 [62].

1. Vaccination Inflection Point Score

Vaccination Inflection Point score was developed to categorize countries based on their actual time to achieving secondary vaccination inflection point, representing the time of the most significant CFR reduction post vaccination, and therefore, identifying the critical threshold signaling the turnaround in the pandemic. Countries were categorized into three groups corresponding to scores 1, 2, and 3, with a score of 1 indicating that the country needed the shortest amount of time to reach their secondary vaccination inflection point. This tool can help with the interpretation of changes in the pandemic dynamic, serve as a learning tool for the importance of the contribution of vaccination to achieving faster herd immunity, and improving the overall pandemic risk of countries.

III. RESULTS

3.1 Correlation Analysis Results

The correlation analysis was performed using Ordinary Least Squares Multifactor Regression Methodology. These analyses were performed as aggregate analysis with 14 variables. The correlation was assessed first with the vaccination rate as the target variable, followed by the case fatality rate. The top four variables most correlated with vaccination rate were: stringency_index, life_expectancy, positive_rate, and total_deaths. The top four variables for the case fatality rate were: stringency_index, aged_65_older, life_expectancy, and positive_rate. The top four variables that are the most correlated with both vaccination and case fatality rates together were derived by using the ranking order of variables across both vaccination and case fatality rates. The final ranking order of the four variables was: stringency_index, aged_65_older, life_expectancy, and positive_rate, representing the exogenous variables that were used in the primary and secondary vaccination inflection point forecasting analyses. The stringency index and positive rate were variables representing the PSI index and aged 65 and older and life expectancy represented the PHI index.

{"image_source":{"path":"images/57099d7fedc2f3fd742721ba203d66690b3315b02c5cf229c26c4c7629749343.jpg"},"content":"","chart_caption":[{"type":"text","content":"Figure 1: Correlation Analysis for Vaccination Rate (Aggregate Analysis)"}],"chart_footnote":[]}

{"image_source":{"path":"images/56f0b214e4a56edf858a5ac8118c6de668f2ad481453f94147e3b6a36024980a.jpg"},"content":"","chart_caption":[{"type":"text","content":"Figure 2: Correlation Analysis for Case Fatality Rate (Aggregate Analysis)"}],"chart_footnote":[]}

3.2 Forecasting Analysis Results

The summary of the conducted analyses is presented in Table 4, first as an aggregate and then per GDP per capita category (>$50,000,$35,000-35,000). Overall, all countries started their vaccination campaigns within the 43 days, starting with Latvia on December 4, 2020, and ending with the UK starting on January 10, 2021. Countries with the higher GDP initiated their vaccination efforts faster than the other countries (15 days vs 26 and 35 days), however, countries with the mid-range GDPs reached the PVIP and SVIP faster than the other two groups, with high and low GDP per capita. PVIP was reached in 37.5 days vs 76 and 131.7 days, and SVIP in 299.2 days vs 336.5 and 380.4 days.

Similar results were observed when median numbers were used, with the mid-range GDP countries again performing better, with the shortest time needed to reach both PVIP (34 days vs 80.5 and 82 days) and SVIP (316 days vs 343.5 and 365 days), and with the highest achieved vaccination rate (74.7% vs 70.6% and 63.3%), for GDP mid-range, high-range, and low-range respectively. Overall, all countries reached an average vaccination rate of 67.8% (mean) and 71.25% (median) at the time they observed the significant CFR drop post-vaccination (SVIP).

The highest vaccination rate was achieved in Portugal (89%) and the lowest in Bulgaria (28%).

Analysis of vaccinations by age group in Our World in Data (except for three countries) showed similar distribution by age $[20]$. The elderly population (60-70, 70-80, and 80+ years of age) achieved the highest vaccination rates in all, but three countries (Latvia, Romania, and Bulgaria), followed by the middle age group (18-24, 25-59). The smallest vaccination rates were observed in the youngest age group (0-17).

The data for the US and UK were not available in the Our World in Data dataset, however, data from official government sites demonstrated the same patterns observed with the rest of the countries $[81,82]$, supplement Tables S10, S11, and S12. There were no official records available for Serbia at the time of this research. This confirms earlier statements that most countries prioritize elderly and frail populations in their vaccination campaigns. Looking at the countries based on their GDP per capita grouping, the mid-range group on average achieved higher vaccination rates of the elderly population than the countries with higher and lower GDP per capita. These findings support the better performance of the countries in the mid-range

GDP group, demonstrating the importance of prioritizing the needs of the elderly population (age 65 and older and life expectancy) in a pandemic setting. It should be assumed that other factors, such as acceptance and robustness of the vaccination campaign and vaccination mandates imposed by governments played a significant role as well [71].

Table 5 presents the ranking order of the countries based on the time to reach SVIP. The UK observed the SVIP in the shortest amount of time, at 161 days (CFR 3.3%, vaccination rate

63.8%), while Romania reached the same point in 560 days (CFR 2.2%, vaccination rate 41.6%).

Supplemental Tables (Table S1, S2A-B, S3A-B) present all results of all forecasting models, the three foundational (ARIMA, PROPHET, LSTM) and the two hybrid forecasting models (double hybrid: SARIMA-Bidirectional LSTM, and triple hybrid: SARIMA-Prophet-Bidirectional LSTM). The baseline data for each country, as well as the actual historical data from the COVID-19 pandemic are also documented in these supplemental tables.

In the dataset used for this research, 65% of countries started their vaccination efforts in December 2020, and 35% started in January 2021. The primary vaccination inflection point representing the first observed reduction in the CFR post vaccination was reached at 83.27 days (mean, range 15-367 days), with 42% of countries seeing the initial impact in less than 50 days, 38.4% in 50-100 days, and 19.2% above 100 days (Figure 3). This reduction was achieved with the initial vaccination rate of 31.1% (mean, range 0.1% to 50%), with 27% of countries reaching the vaccination rate of >25%, 15.3% reaching the rate between 11-25%, and 57.7% reaching the rate of <10% (Figure 4). Finland observed the fastest PVIP in only 15 days (CFR 1.6%, vaccination rate of 1.1%), while Romania had the longest wait to first reduction at 367 days (CFR 3.2%, vaccination rate 27.8%).

The secondary vaccination inflection point (SVIP), representing the most significant reduction in CFR post vaccination, signaling the start of the continuous CFR reduction and turnaround in the pandemic, was reached at 339.31 days (mean, range 161-560 days), with 23.1% of countries observing this impact in less than 300 days, 53.8% from 300-370 days, and 23.1% in more than 370 days (Figure 5). This reduction was achieved with the cumulative vaccination rate of 67.8% (mean, range 8%-89%), with 50% of countries reaching the vaccination rate between 50-75% (Figure 6). Most of the countries reached a significant drop in the CFR in 2021 (73%), out of which 61.5% reached it in the 4 ${}^{th}$ quarter of 2021, 11.5% in the 3 ${}^{rd}$ quarter of 2021, and 27% in early 2022. The highest vaccination rate at this inflection point was achieved in Portugal (89%) on November 16, 2021.

{"image_source":{"path":"images/5b9d93d74d4d9776e50f435c0a9af0adace57596cdd70a6b539e216a784df7cf.jpg"},"content":"","chart_caption":[{"type":"text","content":"Percentage of Countries Reaching PVIP Per Time Category"},{"type":"text","content":"Figure 3: Percentage of Countries Reaching PVIP Per Time Category"}],"chart_footnote":[]}

{"image_source":{"path":"images/553d1e388b372fee6bc72169d2eaeca254c0abc497e4b108041f44173cc64dcb.jpg"},"content":"","chart_caption":[{"type":"text","content":"Figure 4: Percentage of Countries Per Vaccination Rate Categories (PVIP)"}],"chart_footnote":[]}

{"image_source":{"path":"images/e01f78ff4eea42714ab83748df9a1955d94313e574bdc92037e4d7ec9f618e1d.jpg"},"content":"","chart_caption":[{"type":"text","content":"Percentage of Countries Reaching SVIP Per Time Category"},{"type":"text","content":"Figure 5: Percentage of Countries Reaching SVIP Per Time Category"}],"chart_footnote":[]}

{"image_source":{"path":"images/7255356a9f6e6139850383d65f9f992045477a6d828642c31e2a19ce66dde22b.jpg"},"content":"","chart_caption":[{"type":"text","content":"Figure 6: Percentage of Countries Per Vaccination Rate Categories (SVIP)"}],"chart_footnote":[]}

Overall, at the time of the SVIP, all countries with the exception of three, showed a reduction in the CFRs relative to the CFRs at the beginning of the vaccination. The highest CFR at the time of the SVIP was in Bulgaria (4.1%), followed by the UK (3.32%), and Italy (2.75%). Belgium and Romania had the CFRs that were > 2.0%, and the remaining countries had the CFRs <2.0%. The lowest CFRs were documented in Cyprus (0.42%) and Iceland (0.45%). Bulgaria, Latvia and Slovakia had the CFRs at the SVIP that were higher than the CFR at the vaccination start date, however, all three countries showed a reduction in the CFRs from the PVIP to the SVIP, indicating a positive impact of the vaccination.

3.3 Accuracy and Performance Assessment

Accuracy and performance assessment was conducted across all the models (foundational and hybrid models) evaluating vaccination and case fatality rates: Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and Entropy, relative to the actual data. Tables 6-7 showcase the mean and median results for all calculated metrics indicating the superior performance of the triple hybrid model SARIMA-Prophet- Bidirectional LSTM.

3.4 Anomaly and Volatility Analysis Results

Anomaly and Volatility analysis and assessments were conducted across all time-series-analysis and forecasting models utilizing Isolation Forest and Generalized Autoregressive Conditional Heteroskedasticity (GARCH) models. In the Isolation Forest model, precision, recall, and F1-score values above 0.7 indicate good performance. As presented in Tables 8 and 9, both mean and median values were above 0.7, indicating that all forecasting methodologies are performing well and accurately, validating performance of all forecasting models. In the GARCH model, Volatility Performance between 0.7-1, Relative Importance ARCH Term between 0-0.4, and Relative Importance of GARCH Term between 0.3 - 0.9, indicate good performance.

Tables 10 and 11 presented that both mean and median values are within typical and acceptable ranges for all three indicators, suggesting that all forecasting methodologies are performing well

3.5 Vaccination Inflection Point Score Results

Vaccination Inflection Point score was developed to categorize countries based on their actual time to achieving secondary vaccination inflection point, representing the time of the most significant CFR reduction post vaccination. Countries were categorized into three groups with scores 1, 2, and 3, with a score of 1 indicating the country needing the shortest amount of time to reach their secondary vaccination inflection point.

Table 12 and Figure 7 present the distribution of countries per VIP score. This data indicates that the majority of countries (53.8%) reached the SVIP between 300-370 days (score 2). While there is a broad range in the achieved vaccination rates across different countries, the median values show numerically higher vaccination rates in the countries with the shortest time to SVIP, score 1 (72.75%), over score 2 (71.75%), and score 3 countries (62%).

{"image_source":{"path":"images/b913f36f893e6c75d9c40dce07efdb422dcc42fe233e8aad8e7610f1fbdf6f64.jpg"},"content":"","chart_caption":[{"type":"text","content":"Figure 7: Percentage of Countries per days to reach SVIP"}],"chart_footnote":[]}

IV. DISCUSSION

The significant negative impact of the COVID-19 pandemic highlighted the need for better preparedness of countries and more sophisticated tools to guide the efforts of public health officials and governments. A wealth of data collected during the COVID-19 pandemic and advanced AI methodologies are allowing researchers to expand our collective knowledge and guide these efforts. This paper builds on the research that highlighted the importance of the non-pandemic predictors in the assessment of pandemic risk $[35]$. It expands into pandemic predictors and utilizes COVID-19 actual data to refine forecasting tools for future outbreaks. This research was conducted to identify the vaccination inflection points and the time needed to reach the critical cumulative vaccination rate thresholds to observe continuous decrease of the case fatality rates. It was conducted both at the aggregate and country levels, signaling the turnaround point in the pandemic.

The analysis of the actual COVID-19 historical data shows that all of the countries, in aggregate, had the highest fatality rates during the first year of the pandemic. Implementation of the pandemic measures, such as masks, social distancing, school and workplace lockdowns, and testing, had a significant impact on the initial lowering of the case fatality rates. With the introduction of first vaccines in December of 2020, the case fatality rates decreased even further, often reflected as steep downward slopes in graphs. Several types of vaccines were available at the time of the initial vaccination: genetically engineered messenger RNA, viral vector vaccines, and protein subunit vaccine $[63]$. The initial vaccinations were delivered as single dose or 2-dose vaccines, followed by single dose booster vaccines to improve already established immunity. The first booster dose was approved for use in the third quarter of 2021, followed by two in 2022, and one in 2023, for a total of four booster doses, in developed countries $[16]$. As of today, there are approximately 40 different vaccines that were approved by regulatory agencies for full emergency use authorization across different countries $[36]$.

In the dataset used for this research, 65% of countries started their vaccination efforts in December 2020, and 35% started in January 2021. On average, looking at the mean values, the time to reach the primary vaccination inflection point, the first reduction in the case fatality rate post vaccination, was on day 83.27 at the vaccination rate of 31%. The secondary vaccination inflection point, representing the most significant and continuous CFR drop post vaccination, was reached at day 339.31 at the average vaccination rate of 67.8%. All four parameters had a very large range, signaling the presence of outliers. Median values indicate a shorter time to reach the PVIP (57.5 days), lower vaccination rate at PVIP (6.05%), a longer time to reach the SVIP (355.5 days) and a higher overall vaccination rate at SVIP (71.25%). Countries with the mid-level GDP per capita implemented their vaccination campaigns in the most successful way, securing the shortest times to reach both vaccination inflection points looking at both mean and median values. Regarding the individual countries, Finland was the first country to reach the PVIP in only 15 days with the vaccination rate of 1.1%, while Romania had the longest wait, reaching the PVIP in 367 days with a vaccination rate of 27.8%. The UK observed the most significant CFR reduction (SVIP) in the shortest amount of time in 161 days (vaccination rate 63.8%), while Romania reached the same point in 560 days (vaccination rate 41.6%). The highest vaccination rate at SVIP was achieved in Portugal (89%) on November 16, 2021. The SVIP score was developed to categorize countries based on their actual time to achieving secondary vaccination inflection point. The majority of countries reached the SVIP between 300-370 days, while the countries with the lowest score (shorter time to SVIP) had the highest median vaccination rates. This tool can help with the interpretation of changes in the dynamics of the pandemic.

Looking specifically at the US (Supplement Table S1), PVIP was achieved after 94 days post-vaccination, at a vaccination rate of 24.8% and CFR of 1.8%. The most significant reduction in CFR (SVIP) was achieved at 363 days after the vaccination start date or 269 days after the PVIP date. The vaccination threshold at the SVIP was 70.3% with the CFR of 1.59%. The CFR reduced from 1.82 at the time of start of vaccination to 1.59 at the time when it reached the SVIP. In most countries, including the US, priority for COVID-19 vaccination was given to health care workers, residents and personnel of long-term care facilities, elderly patients, and patients with certain comorbidities. The US was grouped with Switzerland, Luxembourg, and Ireland in the high GDP per capita countries (>$50,000). Within this group, the US was the first country to start the vaccination campaign; however, it needed a longer time than other countries to reach both vaccination inflection points. These results were most likely influenced by the impact of widespread anti-vaccine campaigns, scientific misinformation, and overall lack of readiness of certain parts of the population to support government efforts [21, 22, 23].

It is important to recognize that the impact of vaccination is dependent on many factors, such as speed of implementation of the campaign, availability of vaccines, acceptance of the vaccine by the targeted population, and others. The direct impact of vaccination at the population level will often lag and the data may show some initial misalignment that can be explained. For example, the most significant reduction in CFR in the US was observed in December 2021, signaling the turnaround of the pandemic in the US, with the steady decline of the ratio of total infections and death cases. However, in the next few months in 2022 there was a significant increase in new infections and deaths [20]. While the vaccination rate in the US at that time was reaching $70\mathrm{\%}$, it can be assumed that the increase in new cases was caused by several factors, such as the delay in immunity development post vaccination, breakthrough infections, lack of booster vaccination, higher vulnerability of the unvaccinated population, relaxation of pandemic measures and, most significantly, the emergence of new variants with limited immunity coverage from existing vaccines (e.g., omicron variant BA.2.86 that emerged in Nov of 2021).

To understand the findings and applications of this research, it is important to examine the potential variables that may have influenced the results. This research indicates that there are four most important factors that influence the vaccination, and the case fatality rates in any country. Two are non-pandemic variables (not immediately influenced by the pandemic): percentage of people in the population who are 65 years of age or older, and the life expectancy of the population. An additional two variables are pandemic variables: percentage of people who had a confirmed COVID-19 infection with testing, and the level and scope of the pandemic measures that were implemented. The final ranking order of importance of the four variables was: stringency_index, aged_65_older, life_expectancy, and positive_rate. It would be expected that the same factors would be the most important in a potential new pandemic as well, due to the increased vulnerability of the elderly and sick patient populations to any infectious disease, and the importance of the infection transmission rates and the speed of implementation of response measures. Simply put, if the vaccines were available at the outbreak of COVID-19 and campaigns were implemented fast in targeted populations, the world would not have a pandemic. This is critical learning highlighting the need to take care of the most vulnerable parts of the populations and implementing appropriate procedures for testing, vaccination, and other public health measures.

This research may have been influenced by the inherited challenges of the vaccination process. Published literature highlights the challenges introduced by the disparity in the distribution of COVID-19 vaccines, where the majority of the vaccines were initially delivered to high – and upper middle-income countries vs lower-income countries $[64]$. This was evident by the differences in times to first vaccination inflection point, demonstrating that lower-income countries had a higher case fatality rate and needed a longer time to observe the CFR reduction as a result of vaccinations, than the higher-income countries. Lack of availability of sufficient doses of vaccines, less organized execution of vaccine campaigns, including the order of vaccination (elderly and immunocompromised population) may have also influenced the results across countries. In addition, factors affecting vaccination acceptance, confidence in safety and efficacy and the risk of side effects, preference for natural immunity, scientifically sounding misinformation, as well as different cultures and political systems, also played a role in the observed vaccination patterns, spread of infection, and mortality of COVID-19 $[65,66,67,68,69,70]$.

All foundational forecasting methodologies utilized in this research (ARIMA, Prophet, and LSTM) showed good accuracy and precision, with only small numerical differences in results, relative to the actual values. They performed well and continue to be a true foundational platform for time series forecasting. Utilization of enhancement features to improve limitations of foundational models is already an established approach, and customizing enhancement based on specificities of data allows for more robust analyses. Combining models into hybrids of foundational or foundational with enhancement models is a newer approach requiring validation. The two hybrid forecasting models (double hybrid: SARIMA-Bidirectional LSTM, and triple hybrid: SARIMA-Prophet- Bidirectional LSTM) utilized in this research are both novel models and their validation was conducted by comparing them to foundational models alone, to each other, and to the actual historical data. They both performed well with high accuracy and precision, and better than the foundational models.

However, the performance and accuracy of the triple hybrid SARIMA-Prophet-Bidirectional LSTM model was superior to other models. In addition, the anomaly and volatility detection analyses, conducted using Isolation Forest and GARCH models, validated performance of all forecasting methodologies, reporting all indicators within the typical and acceptable ranges. In summary, all foundational and hybrid models used for forecasting showed comparable results at the primary and secondary vaccination inflection timepoints and performed with high accuracy relative to the actual data. The best performance was observed with the novel triple hybrid SARIMA-Prophet-Bidirectional LSTM, indicating that hybrid models, combining models with enhanced capabilities, can result in higher accuracy and greater sophistication in analysis. Ability to predict the vaccination inflection point and measure its immediate, as well as the most pronounced impacts, allows for a deeper understanding of the dynamics between the vaccination and case fatality rates. In addition, it is important to remember that the data for this research was trained based on the specificities of the COVID-19 pandemic. For the use of these forecasting models for future pandemics, they may need to be re-trained with the data specific to the new pandemic.

The results of this research can guide countries in the assessment of the pandemic risk and inform public health policy makers in creating measures to minimize the impact of any potential infectious disease pandemic on the people, environment, and socio-economic systems. It is determined that countries can achieve a maximum vaccination rate of 70% with milder measures, and that 90% can be reached only with strict mandates imposed by governments [71]. This highlights the need to plan, organize and execute efficient vaccination campaigns, and improve surveillance and monitoring to substantially reduce morbidity and mortality and avoidance of breakdown of health care systems in countries to control potential new pandemics [72, 73, 74, 75, 76, 77, 78, 79, 80].

V. CONCLUSION AND FUTURE RESEARCH

In conclusion, the research conducted in this paper will add to the knowledge base in the areas of machine and deep learning, and public health. It demonstrates that the novel hybrid time series forecasting models, combining foundational models with enhanced features, provides better performance and higher accuracy over traditional foundational models. The performance and accuracy of the triple hybrid SARIMA-Prophet-Bidirectional LSTM model was superior to other models and was successfully validated with anomaly and volatility detection analyses. In addition, it shows that 42% of countries had seen an immediate effect of vaccination in <50 days, and 23.1% of countries reached the most pronounced impact in <300 days, suggesting the need for improvements. Applying advanced AI methodologies to forecast time to country specific vaccination inflection points, and assessing the vaccination rates relative to the case fatality rates, can provide another useful tool to guide countries in their pandemic risk preparedness.

This paper has several limitations that can be utilized to guide further research, such as: (1) inherited limitations and variabilities of the vaccination campaigns in different countries (supply, distribution, new variants reducing the effectiveness of current vaccines; (2) differences in the health system infrastructures, speed and scope of implementation of other pandemic measures across countries; (3) limitations of the Our Word In Data dataset (e.g., size, completeness, and accuracy, due to the voluntary data reporting and possible underreporting of infection and death cases; and (4) selection of machine and deep learning methodologies and enhancements.

Conflict of Interest

Marco M. Vlajnic and Steven J. Simske declare that there is no conflict of interest.