Graphical abstract

Introduction

The coronavirus disease 2019 (COVID-19) pandemic, caused by the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has led to an unprecedented global health crisis. As of October 23, 2023, the world has seen 771,679,618 confirmed cases of COVID-19 and 6,977,023 deaths [1]. The development and distribution of vaccines to combat this viral threat have been crucial in mitigating the pandemic. A variety of COVID-19 vaccines with different mechanisms of action, formulations, and efficacy profiles have been developed and administered across the globe [2]. The urgent need to address this pandemic necessitated a rapid rollout of these vaccines. It is therefore vital to evaluate and compare their effectiveness in curbing the spread of the virus and reducing the severity of the disease.

Quantifying the real-world effectiveness of COVID-19 vaccines against specific viral variants and in diverse populations has emerged as a top research priority. Research on vaccine effectiveness (VE) has traditionally been focused on determining absolute VE (aVE), which compares outcomes between vaccinated and unvaccinated individuals. In contrast, relative VE (rVE) is often employed to assess the comparative risk reduction benefits provided by different vaccine products [3]. Systematic reviews of the methodologies used in comparative studies can be essential for synthesizing and critically appraising the available evidence. These reviews are instrumental in guiding policy decisions, shaping vaccination strategies, and improving our understanding of the real-world impacts of COVID-19 vaccines.

This study aimed to review the methodological approaches employed in research comparing the effectiveness of COVID-19 vaccines. We investigated the designs, study populations, methods for VE estimation, and outcomes of these studies. The primary objectives were to understand relevant trends, refine our comprehension of VE, and identify potential areas of improvement for VE assessment in various settings.

Materials and Methods

Eligibility Criteria

The search and review processes adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Only studies published in English were included. To encompass research on the relative effectiveness of COVID-19 vaccines that did not provide relative estimates, we also incorporated studies on VE.

Information Sources

On June 13, 2024, we systematically searched several databases, including PubMed, Embase, and the Cochrane Library.

Search Strategy

The search terms used were:

1. “COVID” OR “COVID-19” OR “Coronavirus” OR “SARS-COV-2”

2. “vaccine” OR “vaccination”

3. “relative effectiveness” OR “relative efficacy”

4. #2 AND #3

5. “relative vaccine effectiveness” OR “relative vaccine efficacy” OR “relative VE”

6. “vaccine effectiveness” OR “VE”

7. #4 OR #5 OR #6

8. #1 AND #7

Selection Process

In accordance with the search strategy, articles were initially screened to identify and remove duplicates. Three researchers (EB, JS, and SAC) independently screened the titles, abstracts, and full texts of the remaining studies. E.B. then verified the review results for consistency and accuracy. The inclusion criteria restricted the studies to interventional or observational research that provided COVID-19 VE estimates for 2 or more vaccine platforms. These studies were required to report outcomes of COVID-19 infection, symptomatic disease, severe disease, COVID-19–related hospitalization, intensive care unit (ICU) admission, and/or death. Given the scarcity of comparative studies on protein versus non–protein and monovalent versus bivalent vaccines, our review was focused on studies comparing mRNA and non-mRNA vaccines. We excluded studies evaluating the VE of a single COVID-19 vaccine type; those unrelated to COVID-19 VE; those concentrating on laboratory measurements of immunogenicity, determinants of vaccine uptake, or cost-effectiveness; simulation studies; and comparisons of vaccination schedules and doses. Additionally, we excluded preprints, systematic reviews, ongoing studies, Delphi studies, modeling studies, and articles without full-text availability.

Data Items

We extracted data on the publication year, country, study design, sample size, and composition of study participants, as well as the definition of VE, outcomes, and covariates. In the assessment of rVE studies, we collected information on the availability of aVE estimates and the criteria used to define aVE. Vaccine types were classified into 3 primary categories: mRNA versus non-mRNA, protein-based versus non–protein-based, and monovalent versus bivalent COVID-19 vaccines. Our analysis evaluated the effectiveness of COVID-19 vaccines in preventing infection, symptomatic disease, severe disease, hospitalization, ICU admission, and mortality, with results expressed in either relative or absolute terms. Additionally, 3 reviewers conducted the bias assessment. In accordance with the World Health Organization (WHO) interim guidance on evaluating COVID-19 VE [4], we assessed the extent to which studies adjusted for core covariates in 4 categories: time (such as date of illness onset or date of specimen collection), susceptibility to infection (namely previous COVID-19 infection), sociodemographic factors, and chronic medical conditions.

Assessment of Study Risk of Bias

The Risk of Bias in Non-randomized Studies (ROBINS-I) tool [5] was employed to assess the risk of bias, which was categorized as “low,” “moderate,” “serious,” “critical,” or “no information” across 7 domains for observational studies.

This research has been registered with PROSPERO (https://www.crd.york.ac.uk/prospero/), an international prospective register of systematic reviews, under the registration number CRD42023442560.

Results

Study Selection

The initial review of PubMed, Embase, and the Cochrane Library yielded a total of 2,921 articles. After 1,054 duplicates were removed and the inclusion criteria were applied, 151 articles remained for analysis (Figure 1).

Study Characteristics

rVE was estimated to compare mRNA vaccines (Pfizer, also known as BNT162b2 and Comirnaty, and Moderna, also known as mRNA-1273 and Spikevax) with non-mRNA vaccines (AstraZeneca, Janssen, Novavax, Sinovac-Coronavac, Sinopharm, Sputnik V, and Bharat-Biotech Covaxin [BBV152]). This comparison included 20 studies (13.2%), with 5 studies examining monovalent and bivalent mRNA COVID-19 vaccines (3.3%). In contrast, VE was estimated for mRNA versus non-mRNA vaccines in 118 studies (78.1%), monovalent versus bivalent mRNA vaccines in 5 studies (3.3%), and protein-based versus non–protein-based COVID-19 vaccines in 3 studies (2.0%). Most of these studies employed a retrospective cohort design (n=70, 46.4%), followed by test-negative (n=39, 25.8%), prospective cohort (n=18, 11.9%), case-control (n=12, 7.9%), and cross-sectional (n=3, 2.0%) designs. The most commonly used definitions of rVE were hazard ratio (HR; n=7, 35.0%), the formula (1−HR)×100 (n=6, 30%), and odds ratio (OR; n=4, 20.0%). For aVE, the definitions used were the formulas (1−OR)×100 (n=54, 42.9%), (1−HR)×100 (n=34, 27.0%), and (1−risk ratio [RR])×100 (n=10, 7.9%) (Figure 2).

Risk of Bias in Studies

We evaluated 25 studies assessing rVE for COVID-19 vaccines and 126 studies assessing only VE, using the ROBINS-I tool for quality assessment. Of the former, 20 studies were considered to have a low risk of bias. Two studies were categorized as having “no information” regarding risks, 1 study was identified as having a critical risk of bias, 1 presented a serious risk, and the remaining study displayed a moderate risk (Figure 3) [6]. For the study presenting critical risk, the potential bias stemmed from the lack of adjustment for confounding variables [7]. Outcome measurement was not specified in the databases of 2 studies [8,9], and 1 study did not mention the database source of exposure information [8]. Regarding the 2 studies identified as having serious or moderate selection bias risk, the cause was the limited sample size in 1 study [7] and the selection of volunteer participants in the other [10].

In studies assessing VE alone, the overall risk of bias was categorized as moderate in 51 studies (40.5%), low in 46 (36.5%), and indeterminate due to lack of information in 14 (11.1%). Additionally, 7 studies (5.6%) were classified as having a serious risk of bias, while 8 studies (6.3%) were deemed to be at critical risk (Figure 4) [6]. Regarding the confounding bias domain, 24 studies (19.0%) were found to have a moderate risk of bias, 4 (3.2%) [11–14] had serious risk, and 4 (3.2%) [15–18] had a critical risk. In terms of deviations from planned interventions and reporting of results, all studies were considered to have either a low or a moderate risk of bias. Participant selection bias was rated as serious in 3 studies [19–21] and as critical in 2 studies [22,23]. The risk of bias in intervention classification was moderate in 11 studies and critical in 3 studies [15,24,25]. Information on missing data was not provided in 39 (37.9%) studies, and 6 studies (5.8%) [26–31] were assessed as having a moderate risk in this domain. Outcome measurement information was absent in 1 study [32], and a moderate risk of bias in outcome measurement was identified in 11 studies (10.7%).

Methodology of rVE Studies

Overall, 59,915,206 participants were included in 20 comprehensive studies that investigated the rVE of various mRNA and non-mRNA COVID-19 vaccines [7–10,33–48]. The studies were published between 2021 and 2024, with the research conducted from 2020 to 2022 (Table 1) [7–10,33–48]. The largest proportion of studies took place in the United States (n=6, 30.0%) [34,36,39,40,43,45], followed by the United Kingdom (n=5, 25.0%) [33,35,42,44]. Most studies targeted the general population (n=15, 75.0%), with some focusing on hospitalized patients (n=4, 20.0%) [10,39,41,43] and healthcare workers (n=1, 5.0%) [7]. Regarding study design, the evaluation of rVE among COVID-19 vaccine types utilized various methodologies, including retrospective cohort (n=11; 55.0%) [7,8,34–36,39–41,45,46,48], prospective cohort (n=6; 30.0%) [9,10,33,37,38,44], and case-control (n=3; 15.0%) designs [42,43,47]. The age distribution of participants varied, with 12 studies including individuals aged ≥18 years; 1 study each encompassing participants aged ≥12, ≥16, >20, >40, and >50 years; and 3 studies (15.0%) that did not provide age specifications.

The comparative analyses in these studies were primarily focused on the Pfizer and Moderna vaccines (n=9, 45.0%) [8,10,34,38–41,45,47], with other studies examining only the Pfizer vaccine (n=9, 45.0%) [7,9,33,35,37,42,44,46,48], only the Moderna vaccine (n=1, 5.0%) [43], or an unspecified mRNA vaccine (n=1, 5.9%) [36]. The comparator vaccines evaluated against mRNA vaccines included Janssen (n=10, 58.8%), AstraZeneca (n=9, 52.9%), and Sinovac-Coronavac (n=2, 11.8%). The most frequently reported outcomes across the studies were COVID-19–related infection (n=14, 70.0%), hospitalization (n=12, 60.0%), and mortality (n=6, 30.0%). The most common definitions of rVE were HR (n=7, 35.0%) [7,10,33,34,40,42,44], followed by the formula (1−HR)×100 (n=4, 20.0%) [36,37,39,45], OR (n=4, 20.0%) [9,41,43,47], incidence rate ratio (n=2, 10.0%) [8,35], the formula (1−RR)×100 (n=2, 10.0%) [38,48], and RR (n=1, 5.0%) [46]. Only 8 studies (40.0%) reported and defined both aVE and rVE [9,33,35,38,43,44,47,48].

Among the studies, the variables most commonly controlled for were age (n=16, 94,1%), sex (n=15, 88.2%), and comorbidities (n=14, 82.4%). Less frequently controlled variables included ethnicity (n=7, 41.2%), socioeconomic status (SES) (n=6, 35.3%), and body mass index (BMI; n=5, 29.4%), as well as geographic region and previous healthcare utilization (both n=4, 23.5%); calendar time, time of vaccination, and smoking status (each n=3, 17.6%); and care home residential status, influenza vaccine history, number of prior tests for COVID-19, COVID-19 infection history, immunocompromised status, and antibody levels (each n=2, 11.8%). Additional variables included learning disability, serious mental illness, current pregnancy, eligibility for first vaccine dose, clinical characteristics, electronic frailty index score, and alcohol consumption. Of the studies, 7 (35.0%) adjusted for the 3 WHO-recommended core sociodemographic covariates of age, sex, and SES; 15 (75.0%) adjusted for chronic medical conditions; 6 (30.0%) for time-related variables (date of illness, date of testing); and 3 (15.0%) for susceptibility to infection. Only 1 study accounted for all of the WHO-recommended core confounding variables, including age, sex, SES, chronic medical conditions, time-related variables, and susceptibility to infection [34]. One study did not make adjustments for any variables [7].

Only 5 studies investigated the rVE of bivalent versus monovalent mRNA vaccines (Table 2) [49–53]. These studies, conducted between 2022 and 2024 in the Netherlands, Japan, Italy, the United Kingdom, and the United States, encompassed a total of 2,253,040 individuals. The rVE assessment predominantly employed a test-negative design, using the definition (1−OR)×100. Among these 5 studies, 4 (80.0%) adjusted for the WHO-recommended core sociodemographic variables of age, sex, and SES along with chronic medical conditions, and 3 studies (60.0%) considered the core variables related to time and susceptibility to infection. Two of these studies adjusted for all WHO core confounding variables [49,50].

Methodology of aVE Studies

Across the 118 studies of VE for mRNA versus non-mRNA vaccine types, the cumulative sample size was 203,805,491 participants (Table S1). The greatest proportion of these studies took place in 2021 (n=50, 42.4%), followed by 2022 (n=28, 23.7%) and 2020–2021 (n=26, 22.0%). The United Kingdom was the most common study location (n=25, 21.2%), with the United States (n=17, 14.4%), Hong Kong (n=7, 5.9%), China and Spain (each n=6, 5.1%), Italy (n=5, 4.2%), and several other countries, including Canada, Brazil, Colombia, and France (each n=4, 3.4%), also contributing. The studies predominantly included participants from the general population (n=77, 65.3%). Hospitalized patients were the next largest group (n=24, 20.3%), followed by hospital workers (n=7, 5.9%), urgent care patients and long-term care residents (each n=2, 1.7%), patients in hospital wards and public sector healthcare users (each n=1, 0.8%), children (n=3, 2.5%), and military medical academy personnel (n=1, 0.8%). Among hospitalized patients, specific subgroups were identified, including patients undergoing hemodialysis (n=3, 2.5%) [54–56], individuals with kidney disease (n=2, 1.7%) [57,58], solid organ transplant recipients (n=2, 1.7%) [59,60], patients using immunosuppressants (n=1, 0.8%) [61], and patients with cirrhosis (n=1, 0.8%) [62]. The studies involved participants with distinct attributes, such as US military personnel [63], attendees of a human immunodeficiency virus outpatient clinic [15], public emergency department patients [20], and those who underwent pharmacy-based testing for COVID-19 [64]. Concerning age, 68 studies (57.6%) involved adults aged 16 years and older, 18 studies (15.3%) included adults aged 50 years and older, and 9 studies (7.6%) did not specify participant ages. Seven studies (5.9%) encompassed participants of all ages. Three studies (2.5%) focused on children aged 3 to 17, 3 to 18, and 5 to 11 years, respectively. The remaining studies included age groups of ≥5, ≥10, ≥12, and ≥15 years.

The primary comparator among the mRNA vaccine groups was most frequently Moderna and Pfizer (n=60, 50.8%), followed by Pfizer alone (n=56, 47.5%), Moderna alone (n=1, 0.8%), and unspecified mRNA vaccine (n=2, 1.7%). Other comparator groups included AstraZeneca (n=75, 63.6%), Janssen (n=36, 30.5%), Sinovac-CoronaVac (n=32, 27.1%), Sputnik V (n=10, 8.5%), Sinopharm (n=9, 7.6%), a combination of Sinovac-CoronaVac and Sinopharm (n=3, 2.5%), Cansino (n=3, 2.5%), Bharat-Biotech BBV152 (n=1, 0.8%), and others (n=3, 2.5%). The studies primarily assessed the outcome of COVID-19 infection (n=76, 64.4%), followed by hospitalization (n=56, 47.5%), death (n=48, 40.7%), symptomatic disease (n=18, 15.3%), severe disease (n=18, 15.3%), ICU admission (n=4, 3.4%), all-cause mortality (n=4, 3.4%), reinfection (n=2, 1.7%) and breakthrough infection (n=1, 0.8%). The most common VE definitions were (1−OR)×100 (n=50, 42.4%), (1−HR)×100 (n=34, 28.8%), and (1−RR)×100 (n=10, 8.5%).

Age was the most frequently controlled variable across the studies (n=99, 83.9%), followed by sex (n=89, 75.4%), comorbidity (n=58, 49.2%), residential status (n=47, 40.0%), calendar time (n=32, 27.1%), SES (n=28, 23.7%), ethnicity (n=24, 20.3%), COVID-19 testing date (n=12, 10.2%), care home residential status (n=12, 10.2%), number of prior COVID-19 tests (n=15, 12.7%), previous healthcare utilization (n=16,13.6%), history of COVID-19 infection (n=10, 8.5%), clinical characteristics (n=8, 6.8%), BMI (n=7, 5.9%), influenza vaccine history (n=6, 5.1%), vaccination status (n=11, 9.3%), time of vaccination (n=8, 6.8%), and immunocompromised status (n=2, 1.7%). Additional confounding variables considered in these studies included incidence rate, the interval between COVID-19 onset and treatment initiation, cumulative person-days, time on dialysis, test type (antigen or genomic), local virus circulation, contact setting (household or other), travel history, and non-comorbid conditions. Of the 118 studies, 22 (18.6%) adjusted for the WHO-recommended core sociodemographic variables, including age, sex, and SES; 58 (49.2%) accounted for chronic medical conditions; 36 (30.5%) considered time-related factors; and 12 (10.2%) adjusted for susceptibility to infection. Nine studies did not incorporate adjustments for any variables.

Five studies compared the VE of bivalent versus monovalent mRNA vaccines (Table S2). These studies were conducted between 2023 and 2024 in Denmark, Finland, Norway, Sweden, the Republic of Korea, Canada, and the United States, including a total of 6,163,590 adults and 90,905 children. The VE assessments primarily utilized a prospective cohort design, employing 5 distinct definitions: 1−RR, 1−HR, (1−OR)×100, HR, and 1−OR. Of the 5 studies, 1 (20.0%) adjusted for the 3 WHO-recommended core sociodemographic variables of age, sex, and SES. Three studies (60%) adjusted for chronic medical conditions, 2 (40%) considered core time-related covariates, and 1 (20%) accounted for susceptibility to infection.

Three studies assessed the VE of COVID-19 vaccines, differentiating between protein-based and non–protein-based vaccines. These studies were conducted in Russia, with a combined population of 327,997 individuals. Participants in the studies were adults aged 18 years and older. The methodologies employed were test-negative [65], retrospective cohort [18], and case-control [66] designs. To calculate effectiveness, all studies utilized the formula (1−OR)×100. In all 3 studies, a protein-based vaccine (EpiVacCorona) was compared with the non–protein-based vaccines Gam-COVID-Vac and CoviVac. The COVID-19–related outcomes evaluated included lung injury [18,65], death [65], and reinfection [66]. Of the 3 studies, 2 adjusted for the confounding variables of age and sex. One study adjusted for the region of the healthcare facility, while another adjusted for confirmed history of COVID-19. One study did not make adjustments for any variables.

Discussion

We examined the methodologies used to estimate VE in comparative studies of different COVID-19 vaccines. In this study, we observed results consistent with those recommended by the WHO guidelines for assessing vaccine efficacy. However, findings that reflected real-world conditions were also evident. Notably, only a small proportion of the studies (15%) focused on evaluating rVE when comparing various COVID-19 vaccine types; the majority concentrated on VE. Most comparative studies assessed the effectiveness of mRNA vaccines versus non-RNA vaccines, with only 3 studies comparing protein-based and non–protein-based options. This underscores the importance of considering methodologies that include both rVE and aVE in VE studies and emphasizes the need to explore vaccine types beyond mRNA, such as protein-based and bivalent vaccines. The rVE estimates alone did not provide information on the aVE of each vaccine. Consequently, a methodological approach that accounts for both rVE and aVE is crucial when making public health decisions and formulating policies related to VE.

This review employed a methodical and systematic approach to deepen our understanding of the challenges and opportunities in this critical research area. Our findings are poised to guide future research and inform public health initiatives. We aimed to enrich the current landscape of methodologies for comparing the recently introduced COVID-19 vaccines, thereby supporting future empirical studies. Our results lay the groundwork for informed decision-making about vaccine types and the optimization of the global response to the ongoing pandemic.

Across all the study designs, most VE estimators relied on ORs or HRs. These studies did not employ extensions of the standard methodological models. The prevalence of the retrospective cohort design highlighted the need to explore alternative designs, such as the test-negative design commonly used in influenza and rotavirus VE studies [67,68]. Retrospective cohort studies are susceptible to recall and misclassification biases during data collection, and it is difficult to eliminate the influence of unmeasured latent confounding factors. Measuring VE involves assessing the simple association between ongoing vaccine administration and disease risk, as well as evaluating the strength of causal effects. However, retrospective cohort studies are limited in their capacity to elucidate causal relationships. Test-negative designs could be more feasible for clinicians to conduct than case-control studies. Nonetheless, challenges exist, including the fact that rapid diagnostic testing may not be widely available in some countries, the potential underestimation of VE due to the tests’ imperfect sensitivity and specificity, and the possibility of selection bias when enrolling participants in routine clinical settings [67].

The WHO has suggested that key covariates to be considered in COVID-19 VE studies include sociodemographic variables such as age, sex, a proxy for SES, pre-existing chronic conditions, the date of illness onset, the date of specimen collection, previous SARS-CoV-2 infection, and healthcare worker status [4]. In the reviewed studies, age and sex were adequately controlled. However, regarding other confounding factors, nearly half of the studies failed to consider comorbidities, more than one-third did not adjust for SES, and almost 90% neglected to account for the date of testing or prior SARS-CoV-2 infection. To mitigate outcome bias resulting from healthcare behaviors that are difficult to measure, several studies acknowledged limitations due to challenges in data collection, especially in adjusting for occupation and SES. As the pandemic has continued, the proportion of reinfections has approached half of all confirmed cases, with variations observed across countries. In terms of COVID-19 severity and mortality, outcome bias may arise from immunity gained through previous infections. Adjustment will be necessary to address this issue.

Despite the insights gleaned from the examined studies, the research has certain limitations. The emphasis on aVE over rVE, particularly when considering the diversity of COVID-19 vaccine types available, may have compromised the comprehensiveness of the findings. Furthermore, the disproportionate emphasis on comparing mRNA vaccines to their non-mRNA counterparts neglects the assessment of other vaccine categories, such as protein-based and bivalent vaccines. The methodological homogeneity, marked by the frequent use of ORs or HRs without considering more complex models, suggests a lack of innovation in statistical approaches [69,70]. The common reliance on a retrospective cohort design also prompts concerns about possible biases and the inherent limitations of this approach [71]. Furthermore, the lack of full adjustment for key confounding factors, including comorbidities, SES, and other essential variables recommended by the WHO, could have introduced bias and impacted validity [72]. Collectively, these limitations highlight the need for more diverse methodological approaches that include various vaccine types and robust adjustments for confounding factors, thereby improving the reliability and applicability of VE study findings across various settings.

The present review also has limitations. For instance, we did not assess VE across subgroups defined by vaccination status, number of doses received, vaccination regimen, booster status, or whether vaccination was homologous or heterologous. Furthermore, we did not incorporate meta-analyses of VE from other research or consider estimates of effectiveness that may vary over time. The periods characterized by the dominance of pre-Delta, Delta, and Omicron variants were not differentiated. Moreover, the effectiveness of other vaccine types, such as inactivated and viral vector vaccines, was not examined in this review.

Despite the identified limitations, a strength of this study is its comprehensive review and critical assessment of the methodologies used in COVID-19 VE research. The structured and systematic approach of this review improves our understanding of the challenges and opportunities in this field. By highlighting the prevailing methodological trends and emphasizing the need for more nuanced evaluation, this article serves as a valuable guide for future research and public health initiatives. Acknowledging the limitations demonstrates the study’s commitment to transparency and contributes to the ongoing discourse on refining VE research methodologies, ultimately advancing our understanding of VE in diverse settings.

Conclusion

This study highlights the lack of focus on the potential of VE studies to report on rVE and aVE, as well as the scarcity of comparative studies of vaccine types that do not use mRNA technology. To obtain valid and reliable results, it is also crucial that the core confounding factors in COVID-19 VE studies be considered. Furthermore, it may be beneficial to include assessments of effectiveness within specific subgroups and the efficacy against certain variants, which are areas currently underrepresented in the evidence. These considerations should be incorporated into more thorough evaluations in future clinical trials.

HIGHLIGHTS

• Most comparative studies on coronavirus disease 2019 vaccines have not presented relative estimates.

• Comparative studies on vaccine effectiveness have focused on mRNA and non-mRNA vaccines, overlooking protein-based and bivalent vaccines.

• Advanced methodologies should be incorporated into comparative vaccine studies.

Supplementary Material

Article information

Ethics Approval

Not applicable.

Conflicts of Interest

The authors have no conflicts of interest to declare.

Funding

This study was partially supported by a grant from the Korea Disease Control and Prevention Agency (No. Q2313421).

Availability of Data

All data generated or analyzed during this study are included in this published article. Other data may be requested through the corresponding author.

Authors’ Contributions

Conceptualization: YJC, SAC; Data curation: EB, SAC; Formal analysis: EB, JS, YSC, SAC; Funding acquisition: YJC; Investigation: SAC; Methodology: YJC, SAC; Project administration: YJC; Supervision: SAC; Validation: all authors; Visualization: EB, SAC; Writing–original draft: EB, SAC; Writing–review & editing: all authors. All authors read and approved the final manuscript.

Figure 1.

Flow diagram of studies identified through searches of the PubMed, Embase, Cochrane Library, and KoreaMed databases.

VE, vaccine effectiveness; COVID-19, coronavirus disease 2019; KAP, knowledge, attitude, practice; rVE, relative vaccine effectiveness; aVE, absolute vaccine effectiveness.

Figure 2.

Definitions of absolute vaccine effectiveness (VE) by study design in comparative studies of coronavirus disease 2019 vaccines.

HR, hazard ratio; IRR, incidence rate ratio; OR, odds ratio; RR, risk ratio.

Figure 3.

Risk-of-bias assessments for 25 relative vaccine effectiveness studies for coronavirus disease 2019, comparing mRNA vaccines to non-mRNA vaccines and bivalent to monovalent mRNA vaccines. These assessments were conducted using the Risk of Bias in Non-randomized Studies (ROBINS-I) tool and are visualized with the Robvis tool. Modified from McGuinness et al. Res Synth Methods 2021;12:55–61 [6].

Figure 4.

Risk-of-bias assessments for 126 vaccine effectiveness studies for coronavirus disease 2019, comparing mRNA to non-mRNA, bivalent to monovalent mRNA, and protein-based to non–protein-based vaccines. These assessments were conducted using the Risk of Bias in Non-randomized Studies (ROBINS-I) tool and are visualized with the Robvis tool Modified from McGuinness et al. Res Synth Methods 2021;12:55–61 [6].

References

• 1. World Health Organization (WHO) (CH). WHO coronavirus (COVID-19) dashboard October 23, 2023 [Internet]. WHO; 2023 [cited 2024 Jan 5]. Available from: https://covid19.who.int/.
• 2. Han X, Xu P, Ye Q. Analysis of COVID-19 vaccines: types, thoughts, and application. J Clin Lab Anal 2021;35:e23937.ArticlePubMedPMCPDF
• 3. Lewis NM, Chung JR, Uyeki TM, et al. Interpretation of relative efficacy and effectiveness for influenza vaccines. Clin Infect Dis 2022;75:170−5.ArticlePubMedPDF
• 4. World Health Organization (WHO) (CH). Evaluation of COVID-19 vaccine effectiveness: interim guidance: WHO team, immunization, vaccines and biologicals (IVB) 17 March 2021 [Internet]. WHO; 2021 [cited 2024 Jan 5]. Available from: https://www.who.int/publications/i/item/WHO-2019-nCoV-vaccine_effectiveness-measurement-2021.1.
• 5. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919. ArticlePubMedPMC
• 6. McGuinness LA, Higgins JP. Risk-of-bias VISualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments. Res Synth Methods 2021;12:55−61.ArticlePubMedPDF
• 7. Cegolon L, Negro C, Mastrangelo G, et al. Primary SARS-CoV-2 infections, re-infections and vaccine effectiveness during the omicron transmission period in healthcare workers of Trieste and Gorizia (Northeast Italy), 1 December 2021-31 May 2022. Viruses 2022;14:2688. ArticlePubMedPMC
• 8. Premikha M, Chiew CJ, Wei WE, et al. Comparative effectiveness of mRNA and inactivated whole-virus vaccines against coronavirus disease 2019 infection and severe disease in Singapore. Clin Infect Dis 2022;75:1442−5.ArticlePubMedPDF
• 9. Abdel-Qader DH, Abdel-Qader H, Silverthorne J, et al. Real-world effectiveness of four types of COVID-19 vaccines. Vaccines (Basel) 2023;11:985. ArticlePubMedPMC
• 10. Romero-Ibarguengoitia ME, Rivera-Salinas D, Sarti R, et al. Efficacy of six different SARS-CoV-2 vaccines during a six-month follow-up and five COVID-19 waves in Brazil and Mexico. Vaccines (Basel) 2023;11:842. ArticlePubMedPMC
• 11. Intawong K, Chariyalertsak S, Chalom K, et al. Effectiveness of heterologous third and fourth dose COVID-19 vaccine schedules for SARS-CoV-2 infection during delta and omicron predominance in Thailand: a test-negative, case-control study. Lancet Reg Health Southeast Asia 2023;10:100121. ArticlePubMed
• 12. Berec L, Smid M, Pribylova L, et al. Protection provided by vaccination, booster doses and previous infection against COVID-19 infection, hospitalisation or death over time in Czechia. PLoS One 2022;17:e0270801.ArticlePubMedPMC
• 13. Kokic Z, Kon P, Djurkovic-Djakovic O. Effectiveness of vaccination in preventing COVID-19: a community study comparing four vaccines. Vaccines (Basel) 2023;11:544. ArticlePubMedPMC
• 14. Culpan HC, Aydin SN, Uygur A, et al. Effectiveness of the BNT162b2 and the CoronaVac vaccines and boosters in healthcare workers. Hum Vaccin Immunother 2023;19:2275445. PubMedPMC
• 15. Bienkowski C, Skrzat-Klapaczynska A, Firlag-Burkacka E, et al. The clinical effectiveness and safety of vaccinations against COVID-19 in HIV-positive patients: data from observational study in Poland. Vaccines (Basel) 2023;11:514. ArticlePubMedPMC
• 16. Petrovic V, Vukovic V, Markovic M, et al. Early effectiveness of four SARS-CoV-2 vaccines in preventing COVID-19 among adults aged ≥60 years in Vojvodina, Serbia. Vaccines (Basel) 2022;10:389. ArticlePubMedPMC
• 17. Clifford S, Waight P, Hackman J, et al. Effectiveness of BNT162b2 and ChAdOx1 against SARS-CoV-2 household transmission: a prospective cohort study in England. Wellcome Open Res 2023;8:96. ArticlePubMedPMCPDF
• 18. Matveeva O, Ershov A. Retrospective cohort study of the effectiveness of the Sputnik V and EpiVacCorona vaccines against the SARS-CoV-2 Delta variant in Moscow (June-July 2021). Vaccines (Basel) 2022;10:984. ArticlePubMedPMC
• 19. Al-Momani H, Aldajah K, Alda'ajah E, et al. Effectiveness of Pfizer/BioNTech and Sinopharm COVID-19 vaccines in reducing hospital admissions in prince Hamza hospital, Jordan. Front Public Health 2022;10:1008521. ArticlePubMedPMC
• 20. Brazete C, Brazete J, Alves F, et al. COVID-19 vaccines effectiveness against symptomatic disease and severe outcomes, 2021-2022: a test-negative case-control study. Public Health 2023;218:84−91.ArticlePubMed
• 21. Saadh MJ, Jaber SA. Efficacy of COVID-19 vaccines. Microb Pathog 2022;171:105729. ArticlePubMedPMC
• 22. Hall V, Foulkes S, Insalata F, et al. Protection against SARS-CoV-2 after COVID-19 vaccination and previous infection. N Engl J Med 2022;386:1207−20.PubMed
• 23. Sritipsukho P, Khawcharoenporn T, Siribumrungwong B, et al. Real-life effectiveness of COVID-19 vaccine during the Omicron variant-dominant pandemic: how many booster doses do we need? Emerg Microbes Infect 2023;12:2174779. ArticlePubMedPMC
• 24. Rane MS, Robertson MM, Kulkarni SG, et al. Effectiveness of COVID-19 vaccines against symptomatic and asymptomatic SARS-CoV-2 infections in an urgent care setting. Vaccine 2023;41:989−98.ArticlePubMed
• 25. Reynolds MW, Secora A, Joules A, et al. Evaluating real-world COVID-19 vaccine effectiveness using a test-negative case-control design. J Comp Eff Res 2022;11:1161−72.ArticlePubMed
• 26. Ng RW, Sze RK, Chong KC, et al. Effectiveness of mRNA and inactivated COVID-19 vaccines: a test-negative study in an infection-naive Hong Kong population. J Infect 2023;87:136−43.PubMed
• 27. Robles-Fontan MM, Nieves EG, Cardona-Gerena I, et al. Effectiveness estimates of three COVID-19 vaccines based on observational data from Puerto Rico. Lancet Reg Health Am 2022;9:100212. ArticlePubMedPMC
• 28. Self WH, Tenforde MW, Rhoads JP, et al. Comparative effectiveness of Moderna, Pfizer-BioNTech, and Janssen (Johnson & Johnson) vaccines in preventing COVID-19 hospitalizations among adults without immunocompromising conditions: United States, March-August 2021. MMWR Morb Mortal Wkly Rep 2021;70:1337−43.PubMedPMC
• 29. Huang C, Wei Y, Yan VK, et al. Vaccine effectiveness of BNT162b2 and CoronaVac against SARS-CoV-2 Omicron infection and related hospital admission among people with substance use disorder in Hong Kong: a matched case-control study. Lancet Psychiatry 2023;10:403−13.ArticlePubMedPMC
• 30. Skowronski DM, Setayeshgar S, Zou M, et al. Comparative single-dose mRNA and ChAdOx1 vaccine effectiveness against severe acute respiratory syndrome coronavirus 2, including variants of concern: test-negative design, British Columbia, Canada. J Infect Dis 2022;226:485−96.ArticlePubMedPMCPDF
• 31. Perry M, Gravenor MB, Cottrell S, et al. COVID-19 vaccine uptake and effectiveness in adults aged 50 years and older in Wales UK: a 1.2m population data-linkage cohort approach. Hum Vaccin Immunother 2022;18:2031774. ArticlePubMedPMC
• 32. Alali WQ, Ali LA, AlSeaidan M, et al. Effectiveness of BNT162b2 and ChAdOx1 vaccines against symptomatic COVID-19 among healthcare workers in Kuwait: a retrospective cohort study. Healthcare (Basel) 2021;9:1692. ArticlePubMedPMC
• 33. Horne EM, Hulme WJ, Keogh RH, et al. Waning effectiveness of BNT162b2 and ChAdOx1 covid-19 vaccines over six months since second dose: OpenSAFELY cohort study using linked electronic health records. BMJ 2022;378:e071249.ArticlePubMedPMC
• 34. Islam N, Griffin DO, Jarvis MS, et al. Comparative effectiveness of the sars-CoV-2 vaccines during delta dominance. Heliyon 2023;9:e16006.ArticlePubMedPMC
• 35. Kaura A, Trickey A, Shah AS, et al. Comparing the longer-term effectiveness of a single dose of the Pfizer-BioNTech and Oxford-AstraZeneca COVID-19 vaccines across the age spectrum. EClinicalMedicine 2022;46:101344. ArticlePubMedPMC
• 36. Kompaniyets L, Wiegand RE, Oyalowo AC, et al. Relative effectiveness of coronavirus disease 2019 vaccination and booster dose combinations among 18.9 million vaccinated adults during the early severe acute respiratory syndrome coronavirus 2 Omicron period: United States, 1 January 2022 to 31 March 2022. Clin Infect Dis 2023;76:1753−60.ArticlePubMedPDF
• 37. Liu B, Gidding H, Stepien S, et al. Relative effectiveness of COVID-19 vaccination with 3 compared to 2 doses against SARS-CoV-2 B.1.1.529 (Omicron) among an Australian population with low prior rates of SARS-CoV-2 infection. Vaccine 2022;40:6288−94.ArticlePubMedPMC
• 38. Martinez-Baz I, Trobajo-Sanmartin C, Miqueleiz A, et al. Product-specific COVID-19 vaccine effectiveness against secondary infection in close contacts, Navarre, Spain, April to August 2021. Euro Surveill 2021;26:2100894. PubMedPMC
• 39. Hung Nguyen V, Boileau C, Bogdanov A, et al. Relative effectiveness of BNT162b2, mRNA-1273, and Ad26. COV2.S vaccines and homologous boosting in preventing COVID-19 in adults in the US. Open Forum Infect Dis 2023;10:ofad288.
• 40. Risk M, Shen C, Hayek SS, et al. Comparative effectiveness of coronavirus disease 2019 (COVID-19) vaccines against the Delta variant. Clin Infect Dis 2022;75:e623−9.ArticlePubMedPMCPDF
• 41. Robalo Q, De Mot L, Vandromme M, et al. Association between COVID-19 primary vaccination and severe disease caused by SARS-CoV-2 Delta variant among hospitalized patients: a Belgian retrospective cohort study. Vaccines (Basel) 2022;11:14. ArticlePubMedPMC
• 42. Wei J, Zhang W, Doherty M, et al. Comparative effectiveness of BNT162b2 and ChAdOx1 nCoV-19 vaccines against COVID-19. BMC Med 2023;21:78. ArticlePubMedPMCPDF
• 43. Wright BJ, Tideman S, Diaz GA, et al. Comparative vaccine effectiveness against severe COVID-19 over time in US hospital administrative data: a case-control study. Lancet Respir Med 2022;10:557−65.ArticlePubMedPMC
• 44. Xie J, Feng S, Li X, et al. Comparative effectiveness of the BNT162b2 and ChAdOx1 vaccines against COVID-19 in people over 50. Nat Commun 2022;13:1519. ArticlePubMedPMCPDF
• 45. Yamal JM, Appana S, Wang M, et al. Trends and correlates of breakthrough infections with SARS-CoV-2. Front Public Health 2022;10:856532. ArticlePubMedPMC
• 46. Kim HK, Park SK, Choe SA, et al. Risk of SARS-CoV-2 breakthrough infection following NVX-CoV2373 and BNT162b2 vaccinations in Korean adults: a population-based observational study. Vaccine 2024;42:1440−4.ArticlePubMed
• 47. Gim H, Seo H, Chun BC. Vaccine effectiveness against severe acute respiratory syndrome coronavirus 2 reinfection by type and frequency of vaccine: a community-based case-control study. J Korean Med Sci 2024;39:e174.ArticlePubMedPMCPDF
• 48. Kerr S, Bedston S, Bradley DT, et al. Waning of first- and second-dose ChAdOx1 and BNT162b2 COVID-19 vaccinations: a pooled target trial study of 12.9 million individuals in England, Northern Ireland, Scotland and Wales. Int J Epidemiol 2023;52:22−31.ArticlePubMedPDF
• 49. Arashiro T, Arima Y, Muraoka H, et al. Coronavirus disease 19 (COVID-19) vaccine effectiveness against symptomatic severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection during Delta-dominant and Omicron-dominant periods in Japan: a multicenter prospective case-control study (factors associated with SARS-CoV-2 infection and the effectiveness of COVID-19 vaccines study). Clin Infect Dis 2023;76:e108−15.ArticlePubMedPDF
• 50. Ackerson BK, Bruxvoort KJ, Qian L, et al. Effectiveness and durability of mRNA-1273 BA.4/BA.5 bivalent vaccine (mRNA-1273.222) against SARS-CoV-2 BA.4/BA.5 and XBB sublineages. Hum Vaccin Immunother 2024;20:2335052. ArticlePubMedPMC
• 51. Huiberts AJ, de Gier B, Hoeve CE, et al. Effectiveness of bivalent mRNA booster vaccination against SARS-CoV-2 Omicron infection, the Netherlands, September to December 2022. Euro Surveill 2023;28:2300087. ArticlePubMedPMC
• 52. Chatzilena A, Hyams C, Challen R, et al. Relative vaccine effectiveness of mRNA COVID-19 boosters in people aged at least 75 years during the spring-summer (monovalent vaccine) and autumn-winter (bivalent vaccine) booster campaigns: a prospective test negative case-control study, United Kingdom, 2022. Euro Surveill 2023;28:2300173. ArticlePubMedPMC
• 53. Mateo-Urdiales A, Sacco C, Fotakis EA, et al. Relative effectiveness of monovalent and bivalent mRNA boosters in preventing severe COVID-19 due to omicron BA.5 infection up to 4 months post-administration in people aged 60 years or older in Italy: a retrospective matched cohort study. Lancet Infect Dis 2023;23:1349−59.ArticlePubMed
• 54. Torres R, Toro L, Sanhueza ME, et al. Clinical efficacy of SARS-CoV-2 vaccination in hemodialysis patients. Kidney Int Rep 2022;7:2176−85.PubMedPMC
• 55. Chen PC, Huang CC, Fu CM, et al. Real-world effectiveness of SARS-CoV-2 vaccine booster in hemodialysis patients with COVID-19 receiving Molnupiravir. Viruses 2023;15:543. ArticlePubMedPMC
• 56. Manley HJ, Li NC, Aweh GN, et al. SARS-CoV-2 vaccine effectiveness and breakthrough infections among patients receiving maintenance dialysis. Am J Kidney Dis 2023;81:406−15.ArticlePubMed
• 57. Cheng FW, Fan M, Wong CK, et al. The effectiveness and safety of mRNA (BNT162b2) and inactivated (CoronaVac) COVID-19 vaccines among individuals with chronic kidney diseases. Kidney Int 2022;102:922−5.ArticlePubMedPMC
• 58. OpenSAFELY Collaborative; Parker EP, Horne EM, et al. Comparative effectiveness of two- and three-dose COVID-19 vaccination schedules involving AZD1222 and BNT162b2 in people with kidney disease: a linked OpenSAFELY and UK Renal Registry cohort study. Lancet Reg Health Eur 2023;30:100636. ArticlePubMedPMC
• 59. Callaghan CJ, Curtis RM, Mumford L, et al. Vaccine effectiveness against the SARS-CoV-2 B.1.1.529 Omicron variant in solid organ and islet transplant recipients in England: a national retrospective cohort study. Transplantation 2023;107:1124−35.ArticlePubMedPMC
• 60. Pinto-Alvarez M, Fernandez-Nino JA, Arregoces-Castillo L, et al. Real-world evidence of COVID-19 vaccines effectiveness in solid-organ transplant recipient population in Colombia: a study nested in the Esperanza Cohort. Transplantation 2023;107:216−24.ArticlePubMed
• 61. Shen C, Risk M, Schiopu E, et al. Efficacy of COVID-19 vaccines in patients taking immunosuppressants. Ann Rheum Dis 2022;81:875−80.ArticlePubMed
• 62. John BV, Sidney Barritt A 4th, Moon A, et al. Effectiveness of COVID-19 viral vector Ad.26.COV2.S vaccine and comparison with mRNA vaccines in Cirrhosis. Clin Gastroenterol Hepatol 2022;20:2405−8.ArticlePubMedPMC
• 63. Eick-Cost AA, Ying S, Wells N. Effectiveness of mRNA-1273, BNT162b2, and JNJ-78436735 COVID-19 vaccines among US military personnel before and during the predominance of the Delta variant. JAMA Netw Open 2022;5:e228071.ArticlePubMedPMC
• 64. Britton A, Fleming-Dutra KE, Shang N, et al. Association of COVID-19 vaccination with symptomatic SARS-CoV-2 infection by time since vaccination and Delta variant predominance. JAMA 2022;327:1032−41.ArticlePubMedPMC
• 65. Barchuk A, Bulina A, Cherkashin M, et al. Gam-COVID-Vac, EpiVacCorona, and CoviVac effectiveness against lung injury during Delta and Omicron variant surges in St. Petersburg, Russia: a test-negative case-control study. Respir Res 2022;23:276. ArticlePubMedPMCPDF
• 66. Barchuk A, Bulina A, Cherkashin M, et al. COVID-19 vaccines effectiveness against symptomatic SARS-CoV-2 during Delta variant surge: a preliminary assessment from a case-control study in St. Petersburg, Russia. BMC Public Health 2022;22:1803. ArticlePubMedPMCPDF
• 67. Fukushima W, Hirota Y. Basic principles of test-negative design in evaluating influenza vaccine effectiveness. Vaccine 2017;35:4796−800.ArticlePubMed
• 68. Sullivan SG, Feng S, Cowling BJ. Potential of the test-negative design for measuring influenza vaccine effectiveness: a systematic review. Expert Rev Vaccines 2014;13:1571−91.ArticlePubMedPMC
• 69. Chen D, Petersen ML, Rytgaard HC, et al. Beyond the Cox hazard ratio: a targeted learning approach to survival analysis in a cardiovascular outcome trial application. Stat Biopharm Res 2023;15:524−39.Article
• 70. Chatterjee A, Woodruff H, Wu G, et al. Limitations of only reporting the odds ratio in the age of precision medicine: a deterministic simulation study. Front Med (Lausanne) 2021;8:640854. ArticlePubMedPMC
• 71. Ramirez-Santana M. Limitations and biases in cohort studies. In: Mauricio Barría R. In: Cohort Studies in Health Sciences [Internet] InTech; 2018 https://doi.org/10.5772/intechopen.74324.
• 72. European Centre for Disease Prevention and Control (ECDC) (SWE). Core protocol for ECDC studies of COVID19 vaccine effectiveness against hospitalisation with Severe Acute Respiratory Infection laboratory-confirmed with SARS-CoV-2, version 1.0 [Internet]. ECDC; 2021 [cited 2023 Jan 30]. Available from: https://www.ecdc.europa.eu/sites/default/files/documents/Core-protocol-for-ECDC-studies-of-COVID-19-vaccine-effectiveness-against-hospitalisation-with-SARI.pdf.

Figure & Data

References

Citations

Citations to this article as recorded by