Materials and Methods
Eligibility Criteria
The search and review processes adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Only studies published in English were included. To encompass research on the relative effectiveness of COVID-19 vaccines that did not provide relative estimates, we also incorporated studies on VE.
Information Sources
On June 13, 2024, we systematically searched several databases, including PubMed, Embase, and the Cochrane Library.
Search Strategy
The search terms used were:
1. “COVID” OR “COVID-19” OR “Coronavirus” OR “SARS-COV-2”
2. “vaccine” OR “vaccination”
3. “relative effectiveness” OR “relative efficacy”
4. #2 AND #3
5. “relative vaccine effectiveness” OR “relative vaccine efficacy” OR “relative VE”
6. “vaccine effectiveness” OR “VE”
7. #4 OR #5 OR #6
8. #1 AND #7
Selection Process
In accordance with the search strategy, articles were initially screened to identify and remove duplicates. Three researchers (EB, JS, and SAC) independently screened the titles, abstracts, and full texts of the remaining studies. E.B. then verified the review results for consistency and accuracy. The inclusion criteria restricted the studies to interventional or observational research that provided COVID-19 VE estimates for 2 or more vaccine platforms. These studies were required to report outcomes of COVID-19 infection, symptomatic disease, severe disease, COVID-19–related hospitalization, intensive care unit (ICU) admission, and/or death. Given the scarcity of comparative studies on protein versus non–protein and monovalent versus bivalent vaccines, our review was focused on studies comparing mRNA and non-mRNA vaccines. We excluded studies evaluating the VE of a single COVID-19 vaccine type; those unrelated to COVID-19 VE; those concentrating on laboratory measurements of immunogenicity, determinants of vaccine uptake, or cost-effectiveness; simulation studies; and comparisons of vaccination schedules and doses. Additionally, we excluded preprints, systematic reviews, ongoing studies, Delphi studies, modeling studies, and articles without full-text availability.
Data Items
We extracted data on the publication year, country, study design, sample size, and composition of study participants, as well as the definition of VE, outcomes, and covariates. In the assessment of rVE studies, we collected information on the availability of aVE estimates and the criteria used to define aVE. Vaccine types were classified into 3 primary categories: mRNA versus non-mRNA, protein-based versus non–protein-based, and monovalent versus bivalent COVID-19 vaccines. Our analysis evaluated the effectiveness of COVID-19 vaccines in preventing infection, symptomatic disease, severe disease, hospitalization, ICU admission, and mortality, with results expressed in either relative or absolute terms. Additionally, 3 reviewers conducted the bias assessment. In accordance with the World Health Organization (WHO) interim guidance on evaluating COVID-19 VE [
4], we assessed the extent to which studies adjusted for core covariates in 4 categories: time (such as date of illness onset or date of specimen collection), susceptibility to infection (namely previous COVID-19 infection), sociodemographic factors, and chronic medical conditions.
Assessment of Study Risk of Bias
The Risk of Bias in Non-randomized Studies (ROBINS-I) tool [
5] was employed to assess the risk of bias, which was categorized as “low,” “moderate,” “serious,” “critical,” or “no information” across 7 domains for observational studies.
This research has been registered with PROSPERO (
https://www.crd.york.ac.uk/prospero/), an international prospective register of systematic reviews, under the registration number CRD42023442560.
Results
Study Selection
The initial review of PubMed, Embase, and the Cochrane Library yielded a total of 2,921 articles. After 1,054 duplicates were removed and the inclusion criteria were applied, 151 articles remained for analysis (
Figure 1).
Study Characteristics
rVE was estimated to compare mRNA vaccines (Pfizer, also known as BNT162b2 and Comirnaty, and Moderna, also known as mRNA-1273 and Spikevax) with non-mRNA vaccines (AstraZeneca, Janssen, Novavax, Sinovac-Coronavac, Sinopharm, Sputnik V, and Bharat-Biotech Covaxin [BBV152]). This comparison included 20 studies (13.2%), with 5 studies examining monovalent and bivalent mRNA COVID-19 vaccines (3.3%). In contrast, VE was estimated for mRNA versus non-mRNA vaccines in 118 studies (78.1%), monovalent versus bivalent mRNA vaccines in 5 studies (3.3%), and protein-based versus non–protein-based COVID-19 vaccines in 3 studies (2.0%). Most of these studies employed a retrospective cohort design (
n=70, 46.4%), followed by test-negative (
n=39, 25.8%), prospective cohort (
n=18, 11.9%), case-control (
n=12, 7.9%), and cross-sectional (
n=3, 2.0%) designs. The most commonly used definitions of rVE were hazard ratio (HR;
n=7, 35.0%), the formula (1−HR)×100 (
n=6, 30%), and odds ratio (OR;
n=4, 20.0%). For aVE, the definitions used were the formulas (1−OR)×100 (
n=54, 42.9%), (1−HR)×100 (
n=34, 27.0%), and (1−risk ratio [RR])×100 (
n=10, 7.9%) (
Figure 2).
Risk of Bias in Studies
We evaluated 25 studies assessing rVE for COVID-19 vaccines and 126 studies assessing only VE, using the ROBINS-I tool for quality assessment. Of the former, 20 studies were considered to have a low risk of bias. Two studies were categorized as having “no information” regarding risks, 1 study was identified as having a critical risk of bias, 1 presented a serious risk, and the remaining study displayed a moderate risk (
Figure 3) [
6]. For the study presenting critical risk, the potential bias stemmed from the lack of adjustment for confounding variables [
7]. Outcome measurement was not specified in the databases of 2 studies [
8,
9], and 1 study did not mention the database source of exposure information [
8]. Regarding the 2 studies identified as having serious or moderate selection bias risk, the cause was the limited sample size in 1 study [
7] and the selection of volunteer participants in the other [
10].
In studies assessing VE alone, the overall risk of bias was categorized as moderate in 51 studies (40.5%), low in 46 (36.5%), and indeterminate due to lack of information in 14 (11.1%). Additionally, 7 studies (5.6%) were classified as having a serious risk of bias, while 8 studies (6.3%) were deemed to be at critical risk (
Figure 4) [
6]. Regarding the confounding bias domain, 24 studies (19.0%) were found to have a moderate risk of bias, 4 (3.2%) [
11–
14] had serious risk, and 4 (3.2%) [
15–
18] had a critical risk. In terms of deviations from planned interventions and reporting of results, all studies were considered to have either a low or a moderate risk of bias. Participant selection bias was rated as serious in 3 studies [
19–
21] and as critical in 2 studies [
22,
23]. The risk of bias in intervention classification was moderate in 11 studies and critical in 3 studies [
15,
24,
25]. Information on missing data was not provided in 39 (37.9%) studies, and 6 studies (5.8%) [
26–
31] were assessed as having a moderate risk in this domain. Outcome measurement information was absent in 1 study [
32], and a moderate risk of bias in outcome measurement was identified in 11 studies (10.7%).
Methodology of rVE Studies
Overall, 59,915,206 participants were included in 20 comprehensive studies that investigated the rVE of various mRNA and non-mRNA COVID-19 vaccines [
7–
10,
33–
48]. The studies were published between 2021 and 2024, with the research conducted from 2020 to 2022 (
Table 1) [
7–
10,
33–
48]. The largest proportion of studies took place in the United States (
n=6, 30.0%) [
34,
36,
39,
40,
43,
45], followed by the United Kingdom (
n=5, 25.0%) [
33,
35,
42,
44]. Most studies targeted the general population (
n=15, 75.0%), with some focusing on hospitalized patients (
n=4, 20.0%) [
10,
39,
41,
43] and healthcare workers (
n=1, 5.0%) [
7]. Regarding study design, the evaluation of rVE among COVID-19 vaccine types utilized various methodologies, including retrospective cohort (
n=11; 55.0%) [
7,
8,
34–
36,
39–
41,
45,
46,
48], prospective cohort (
n=6; 30.0%) [
9,
10,
33,
37,
38,
44], and case-control (
n=3; 15.0%) designs [
42,
43,
47]. The age distribution of participants varied, with 12 studies including individuals aged ≥18 years; 1 study each encompassing participants aged ≥12, ≥16, >20, >40, and >50 years; and 3 studies (15.0%) that did not provide age specifications.
The comparative analyses in these studies were primarily focused on the Pfizer and Moderna vaccines (
n=9, 45.0%) [
8,
10,
34,
38–
41,
45,
47], with other studies examining only the Pfizer vaccine (
n=9, 45.0%) [
7,
9,
33,
35,
37,
42,
44,
46,
48], only the Moderna vaccine (
n=1, 5.0%) [
43], or an unspecified mRNA vaccine (
n=1, 5.9%) [
36]. The comparator vaccines evaluated against mRNA vaccines included Janssen (
n=10, 58.8%), AstraZeneca (
n=9, 52.9%), and Sinovac-Coronavac (
n=2, 11.8%). The most frequently reported outcomes across the studies were COVID-19–related infection (
n=14, 70.0%), hospitalization (
n=12, 60.0%), and mortality (
n=6, 30.0%). The most common definitions of rVE were HR (
n=7, 35.0%) [
7,
10,
33,
34,
40,
42,
44], followed by the formula (1−HR)×100 (
n=4, 20.0%) [
36,
37,
39,
45], OR (
n=4, 20.0%) [
9,
41,
43,
47], incidence rate ratio (
n=2, 10.0%) [
8,
35], the formula (1−RR)×100 (
n=2, 10.0%) [
38,
48], and RR (
n=1, 5.0%) [
46]. Only 8 studies (40.0%) reported and defined both aVE and rVE [
9,
33,
35,
38,
43,
44,
47,
48].
Among the studies, the variables most commonly controlled for were age (
n=16, 94,1%), sex (
n=15, 88.2%), and comorbidities (
n=14, 82.4%). Less frequently controlled variables included ethnicity (
n=7, 41.2%), socioeconomic status (SES) (
n=6, 35.3%), and body mass index (BMI;
n=5, 29.4%), as well as geographic region and previous healthcare utilization (both
n=4, 23.5%); calendar time, time of vaccination, and smoking status (each
n=3, 17.6%); and care home residential status, influenza vaccine history, number of prior tests for COVID-19, COVID-19 infection history, immunocompromised status, and antibody levels (each
n=2, 11.8%). Additional variables included learning disability, serious mental illness, current pregnancy, eligibility for first vaccine dose, clinical characteristics, electronic frailty index score, and alcohol consumption. Of the studies, 7 (35.0%) adjusted for the 3 WHO-recommended core sociodemographic covariates of age, sex, and SES; 15 (75.0%) adjusted for chronic medical conditions; 6 (30.0%) for time-related variables (date of illness, date of testing); and 3 (15.0%) for susceptibility to infection. Only 1 study accounted for all of the WHO-recommended core confounding variables, including age, sex, SES, chronic medical conditions, time-related variables, and susceptibility to infection [
34]. One study did not make adjustments for any variables [
7].
Only 5 studies investigated the rVE of bivalent versus monovalent mRNA vaccines (
Table 2) [
49–
53]. These studies, conducted between 2022 and 2024 in the Netherlands, Japan, Italy, the United Kingdom, and the United States, encompassed a total of 2,253,040 individuals. The rVE assessment predominantly employed a test-negative design, using the definition (1−OR)×100. Among these 5 studies, 4 (80.0%) adjusted for the WHO-recommended core sociodemographic variables of age, sex, and SES along with chronic medical conditions, and 3 studies (60.0%) considered the core variables related to time and susceptibility to infection. Two of these studies adjusted for all WHO core confounding variables [
49,
50].
Methodology of aVE Studies
Across the 118 studies of VE for mRNA versus non-mRNA vaccine types, the cumulative sample size was 203,805,491 participants (
Table S1). The greatest proportion of these studies took place in 2021 (
n=50, 42.4%), followed by 2022 (
n=28, 23.7%) and 2020–2021 (
n=26, 22.0%). The United Kingdom was the most common study location (
n=25, 21.2%), with the United States (
n=17, 14.4%), Hong Kong (
n=7, 5.9%), China and Spain (each
n=6, 5.1%), Italy (
n=5, 4.2%), and several other countries, including Canada, Brazil, Colombia, and France (each
n=4, 3.4%), also contributing. The studies predominantly included participants from the general population (
n=77, 65.3%). Hospitalized patients were the next largest group (
n=24, 20.3%), followed by hospital workers (
n=7, 5.9%), urgent care patients and long-term care residents (each
n=2, 1.7%), patients in hospital wards and public sector healthcare users (each
n=1, 0.8%), children (
n=3, 2.5%), and military medical academy personnel (
n=1, 0.8%). Among hospitalized patients, specific subgroups were identified, including patients undergoing hemodialysis (
n=3, 2.5%) [
54–
56], individuals with kidney disease (
n=2, 1.7%) [
57,
58], solid organ transplant recipients (
n=2, 1.7%) [
59,
60], patients using immunosuppressants (
n=1, 0.8%) [
61], and patients with cirrhosis (
n=1, 0.8%) [
62]. The studies involved participants with distinct attributes, such as US military personnel [
63], attendees of a human immunodeficiency virus outpatient clinic [
15], public emergency department patients [
20], and those who underwent pharmacy-based testing for COVID-19 [
64]. Concerning age, 68 studies (57.6%) involved adults aged 16 years and older, 18 studies (15.3%) included adults aged 50 years and older, and 9 studies (7.6%) did not specify participant ages. Seven studies (5.9%) encompassed participants of all ages. Three studies (2.5%) focused on children aged 3 to 17, 3 to 18, and 5 to 11 years, respectively. The remaining studies included age groups of ≥5, ≥10, ≥12, and ≥15 years.
The primary comparator among the mRNA vaccine groups was most frequently Moderna and Pfizer (n=60, 50.8%), followed by Pfizer alone (n=56, 47.5%), Moderna alone (n=1, 0.8%), and unspecified mRNA vaccine (n=2, 1.7%). Other comparator groups included AstraZeneca (n=75, 63.6%), Janssen (n=36, 30.5%), Sinovac-CoronaVac (n=32, 27.1%), Sputnik V (n=10, 8.5%), Sinopharm (n=9, 7.6%), a combination of Sinovac-CoronaVac and Sinopharm (n=3, 2.5%), Cansino (n=3, 2.5%), Bharat-Biotech BBV152 (n=1, 0.8%), and others (n=3, 2.5%). The studies primarily assessed the outcome of COVID-19 infection (n=76, 64.4%), followed by hospitalization (n=56, 47.5%), death (n=48, 40.7%), symptomatic disease (n=18, 15.3%), severe disease (n=18, 15.3%), ICU admission (n=4, 3.4%), all-cause mortality (n=4, 3.4%), reinfection (n=2, 1.7%) and breakthrough infection (n=1, 0.8%). The most common VE definitions were (1−OR)×100 (n=50, 42.4%), (1−HR)×100 (n=34, 28.8%), and (1−RR)×100 (n=10, 8.5%).
Age was the most frequently controlled variable across the studies (n=99, 83.9%), followed by sex (n=89, 75.4%), comorbidity (n=58, 49.2%), residential status (n=47, 40.0%), calendar time (n=32, 27.1%), SES (n=28, 23.7%), ethnicity (n=24, 20.3%), COVID-19 testing date (n=12, 10.2%), care home residential status (n=12, 10.2%), number of prior COVID-19 tests (n=15, 12.7%), previous healthcare utilization (n=16,13.6%), history of COVID-19 infection (n=10, 8.5%), clinical characteristics (n=8, 6.8%), BMI (n=7, 5.9%), influenza vaccine history (n=6, 5.1%), vaccination status (n=11, 9.3%), time of vaccination (n=8, 6.8%), and immunocompromised status (n=2, 1.7%). Additional confounding variables considered in these studies included incidence rate, the interval between COVID-19 onset and treatment initiation, cumulative person-days, time on dialysis, test type (antigen or genomic), local virus circulation, contact setting (household or other), travel history, and non-comorbid conditions. Of the 118 studies, 22 (18.6%) adjusted for the WHO-recommended core sociodemographic variables, including age, sex, and SES; 58 (49.2%) accounted for chronic medical conditions; 36 (30.5%) considered time-related factors; and 12 (10.2%) adjusted for susceptibility to infection. Nine studies did not incorporate adjustments for any variables.
Five studies compared the VE of bivalent versus monovalent mRNA vaccines (
Table S2). These studies were conducted between 2023 and 2024 in Denmark, Finland, Norway, Sweden, the Republic of Korea, Canada, and the United States, including a total of 6,163,590 adults and 90,905 children. The VE assessments primarily utilized a prospective cohort design, employing 5 distinct definitions: 1−RR, 1−HR, (1−OR)×100, HR, and 1−OR. Of the 5 studies, 1 (20.0%) adjusted for the 3 WHO-recommended core sociodemographic variables of age, sex, and SES. Three studies (60%) adjusted for chronic medical conditions, 2 (40%) considered core time-related covariates, and 1 (20%) accounted for susceptibility to infection.
Three studies assessed the VE of COVID-19 vaccines, differentiating between protein-based and non–protein-based vaccines. These studies were conducted in Russia, with a combined population of 327,997 individuals. Participants in the studies were adults aged 18 years and older. The methodologies employed were test-negative [
65], retrospective cohort [
18], and case-control [
66] designs. To calculate effectiveness, all studies utilized the formula (1−OR)×100. In all 3 studies, a protein-based vaccine (EpiVacCorona) was compared with the non–protein-based vaccines Gam-COVID-Vac and CoviVac. The COVID-19–related outcomes evaluated included lung injury [
18,
65], death [
65], and reinfection [
66]. Of the 3 studies, 2 adjusted for the confounding variables of age and sex. One study adjusted for the region of the healthcare facility, while another adjusted for confirmed history of COVID-19. One study did not make adjustments for any variables.
Discussion
We examined the methodologies used to estimate VE in comparative studies of different COVID-19 vaccines. In this study, we observed results consistent with those recommended by the WHO guidelines for assessing vaccine efficacy. However, findings that reflected real-world conditions were also evident. Notably, only a small proportion of the studies (15%) focused on evaluating rVE when comparing various COVID-19 vaccine types; the majority concentrated on VE. Most comparative studies assessed the effectiveness of mRNA vaccines versus non-RNA vaccines, with only 3 studies comparing protein-based and non–protein-based options. This underscores the importance of considering methodologies that include both rVE and aVE in VE studies and emphasizes the need to explore vaccine types beyond mRNA, such as protein-based and bivalent vaccines. The rVE estimates alone did not provide information on the aVE of each vaccine. Consequently, a methodological approach that accounts for both rVE and aVE is crucial when making public health decisions and formulating policies related to VE.
This review employed a methodical and systematic approach to deepen our understanding of the challenges and opportunities in this critical research area. Our findings are poised to guide future research and inform public health initiatives. We aimed to enrich the current landscape of methodologies for comparing the recently introduced COVID-19 vaccines, thereby supporting future empirical studies. Our results lay the groundwork for informed decision-making about vaccine types and the optimization of the global response to the ongoing pandemic.
Across all the study designs, most VE estimators relied on ORs or HRs. These studies did not employ extensions of the standard methodological models. The prevalence of the retrospective cohort design highlighted the need to explore alternative designs, such as the test-negative design commonly used in influenza and rotavirus VE studies [
67,
68]. Retrospective cohort studies are susceptible to recall and misclassification biases during data collection, and it is difficult to eliminate the influence of unmeasured latent confounding factors. Measuring VE involves assessing the simple association between ongoing vaccine administration and disease risk, as well as evaluating the strength of causal effects. However, retrospective cohort studies are limited in their capacity to elucidate causal relationships. Test-negative designs could be more feasible for clinicians to conduct than case-control studies. Nonetheless, challenges exist, including the fact that rapid diagnostic testing may not be widely available in some countries, the potential underestimation of VE due to the tests’ imperfect sensitivity and specificity, and the possibility of selection bias when enrolling participants in routine clinical settings [
67].
The WHO has suggested that key covariates to be considered in COVID-19 VE studies include sociodemographic variables such as age, sex, a proxy for SES, pre-existing chronic conditions, the date of illness onset, the date of specimen collection, previous SARS-CoV-2 infection, and healthcare worker status [
4]. In the reviewed studies, age and sex were adequately controlled. However, regarding other confounding factors, nearly half of the studies failed to consider comorbidities, more than one-third did not adjust for SES, and almost 90% neglected to account for the date of testing or prior SARS-CoV-2 infection. To mitigate outcome bias resulting from healthcare behaviors that are difficult to measure, several studies acknowledged limitations due to challenges in data collection, especially in adjusting for occupation and SES. As the pandemic has continued, the proportion of reinfections has approached half of all confirmed cases, with variations observed across countries. In terms of COVID-19 severity and mortality, outcome bias may arise from immunity gained through previous infections. Adjustment will be necessary to address this issue.
Despite the insights gleaned from the examined studies, the research has certain limitations. The emphasis on aVE over rVE, particularly when considering the diversity of COVID-19 vaccine types available, may have compromised the comprehensiveness of the findings. Furthermore, the disproportionate emphasis on comparing mRNA vaccines to their non-mRNA counterparts neglects the assessment of other vaccine categories, such as protein-based and bivalent vaccines. The methodological homogeneity, marked by the frequent use of ORs or HRs without considering more complex models, suggests a lack of innovation in statistical approaches [
69,
70]. The common reliance on a retrospective cohort design also prompts concerns about possible biases and the inherent limitations of this approach [
71]. Furthermore, the lack of full adjustment for key confounding factors, including comorbidities, SES, and other essential variables recommended by the WHO, could have introduced bias and impacted validity [
72]. Collectively, these limitations highlight the need for more diverse methodological approaches that include various vaccine types and robust adjustments for confounding factors, thereby improving the reliability and applicability of VE study findings across various settings.
The present review also has limitations. For instance, we did not assess VE across subgroups defined by vaccination status, number of doses received, vaccination regimen, booster status, or whether vaccination was homologous or heterologous. Furthermore, we did not incorporate meta-analyses of VE from other research or consider estimates of effectiveness that may vary over time. The periods characterized by the dominance of pre-Delta, Delta, and Omicron variants were not differentiated. Moreover, the effectiveness of other vaccine types, such as inactivated and viral vector vaccines, was not examined in this review.
Despite the identified limitations, a strength of this study is its comprehensive review and critical assessment of the methodologies used in COVID-19 VE research. The structured and systematic approach of this review improves our understanding of the challenges and opportunities in this field. By highlighting the prevailing methodological trends and emphasizing the need for more nuanced evaluation, this article serves as a valuable guide for future research and public health initiatives. Acknowledging the limitations demonstrates the study’s commitment to transparency and contributes to the ongoing discourse on refining VE research methodologies, ultimately advancing our understanding of VE in diverse settings.
, Jiae Shim2
