Analyzing the Effectiveness of Digital Technology in Vaccine Supply Management in Kano State, Nigeria

I. INTRODUCTION

Full routine immunization for children below 24 months now costs between US\$37 and US\$101 across different settings in low- and middle-income countries as more and costlier vaccines are being introduced into national immunization programs [1]. The rising cost of vaccine and vaccine distribution, global supply shortages, and the introduction of new vaccines have provided the impetus to deploy a real-time management system to negate the uneven distribution of vaccines, improve efficiencies and allow better visibility of the vaccine supply chain [2,3].

A robust vaccine cold chain and logistics system including storage and transportation is the cornerstone of all immunization programmes [4]. Rigorous supply planning based on accurate data should enable such a system to have the right vaccines in the right place, at the right time, in the right quantities, in the right condition, and at the right cost [3].

The 2021 Multiple Indicator Cluster Survey-National Immunization Coverage Survey (MICS-NICS) report showed that only about of all children between 12-23 months old have received all basic vaccination doses in Nigeria; while only about of those aged between 24-35 months had received all the recommended vaccines on the national immunization schedule in Nigeria in 2021 [5].

These indicators for measuring RI performance have shown slow progress in full immunization coverage in Nigeria despite huge investments and improvements in supply chain and distribution of vaccines in the last 5 years. However, the increasing incidence of stockouts and disrupted access to vaccines is of growing concern [3]. The majority of these stockouts are a result of internal issues such as inaccurate forecasts, stock management issues and restock delays [6].

Vaccine supply chain management systems have been prominent features of the Expanded Programme on Immunization (EPI) plans from the onset [7]. System requirements have expanded dramatically over the past several years with the introduction of new vaccines and the frequent mass campaigns to control, eliminate, or eradicate specific diseases, such as polio, measles, rubella, tetanus. While such initiatives tend to be well-funded, the strain they place on the supply chain system are substantial and often not acknowledged, particularly at state levels [7]. The most visible impact of new vaccine introduction is an increase in the volume of products that need to be stored, transported, and tracked, as well as the need for more storage capacity due to the increased use of single-dose vials [8].

These further exacerbates existing and new challenges in vaccine supply chain management like poor cold chain equipment inventory and status; poor vaccine stock, distribution and utilization data collection and management; lack of skilled and properly trained workers in vaccine management operations; poor vaccine storage facility; and unreliable micro-plan data which leads to wastage, stock-outs and overstock, and inequitable distribution of products . It is important to note these challenges and many other vaccine supply chain management problems are related to inefficient database management systems and innovative digital tools .

In 2016, the Ministerial Conference on Immunization in Africa laid the groundwork for the landmark Addis Declaration on Immunization (ADI), including 10 commitments to achieve universal and equitable access to immunization in Africa. The ADI was endorsed by Heads of State from across Africa at the 28th African Union Summit in early 2017, signaling political support for immunization on the continent at the highest possible level [13]. Real-time data and digital tools were identified as a powerful instrument to advance these efforts. Digital platforms make it easy to assess progress and to revisit decisions along the way, instead of waiting for quarterly reports or midline assessments. Digital tools also democratize learning and decision-making, so that stakeholders at all levels of the health system can make meaningful use of data.

In response to these assertions, eHealth Africa, a non-governmental organization (NGO) was funded by the Bill and Melinda Gates Foundation (BMGF) and the Kano State Primary Health Care Development Agencies (KSPHCDA) to conceptualize, design and roll out an electronic logistic management information system, LoMIS Suite to strengthen accountability and data-driven management in vaccine supply chain management in 2014. The Suite consists of an offline-capable mobile application "LoMIS Stock" and a web application and "LoMIS Deliver." LoMIS Stock was designed to help health care workers (HCWs) bypass the traditional paper-based reporting system and submit reports instantly using an app on their mobile device. The app was also built to enable supervisors to get near real-time visibility of stock inventory, consumption rate (vaccine wastage, stock out, utilization), cold chain status and cascade deliveries from all locations where vaccines are stored. LoMIS Stock answers three critical questions "Where are my Vaccines? ""Are they sufficient?" and "Are they potent?" LoMIS Deliver on the other hand aggregate these reports on stock levels, equipment status, and vaccine utilizations into a single web dashboard for real-time data visualization of field operations. Both applications are based on the structure and workflows of the KSPHCMB system.

1.1 Evaluation Questions

This study was conducted to answer the following research questions:

1. Is the LoMIS stock tool being used by the proposed users to report routine immunization vaccine supplies?

2. What is the effect of its use on the duration in reporting stock out and the turnaround time?

3. What is the impact of the tool on facility stock out experiences?

4. How has the tool supported the routine immunization stakeholders to plan for vaccine distribution?

1.2 Evaluation Objectives

In line with the research questions, we were guided by the objective of understanding the impact of using digital data management tools in vaccine supply chain management against the traditional paper-based systems in Kano state. Specifically,

1. To determine the use of LoMIS and the experiences of facility stock out prior and after the use of LoMIS stock tool in Kano state Nigeria

2. To understand effect of LoMIS Stock on vaccine supply planning and management in Kano state, Nigeria

1.3 Research Area

Kano state has an estimated population of approximately 14,994,830 according to the National Population Commission's 2019 projection. A district health information system 2 (DHIS2) tool with a customized routine immunization (RI) module and indicator dashboard was introduced in Kano State, Nigeria, in November 2014 to improve data management and analysis of RI services. A district health information system 2 (DHIS2) tool with a customized routine immunization (RI) module and indicator dashboard was introduced in Kano State, Nigeria, in November 2014 to improve data management and analysis of RI services [14]. DHIS2 has recorded the following success on RI coverage based on integrated technical solutions and interventions. The average routine immunization coverage rate (Penta 3) in Kano state from 2014 till 2021 is as shown in Figure 2. The selected study Local Government Areas (LGAs) within Kano and Bauchi states are indicated in Figure 1.

{"image_source":{"path":"images/775c8091e4e2e0976a4eaf76ff63a038d3eaefc410cc6ecbe4243f3f0aca8891.jpg"},"content":"","chart_caption":[{"type":"text","content":"Figure 1: Map Showing Study Locations in Kano and Bauchi States, Nigeria"}],"chart_footnote":[{"type":"text","content":"Source: DHIS2"}]}

Figure 2: Trend of Penta 3 Coverage (%) in Kano State, Nigeria

Bauchi State, according to the National Population Commission of Nigeria, has an estimated population of approximately 6.3 million people. In 2022, RI coverage in Bauchi State reached 83% as indicated in Figure 3. The Bauchi State Government, through its Task Force on Primary Health Care continues to make a strong commitment to ensuring the availability of adequate stock of vaccines, essential medicines, and consumables in health care facilities, through an improved supply chain system and Drug Revolving Fund.

{"image_source":{"path":"images/350d75a35fbcbbf9133b15c7ca0f15e12b593698f88ed86e5e902f9f7be3236e.jpg"},"content":"","chart_caption":[],"chart_footnote":[{"type":"text","content":"Source: DHIS2"}]}

Figure 3: Trend of Penta 3 Coverage (%) in Bauchi State, Nigeria

1.4 Research Limitations

The main limitation encountered in this paper is the quality of data and data availability in the DHIS2 data. Especially in Kano state, we were unable to conduct robustness checks on stock out by vaccines due to the unavailability of data.

II. STUDY DESIGN AND METHODS

2.1 Study Design

We employed the quasi-experimental design using a mix of quantitative and qualitative research methods to determine what would have been the outcomes in Kano state if the digital tool (intervention) had not been utilized. In this study, we identified Bauchi State vaccine supply chain management system as the comparison group based on validated evidence that Bauchi state was utilizing the paper-based reporting tools for vaccine stock inventory as of the period of this study.

2.2 Sampling techniques

Our study respondents were selected by using a three multi-stage sampling technique to select and recruit sixty (60) and forty (40) Ward Technical Officers/Routine Immunization Officers (10) for the intervention (Kano state) and the comparison (Bauchi state). We stratified all the state LGAs under the three senatorial zones in both Kano and Bauchi States and used a purposive sampling approach to select 6 LGAs under the 6 Administrative Zones of Routine Immunization in the Kano State, and 4 LGAs under the 2 Administrative Zones of Routine Immunization in Bauchi State based on their geographical proximity to each other. Ten (10) Ward Technical Officers (WTOs/RIOs) representing 10 wards from the selected LGAs in both states were randomly selected for the study.

The WTOs from Kano State were users of the Mobile Application (The Health Facility In-Charge) and Dashboard (State and Development Partners in Kano State) while the WTOs from Bauchi State (comparison participants) were Routine Immunization Officers/HFiC and development partners.

2.3 Data Collection and Analysis Methodology

Primary data for this study were collected using the evaluation tools (structured questionnaires and Key in-depth interviews) while secondary data was collected from records on the LoMIS Dashboard and District Health Information System Software (DHIS2). LoMIS dashboard is a platform that enables near-real-time visibility, storage, and retrieval of vaccine stock inventory while DHIS2 enables the collection, storage, retrieval and management of case-based data records. The primary data sources provided information on duration of reporting vaccines, duration of response to reports, and the benefits of LoMIS. The secondary data sources also provided information on the duration of vaccine reports and duration of response to those reports. The secondary data sources provided more information including the usability of LoMIS, availability of stock or stock-out.

All quantitative data was entered, analyzed using Statistical Package for Social Science (SPSS,). Analysis of Variance (ANOVA) was employed to determine the effect of the LoMIS tool on stock-out data in Kano. Paired and independent t-tests were used to determine the significance of the mean difference between stock-out data between pre/post-LoMIS and to compare stock-out reports in Kano and Bauchi respectively. To further understand the impact of LoMIS on stock-out occurrences in facilities, time series analysis (Expert Model) was used to predict 12 months (August 2014 - July 2015) trend of stock-out if LoMIS was not introduced as a reporting tool in Kano State. Qualitative data analyzed with the use of Atlas-ti, thematic results in both States were analyzed and compared.

III. RESULT AND DISCUSSION

immunization areas across Kano (73%) and Bauchi (90%) states (Figure 4). Study participants in Kano (40%) and Bauchi (60%) have worked as WTO/RIO for more than 3 years. All State and development partners interviewed have had relevant work experience as key decision makers whilst planning and managing RI vaccine supplies in Kano and Bauchi States.

3.1 Respondent's Demographics

In this study, it was found that males are the predominant staff working within the routine {"image_source":{"path":"images/8b048968df0de6c107fc7ec90c9e33e5cac809bb9267c0861fdd95fb51b3ffe4.jpg"},"content":"","chart_caption":[],"chart_footnote":[{"type":"text","content":"Source: Study Survey data"}]}

Figure 4: Sex of Participants Across the Study Locations

3.2 Analysis of findings by Research Questions

3.2.1 Research Question 1: The Utilization of LoMIS Mobile Application Tool in Reporting Vaccine Stock Inventory in Kano State (Usability Rate)

Findings from the desk reviews revealed that 484 facility personnel had been trained on how to use LoMIS Stock app for vaccine stock inventory. Analytics from LoMIS Dashboard showed 82% usage rate, that is, 397 of 484 Ward Technical Officers had used LoMIS application to report at least one stock inventory. Among all (60) users of the tool interviewed 60% are current users of LoMIS Stock as of the period of study.

Non-current users (40%) of LoMIS Stock have had experience of its use but issues related to network and faulty/missing phones stopped them from using LoMIS in reporting vaccine stock inventories. The LoMIS web dashboard further revealed increased use of the tool over time. As of December 2017, the usability rate had improved from 105 (22%) (January 2015) to 304 (63%) resulting in 190% increase in usage rate (Figure 5). As expected, the survey findings affirmed that Bauchi state was only using the paper-based reporting tool as of the period of study.

{"image_source":{"path":"images/0c2e484b06e2f9fb79fa863fa2d2a0bdbb23167c169526b0be36e9434dbc2b76.jpg"},"content":"","chart_caption":[],"chart_footnote":[{"type":"text","content":"Source: Google Analytics"}]}

Figure 5: Users of LoMIS Application Overtime

3.2.2 Research Question 2: Duration of reporting RI vaccine stock inventories and turnaround time

The findings related to the amount of time it takes to report RI vaccine inventories via LoMIS Stock in Kano state and the paper-based tools in Bauchi state were compared. In Kano state, it was found that 67% of respondents reported that it takes Pop more than 10 minutes to complete the process of reporting vaccine stock inventories via the paper-based tools while 100% of the respondents reported an average of 5 minutes to complete a report via LoMIS Stock. This is consistent with the findings from the comparison group (Bauchi State) where 63% of them reported spending more than ten minutes whilst reporting through paper-based tools.

Findings from the tool dashboard validates the duration of reporting via LoMIS. Users of LoMIS application tool used an average duration of less than 5 minutes on any stock inventory form as illustrated in Figure 6.

Source: Study Qualitative Finding

Figure 6: Impact of Lomis Stock Tool on Ri Vaccine Inventory Reports and Turnaround Time

Qualitative responses elicited through the interviews conducted further validates improvement in timely report of vaccine stock.

"With LoMIS, I can report in just two to five minutes and can get a response in a day or two. LoMIS is really supporting my work very well because it has many purposes."

State Partner, Kano

Respondents in Bauchi State on the other hand gave unreserved complaints on the use of paper-based tools to report vaccine stock. Majority of the challenges identified is time wastage which LoMIS Stock has been able to address in Kano State

"Paper-based reporting tools are tedious and time-consuming....."

State Partner, Bauchi In Kano state, it was found that out of the 39 respondents that provided answer to the question of average response time to stock out reports, 34 (57%) reported that all stock-out reports done through the LoMIS tool got response under 24 hours as compared with 20% that got response under 24 hours prior the use of LoMIS stock.

Twenty-two (59%) of 37 responses revealed that Cold Chain Equipment breakdown through LoMIS got a response within 24 hours as compared to 10% who got a response in 24 hours before the use of LoMIS (Figure 7).

Source: Study survey data

Figure 7: Comparing the Reported Improvement in 24-Hour Response to Stock Out and Cce Reports via Lomis and Paper-Based Tool in Kano State

3.2.3 Research Question 3: The Impact of LoMIS Stock on experiences of RI vaccine stock-out at facilities

Another major finding from this assessment study is reduction in stock-out experiences in health facilities. It was found that 60% of facilities had at least reported an event of stock-out from January {"image_source":{"path":"images/19f678aa1ec29054ca1682b577ad7d8f5002d1227845efbd071d217934ee6e5d.jpg"},"content":"","chart_caption":[],"chart_footnote":[{"type":"text","content":"Source: DHIS2"}]}

Figure 8: Trend of Stock Out Experiences in Kano and Bauchi State Prior and After the Use of LoMIS

At baseline, stock-out experiences in health facilities in Kano and Bauchi States were of similar figures. However, stock-out occurrences in Kano experienced drastic reduction over time as compared to Bauchi State (Figure 8). In addition, the differences in stock out outcomes in Kano and Bauchi states were confirmed as the Analysis of Variance (ANOVA) showed that the effect of the reporting tool on stock-out was significant (F(1,70,) = 215.506, p = 0.000). The comparison analysis using a paired-sample t-test showed that the mean difference in facility stock-out between a

3-year period prior and after LoMIS was found to be statistically significant (p=0.000) (Table 1).

This implies that LoMIS significantly contributed to reduced experiences of vaccine stock-out in health facilities. The independent sample T-test also revealed a huge significant difference between the means of facility stock-out in Kano (use of LoMIS) and Bauchi (Paper-tool) with a large effect size. Again, this result further validates the contribution of LoMIS to reduced vaccine stock-out in Kano (Table 2).

Table 1: Difference in Facility Stock-Out Prior/Post LoMIS Stock in Kano State

Table 2: Difference in Facility Stock-Out in Kano and Bauchi States: Stock-Out: LoMIS (Kano) vs Post-LoMIS (Bauchi)

3.2.4 Research Question 4: The Impact of LoMIS Stock on Data-Driven RI Vaccine Supply Planning and Management

In this study, it was found that LoMIS Stock has enabled near-real time RI vaccine reports, and this has led to improved visibility to the status of RI vaccines across health facilities in Kano state.

Enhanced vaccine visibility has enabled most of the decision makers to respond to stock-outs, CCE breakdown and over-stock of vaccines immediately. The planning RI vaccine supply has been influenced by the distribution data of RI vaccine status across the facilities.

"With LoMIS Dashboard, the status of the vaccine has been visible, that is visibility of stock availability. This helps me to take decisions in time."

"I use LoMIS Dashboard weekly to ensure there is no Stock Out in any facility. I also identify any CCE breakdown for replacement or repair. Recently, I saw Stock-out of BCG in a facility on the Dashboard and called the facility immediately."

State Partner, Kano

LoMIS is really an engine room, it's a bank for information on vaccine management. At a go, it is easy to see the status of CCE and make decisions at a go too. Non-functional CCE receives speedy repair.

Development Partner, Kano

"LoMIS is real time, for me, I see the status of vaccines immediately and I also have the contact of service providers and I just call. It helps my daily plan. For me, I go on LoMIS every day.

Development Partner, Kano

3.2.5 Research Question 5: Predicted occurrence of facility stock-out with Paper-reporting tools in Kano

To better understand and appreciate the contribution of LoMIS to reduced facility stock out in Kano; time series analysis (Expert Model) was used to forecast a 12-month (August 2014-July 2015) trend of occurrences of stock-out if LoMIS Stock was not introduced as a reporting tool in Kano State. The analysis showed that the trend of high stock-out prior July 2014 would have continued to be the situation if LoMIS Stock was not utilized. It was found that the trend of forecast data in Kano quite synchronizes with the observed stock-out data in Bauchi State (Figure 9). This further validates the effect of LoMIS on reduced stock-out occurrences in Kano state.

{"image_source":{"path":"images/8c85913f84e2f34d19b76fcfca7e86832995366fc799be55f5deb5a2a8d8dba6.jpg"},"content":"","chart_caption":[],"chart_footnote":[{"type":"text","content":"Data Source: DHIS2"}]}

Figure 9: Time Series Analysis: 12 Months Forecast of Stock-Out if LoMIS was Not Introduced

IV. DISCUSSION

We conducted a comprehensive assessment of the impact of LoMIS Stock reporting tool on vaccine stock management in Kano state. The results showed that LoMIS Stock has contributed to effective vaccine management in terms of real-time reports, vaccine visibility and accountability. These have led to timeliness of report, increased response rate, reduction in facilities experiencing stock-out and eventually availability of adequate stock at the last-mile facilities. The findings from this study further corroborates the results of several studies on the impact on the use of digital tools as an effective approach to public health interventions globally .

In Nigeria, one of the challenges of vaccine stock management occurs because of poor vaccine stock visibility at health facilities . Our study has shown that this challenge can be significantly addressed with a digital tool as we found that the use of LoMIS Stock for reporting vaccine stock inventory enhanced access to vaccine stock distribution across the focal facilities. The usability rate (82%) of LoMIS Stock shows that the use of digital tools is a prioritized solution to the challenges of public health interventions in low- and middle-income countries like Nigeria.

This corroborates the findings of a digital health study where it was found that digital technology is a feasible and acceptable approach for public health interventions . Specifically for immunization interventions, a recent study reported high acceptability rating amongst vaccinators and district managers for an iterative mobile-app that was developed to track immunization encounters in Pakistan which shares similar public health demographics with Nigeria . In another study in Malawi, end-users assessed the usability of two digital tools for community case management of under-five children—highlighting that although both tools are useful in gathering important intervention data, their usability differs by a wide margin and this determined the overall success of the tool in supporting health interventions in the study setting . This implies that the ease-of-use that was recorded with the end-users of the LOMIS Stock App contributed significantly to the success of the tool in managing vaccine supply chain management in Nigeria.

We found that the amount of time needed to complete a report of RI vaccine inventory significantly reduced using a digital health tool.

The paper-based responses in Bauchi State validates improvement in duration of response time using LoMIS Stock in Kano State as the responses were consistent with pre-LoMIS findings in Kano State. In another study, the results also posits that the use of digital technology for reporting routine immunization data is feasible and can provide real-time updates to RI performance indicators such as vaccine availability .

In addition, this study sought to assess the impact of the digital tool on facilities experiencing stock-out; and we found that the stock-out experiences reduced over time in Kano state.

Within 3 years of using the LoMIS tool, the facilities experiencing stock out reduced by 92% in Kano (LoMIS) as compared to only 42% reduction in Bauchi (paper-reporting tool) within the same period. The implication of this result is that there was sufficient stock of death-preventing vaccines at health facilities in the study location.

This finding strengthens the discussions around the value of developing and implementing digital health tools in public health interventions .

Furthermore, the impact of the tool, LoMIS Stock on vaccine planning and management was significant as our results showed that the LoMIS dashboard tool enhanced the vaccine operations visibility—providing near real-time data that enabled key decision makers to respond to stock-outs, CCE breakdown, and vaccine over-stock in record time, and ensure that immunization efforts meet expectations. Improved planning and management organization is another notable impact of the LOMIS dashboard on decision makers in the study location as we recorded in this study. This further contributes to existing evidence that data dashboards in public health tools provide reliable information to inform and support policy makers and project managers in iterating public health interventions . Several sources show that a wide range of COVID-19 response interventions utilized many dashboards to collect data on time series, geographic maps, case incidents, contact tracing, community surveillance and clinical trials to refine intervention approach, outputs and outcomes . A similar result was obtained by another researcher where a proactive infection prevention and control (IPC) monitoring tool was pioneered to provide reliable data for real-time response to emerging risks to COVID-19 infection amongst healthcare workers in Guangdong Second Provincial General Hospital in China. The study showed that the worrisome rate of healthcare workers infection prior to the deployment of the tool was reduced; and future plans to integrate artificial algorithms to the tool for improved outcomes are already being muted [30].

V. CONCLUSION

The implication of this result is that there was sufficient stock of death-preventing vaccines at health facilities in the study location. This finding strengthens the discussions around the value of developing and implementing digital health tools in public health interventions .

Furthermore, the impact of the tool, LoMIS Stock on vaccine planning and management was significant as our results showed that the LoMIS dashboard tool enhanced the vaccine operations.

Declarations

Consent for publication

Not Applicable

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests" in this section.

Funding

No Funding

Ethical Consideration

All methods were performed in accordance with the guidelines and recommendations specified within the Helsinki declaration. Official permission was sought and obtained from the participating vaccine stock management stakeholders in Kano and Bauchi states where the study was conducted. Ethical approval was obtained from the Kano state Ministry of Health ethics board (NHREC/17/03//2018). All participants provided written consent before participating in the study.

Informed Consent

All participants provided written informed consent before participating in the study.

Competing interests

The authors declare that they have no conflicts of interest.