An economic evaluation of the Virtual Rural Generalist Service versus usual care in Western NSW Local Health District

The poorer health outcomes and the increasing shortage of health care workers for Australians living in rural and remote areas versus urban communities are well documented.1,2,3,4,5,6 Persistent undersupply of health care workers, particularly general practitioners (also termed family doctors or primary health care providers), has led to recruitment difficulties and high staff turnover, threatening the sustainability of health care services in rural and remote Australia. A survey of 108 Australian rural and remote primary health care services in 2013 found that the highest direct cost of health care worker replacement was for doctors ($74 000) and that the lowest retention rate was for doctors, with the hazard of leaving employment for doctors being 68% higher than that for nurses; it also found that the hazard of leaving a remote position was 23% higher than that for a rural position.1

Telehealth services could be one solution for narrowing the gap and improving local workforce retention, but the barriers to adopting telehealth services in rural and remote areas are also well documented and uptake has been slow.2,3,7 In addition, there is lack of evidence on the effectiveness and cost‐effectiveness of telemedicine in rural hospital settings. To our knowledge, the few studies that have been conducted were concentrated on specialist clinicians, showing similar or improved quality‐of‐care outcomes (reduced numbers of transfers and time to transportation) and mixed results on costs.8,9,10,11,12 However, these findings may not be generalisable to general practitioners working in rural emergency departments (EDs) and inpatient settings, such as those that use the Virtual Rural Generalist Service (VRGS).

The VRGS model of care provides 24‐hour, 7‐days‐a‐week rural generalist doctors, 75% virtual and 25% in person, to provide relief from fatigue and support work–life balance for local general practitioners who are contracted to provide medical services to small rural and remote hospitals. When VRGS doctors support a facility in person, they perform the same duties and see the same patients as non‐VRGS (local and locum) doctors (ie, their in‐person role is consistent with usual care). Patients eligible for treatment provided virtually by a VRGS doctor are those presenting to an ED (predominantly lower acuity) and inpatients admitted by a VRGS doctor during a daily ward round or an ad hoc medical review. Virtual consultation requests are registered in the electronic medical record by on‐site nurses. Proactive ward rounds are automatically scheduled (eg, without an electronic medical record order) at sites where there are no local on‐site doctors. When the nurse and VRGS doctor are ready for a patient consultation, on‐site nurses bring a wireless telehealth cart equipped with high definition videoconferencing and peripheral examination devices to the patient's bedside.

We aimed to evaluate the cost‐effectiveness of the VRGS model of care by examining: incremental economic costs and benefits compared with usual care (ie, treatment by non‐VRGS doctors); maintenance of health service activity levels and workforce configuration after implementation of the VRGS; and the extent to which local doctors and district‐level and executive‐level managers see the VRGS as acceptable.

Methods

Study design

A cost–consequence analysis was performed to evaluate the economic costs and benefits of the VRGS compared with usual care from the perspective of the health care funder, the state government of New South Wales, over a 1‐year horizon. A pre‐defined economic analysis plan was developed and followed (Supporting Information, appendix 1). Total costs and disaggregated benefits were compared between VRGS and non‐VRGS models of care (cohort analysis) and between pre‐VRGS and post‐VRGS periods (pre–post analysis). Health service activity and workforce configuration were examined before and after VRGS implementation. The extent to which local doctors and local district‐level or executive‐level managers saw the VRGS as acceptable was determined from semi‐structured interviews and focus groups (Supporting Information, appendix 2, supplementary table 1).

Study setting

Twenty‐nine rural and remote hospitals in Western NSW Local Health District (WNSWLHD) where VRGS was in operation (VRGS sites) were included in the study. Two sites were excluded from the pre–post analysis as they had very low numbers of VRGS encounters.

Patients

The analysis included patients of any age who presented to an ED or were admitted to hospital at VRGS sites over the pre‐VRGS period (1 February 2019 to 31 January 2020) or post‐VRGS period (1 July 2021 to 30 June 2022). Patients undergoing haemodialysis without any other diagnosis and hospital‐in‐the‐home patients were excluded.

Intervention and comparator

The intervention was the VRGS model of care, which was implemented at all 29 WNSWLHD hospitals from 1 February 2019. The comparator was usual care, defined as in‐person treatment by the local or locum doctor (hereafter termed the non‐VRGS model of care).

Quality‐of‐care outcome measures

Pre‐defined quality‐of‐care measures and the WNSWLHD administrative datasets used for the economic analysis are detailed elsewhere in this supplement.13 The measures presented in this article are ED presentation outcomes: time from arrival to departure; numbers of presentations in which care was completed; numbers of presentations in which the patient was admitted, was transferred, did not wait, left at their own risk or died in the emergency department; and numbers of unplanned re‐presentations within 48 hours at any facility in the WNSWLHD.

Other outcome measures

Maintenance of health service activity from the pre‐VRGS period to the post‐VRGS period was measured by the change in total national weighted activity units (hereafter termed activity units) and the sustainability of the local doctor workforce was measured by changes in locum shifts. The extent to which local doctors and district‐level or executive‐level managers saw the VRGS as acceptable was determined by themes identified from structured interviews and focus groups (Supporting Information, appendix 2, supplementary table 1).

Costs

The cost to the health care funder for an ED presentation or hospital admission encounter is the activity units allocated to the encounter multiplied by the State Efficient Price (ie, the price of one activity unit) for the year. However, simply using one efficient price for each encounter does not account for observable and allocatable variations in cost due to the model of care (eg, medical staff salaries and wages, VRGS technology costs and administration costs). To incorporate this variation in the cohort analysis, the clinical costs that differ between VRGS and non‐VRGS models of care (allocatable costs) were estimated and divided by the total activity units delivered under each model of care to estimate a price per activity unit. The cost of each encounter was estimated as the activity units allocated to the encounter multiplied by the price per activity unit of the model of care used. In this way, the incremental cost per incremental unit of benefit for each model of care was estimated. The same type of process was followed for the pre–post analysis.

Allocatable costs

Additional allocatable costs identified for the VRGS model of care were equipment and telecommunication services for the remote consultations, VRGS administrator salaries, and operating expenses (eg, costs of running VRGS training days). Costs that differed between the VRGS and non‐VRGS models of care were medical salaries and travel expenses due to the differing numbers of in‐person shifts. Resource quantities, unit costs and data sources for allocatable costs are provided in the Supporting Information (appendix 1). Utilisation and unit costs of hardware and software were estimated via interviews with two WNSWLHD staff: the Telehealth Manager in the Health Information and Communications Team and the Rural Health Innovation Lead, Rural Sectors in the Service Delivery Directorate. Hardware purchase costs were annualised, assuming an estimated useful life of 5 years. Medical and administrator salaries were extracted from financial reports prepared by WNSWLHD's Business Manager, Rural Sectors. Data on locum rates and shifts were supplied by WNSWLHD's Medical Workforce Litmus Coordinator. Data on VRGS in‐person shifts were extracted from rosters. Travel expenses were estimated using market prices and statutory allowances.

Analysis and reporting

All costs are presented in 2022 prices.14 Costs and outcomes were not discounted given the 12‐month analysis horizons, but amortisation of equipment purchases was done using a 5% per annum discount rate in line with NSW Treasury guidelines.15 The analysis is reported in accordance with the Consolidated Health Economic Evaluation Reporting Standards 2022 checklist.16 All analyses were performed using R (version 4.2.2).17

Cost–consequence analysis

For the cohort analysis, each encounter in the post‐VRGS period was categorised into one of three models‐of‐care cohorts using the treating doctor information that was documented: VRGS (encounter with VRGS doctor/s only), non‐VRGS (encounter with non‐VRGS doctor/s only) and combined (encounter with both non‐VRGS and VRGS doctors). Incremental cost and disaggregated incremental quality‐of‐care outcomes for encounters in the VRGS cohort compared with encounters in the non‐VRGS cohort were estimated.

For the pre–post analysis, incremental cost and disaggregated incremental quality‐of‐care outcomes for encounters in the post‐VRGS period compared with encounters in the pre‐VRGS period were estimated. Binary outcomes were estimated via log‐binomial regression, and continuous outcomes and costs via linear regression. Bootstrapping with 1000 replicates was used to calculate incremental cost‐effectiveness ratio point estimates for each outcome measure (ie, the incremental cost of VRGS versus non‐VRGS divided by the difference in each quality‐of‐care outcome measure between VRGS and non‐VRGS) and 95% confidence intervals. Point estimates were plotted on cost‐effectiveness planes.

Scenario analysis was undertaken to understand the price per activity unit change from differing allocations of encounters that could not be categorised into a cohort because no treating doctor information was documented, and one‐way sensitivity analyses were conducted to understand the impact of uncertainty in critical inputs on incremental cost estimations.

Adjustment of quality‐of‐care outcomes and cost per encounter by potential confounders

Demographic and clinical confounders between encounters in the VRGS and non‐VRGS cohorts were identified, and the most predictive models were constructed for each quality‐of‐care outcome to adjust for significant confounders, as detailed elsewhere in this supplement.13 Adjusted odds ratios comparing quality‐of‐care outcomes between the VRGS and non‐VRGS models of care were estimated and reported.13

The cost–consequence analysis of quality‐of‐care outcomes and cost per encounter presented in this article did not require adjustment for confounders for several reasons. First, significant confounders were already taken into account in the weighting formulas for activity units by encounter. Second, the distribution of activity units by encounter was found to be almost identical between VRGS and non‐VRGS cohorts and between pre‐VRGS and post‐VRGS periods for ED presentations. Third, ED presentations represent 95% of encounters that involve the VRGS, and our analysis focuses on ED presentations. To mitigate any risk from potential confounders not captured in weighting formulas, results from the adjusted analysis, detailed elsewhere in this supplement,13 were compared against our unadjusted analysis to determine whether the direction, magnitude and significance of our results were valid.

Ethics approval

The study received ethics approval from the Greater Western Human Research Ethics Committee (project numbers 2021/ETH01379 and 2021/ETH01355).

Results

Characteristics of ED presentations and admissions

Data analysis flowcharts for the cohort and pre–post analyses are presented in the Supporting Information (appendix 2, supplementary figures 1 and 2). The characteristics of ED presentations and admissions (patient demographics and clinical complexity) by model of care and by period are also presented in the Supporting Information (appendix 2, supplementary tables 2–5). The comparisons of the distribution of activity units per encounter between cohorts and between periods (Box 1) provide insights into the relative complexity of encounters between models of care and periods, which reflects differences in factors such as patient and facility remoteness, Indigenous status, and clinical complexity.

The cohort analysis of ED presentations showed that the number of encounters in the VRGS cohort was about half of that in the non‐VRGS cohort, and that encounter complexity in the VRGS cohort was only 10% less than that in the non‐VRGS cohort (the remoteness of patients offset a slight decrease in clinical complexity, consistent with the VRGS role to treat lower acuity presentations) (Box 1). Higher clinical complexity was observed in the combined cohort, which had substantially higher mean activity units per encounter.

The cohort analysis of admissions showed that the VRGS dealt with about one‐fifth as many encounters as non‐VRGS services (Box 1). Consistent with the role of the VRGS to perform pre‐emptive ward rounds, the complexity of VRGS admission encounters was 40% lower than that for non‐VRGS admission encounters.

The pre–post analysis showed that the number of ED presentations increased by less than 1% after the VRGS was implemented and that encounter complexity remained the same (Box 1). This analysis also showed that hospital admissions were reduced by 20% (owing to pandemic restrictions) but that patients admitted in the post‐VRGS period had 11% higher clinical complexity than those admitted in the pre‐VRGS period.

Price per activity unit for each model of care and period

Total allocatable costs for each model of care and period are shown in Box 2. In the cohort analysis, total allocatable costs for the non‐VRGS cohort were 3.6 times those for VRGS cohort, driven by a much larger medical salary and wage expense that was not offset by capital investments and ongoing expenses for technology required for the VRGS. In the pre–post analysis, total allocatable costs increased by 20% from the pre‐VRGS period to the post‐VRGS period: medical salary and wage costs increased by 18%; travel, meals and accommodation expenses decreased by 44% owing to fewer locum shifts; and an additional VRGS technology and administration expense of almost $700 000 was incurred in the post‐VRGS period.

A price per activity unit per model of care and period was derived by dividing total allocatable costs by total activity units (Box 2). In the cohort analysis, the combined cohort activity was added in equal parts to the VRGS and non‐VRGS cohorts, as this corresponds with the definition of the combined cohort. Activity units for encounters that could not be categorised into a cohort (because no treating doctor information was documented) were allocated equally as a base case, as extensive investigation found no systematic relationship between model of care and these encounters, and scenario analysis was undertaken using three other allocation methods that showed no matter how these activity units were allocated, the non‐VRGS model of care always resulted in a higher price per activity unit than the VRGS model of care (Supporting Information, appendix 2, supplementary table 6). At its lowest cost, the non‐VRGS model of care still cost 1.14 times more to deliver the same unit of health care compared with VRGS model of care ($1575 v $1379).

Under the base case, the non‐VRGS model of care was estimated to cost 1.7 times more to deliver the same unit of health care compared with the VRGS model of care ($1753 v $1047) (Box 2). One‐way sensitivity analysis using low and high estimates of allocatable costs for equipment and travel‐related expenses resulted in a range of 1.51 to 1.75, respectively, around the 1.7 estimate.

In the pre–post analysis, given the increase in allocatable costs of 20%, it cost 1.25 times as much to deliver one unit of health care in the post‐VRGS period compared with pre‐VRGS period (Box 2). One‐way sensitivity analysis using low and high estimates of allocatable costs (no increase in non‐VRGS medical salaries from the planned 5‐year local doctor contract renegotiation in the post‐VRGS period and no decrease in locum days in the post‐VRGS period) resulted in a range of 1.16 to 1.30, respectively, around the 1.25 estimate.

Incremental cost‐effectiveness per ED encounter

Incremental cost‐effectiveness results and interpretations for the cohort and pre–post analyses are presented in Box 3 and Box 4. The average cost per ED encounter was $105 (95% CI, $103–$107) less, a decrease of 44%, for VRGS encounters compared with non‐VRGS encounters ($134 v $240) (Box 3). VRGS ED encounters were less costly and more effective in terms of time from arrival to departure; less costly and non‐inferior in terms of the number of admissions, departing for other clinical service locations, and deaths; and less costly and less effective in terms of patients not waiting, patients leaving at their own risk, and unplanned re‐presentations within 48 hours. The VRGS cohort had proportionately more consultation completions and fewer transfers than the non‐VRGS cohort, which is consistent with recently published data on telemedicine in rural EDs.18,19

The cost per ED encounter increased by $41 (95% CI, $40–$42), a 25% increase, from the pre‐VRGS period to the post‐VRGS period ($160 v $201) (Box 4). Post‐VRGS ED presentations were more costly and more effective in terms of departing for other clinical service locations; more costly and non‐inferior in terms of transfers, patients leaving at their own risk, and deaths; and more costly and less effective in terms of time from arrival to departure, patients not waiting, and unplanned re‐presentations within 48 hours. In the post VRGS period, there were proportionately more consultation completions and fewer admissions than in the pre‐VRGS period.

Cost‐effectiveness planes for ED presentations

Using a bootstrap procedure to estimate uncertainty, estimates of the incremental cost and incremental effects of VRGS versus non‐VRGS for ED presentations are plotted on cost‐effectiveness planes (Box 5; Supporting Information, appendix 2, supplementary figures 3 and 4).

Health service activity and workforce configuration

It is challenging to separate pandemic and VRGS impacts in the pre–post analysis. However, the analysis helps examine changes in health service activity levels and workforce configuration owing to VRGS implementation during a pandemic. Health service activity was maintained in the post‐VRGS period, only declining by 4% (13 805 v 14 368), which may suggest pandemic resilience of the VRGS model. Although medical salaries rose after implementation of the VRGS, this was mainly a result of a planned 5‐year local doctor contract renegotiation, unfortunately timed during the pandemic. An integral part of these new contracts was VRGS support after hours, and this may have prevented an even larger salary increase, although we were unable to measure this impact. However, we found that workforce composition became more sustainable in the post‐VRGS period, with locum shifts decreasing from 1456 days to 609 days. Unfortunately, during the pandemic, VRGS doctors managed only 11% of shifts in‐person (241/2281), short of the intended 25% that would have further increased local workforce sustainability. This target is now a key focus for VRGS management.

Through our interviews with district‐level and executive‐level managers, we found that the VRGS was seen as an asset in navigating medical workforce shortages and fluctuations because of the agility and scalability of the model. Managers reported that while they continued to try to fill every vacancy with a doctor on the ground, a national shortage of locums made this impossible. The VRGS enabled medical coverage at any site when a local or locum doctor position was not filled. The service could be scaled up or down to boost clinical capacity. However, managers noted that the extra demands of the VRGS on nursing staff have not been factored into staffing ratios and there was a potential need for an additional non‐clinical role of “virtual navigator” or “clinical support officer” to practically and administratively support virtual services.

In our interviews with local doctors, it was confirmed that the VRGS improved the appeal and sustainability of local doctor positions at rural sites by providing fatigue relief (eg, covering night and weekend shifts).20 More details on insights from these interviews are provided in the Supporting Information (appendix 2, supplementary table 1).

Discussion

From the cost–consequence cohort analysis, we found that the price per activity unit (ie, for one unit of health care) was lower for the VRGS model of care than for the non‐VRGS model of care. This was not an expected result given the mixed results on cost outcomes reported from telemedicine economic evaluations8,9,10,11,12,21 and the many additional capital investments required for a virtual service. However, the authors of a systematic review of economic evaluations of telehealth services using real‐time video concluded that the organisational model of care was the most important factor in determining the economic value of a service.10 In addition, another systematic review showed that factors associated with successful and sustainable Australian telehealth services include: having a well defined, efficient process for managing activity (such as the VRGS's flexible and adaptable model that is responsive to local needs), and careful consideration of the equipment used and requirements for technical support.3

We found that the VRGS model of care was more cost‐effective than the non‐VRGS model of care for ED presentations, where encounters were of a similar clinical complexity. Care provided by the VRGS cost less per encounter, and resulted in similar and some improved quality‐of‐care outcomes, compared with non‐VRGS care. While small, the higher rates of VRGS compared with non‐VRGS patients not waiting to be seen, leaving at their own risk, and re‐presenting unplanned within 48 hours at any WNSWLHD ED should be closely monitored to ensure that this difference does not have long term consequences.

Given the finding of increased efficiency of VRGS care over non‐VRGS care, and the cost‐effectiveness of the care provided by the VRGS in EDs, it is interesting to reflect on the pay differential and perceived value of a virtual versus in‐person doctor. At the time of our analysis, daily shift rates were, on average, $400 higher for local and locum doctors than for VRGS doctors. After our analysis was complete, VRGS daily shift rates were increased, reducing the differential to an average of $100 per day — a difference that reflects the value of both attendance modes (75% virtual and 25% in person), and the lower acuity and lower complexity of VRGS encounters when staff are working virtually. We note that even under this VRGS daily rate increase, a non‐VRGS unit of health care would still cost 1.5 times that of a VRGS unit of health care, so this would not change the evaluation results that we have presented in this article.

In terms of clinical capacity and workforce configuration, numerous reviews on the impact of telemedicine on health care workers suggest that telemedicine improves decision‐making processes, enhances delivery of care, and is associated with high satisfaction rates among health care workers.22 Some reviews provide evidence showing that telemedicine might improve efficiency and performance metrics, further career advancement, expand knowledge, and facilitate health care workers’ training and recruitment.22 A few studies also provide evidence showing that telemedicine increases health care worker burden and burnout.22 We are unaware of reviews that provide evidence on how the use of telemedicine might affect retention of health care workers, but the results of some reviews suggest that insufficient staffing, frequent staff changes, and increasing workload for nurses and care coordinators are barriers to telemedicine use in rural and remote Australian health services.2 Similar views were expressed in the interviews that we conducted with district‐level and executive‐level managers; participants noted that while the VRGS complements and supports existing medical capacity, there is a need for additional on‐the‐ground nursing and technical staff to support the service, plus skills development training for local clinical staff to ensure the sustainability of the VRGS.20 Nonetheless, we found that WNSWLHD managers support the VRGS model because of its flexibility to scale up or down according to local needs. Consistent with this finding, adaptability and efficiency were frequently reported as factors that contribute to success and sustainability in a systematic review of 72 telehealth services in rural and remote Australia.3

Our economic analysis of VRGS had some limitations. It was highly manual and time‐intensive. VRGS encounters cannot be easily identified in datasets used for clinical costing, activity‐based funding and performance reporting (eg, ED and admitted patient data collections). Without investment in data capture to record VRGS identifiers for encounters in regular administrative datasets used for clinical costing, activity‐based funding and performance reporting, the service will struggle to transparently demonstrate cost and time savings with comparable benefits to in‐person services over time, which has been found to be an important factor for influencing the success and sustainability of telehealth services in rural and remote Australia.3

In addition, ascertainment of whether shifts were undertaken by a local or locum doctor needed to be done manually, by identifying doctor's names, which may have introduced some measurement errors in the calculation of travel expenses and workforce configuration estimates. However, actual medical salary and wage expenses were used, so this manual identification process did not affect our salary and wage expense data.

Another limitation was the assumption that on‐site staff, particularly nursing staff, spent equal time on VRGS encounters as for non‐VRGS encounters. A time‐and‐motion study would be required to accurately validate or challenge this assumption, and this would be critical in determining a clinical cost and the funding cost for VRGS encounters. In addition, qualitative feedback from WNSWLHD managers suggested that more support is required for local nurses and technical staff, so it is likely that on‐site staff require more time for VRGS encounters.

In conclusion, we found that the VRGS provided lower cost and equivalent quality‐of‐care outcomes when compared with usual care for ED presentations and lower complexity admissions in rural and remote hospitals, supporting local clinical staff to maintain service activity levels even through a pandemic. However, investment is required to ensure administrative datasets can capture VRGS encounters to transparently demonstrate cost and time savings with comparable benefits to in‐person services. With additional investment in data capture, nursing workforce and technical staff to support the service, the VRGS has promise as an economically viable and scalable service for sustaining access to and quality of medical care in rural and remote NSW. Furthermore, this model of care may be applicable in other rural and remote areas in Australia and overseas.

Box 5 – Cost‐effectiveness planes for VRGS versus non‐VRGS models of care for ED presentations*

---

ED = emergency department; NE = north‐east; NW = north‐west; SE = south‐east; SW = south‐west; VRGS = Virtual Rural Generalist Service. * The planes show the incremental cost: per minute saved in arrival to departure time (panel A), per ED presentation within 4 hours (arrival to departure time) (panel B), for avoiding a patient not waiting (panel C), for avoiding a patient leaving at own risk (panel D), for avoiding an unplanned re‐presentation to an ED within 48 hours (panel E), per death in ED avoided (panel F).