Artificial intelligence for surgical services in Australia and New Zealand: opportunities, challenges and recommendations

Artificial intelligence (AI) is being rapidly taken up by society, including health care services, and will inevitably be used broadly within the surgical services of Australia and New Zealand. However, the process of AI implementation must be evidence‐based, safe, and ethically cautious,1 and must adhere to recommendations of the international surgical data science community.2 AI has numerous limitations and should always serve as an adjunct that benefits outcomes, rather than replacement of the staff within surgical systems. This perspective discusses opportunities and challenges for the use of AI in the surgical services of Australia and New Zealand and provides recommendations for the future.

Current non‐surgical views on AI in Australia and New Zealand

Non‐surgical colleges across Australia and New Zealand have begun to discuss AI, but, at the time of writing, of the 25 specialist medical colleges within the Australian Medical Council, only three (12%) have published AI position statements.3,4,5 This will probably change with time, and surgical bodies should remain aware of current opinions held by their non‐surgical counterparts. There is broad consensus that greater implementation of AI may increase care efficiency, potentially facilitating service provision across demographic groups and rural populations; although there also exists a risk of augmenting existing health inequity.5 Further, AI might improve diagnostic accuracy and efficiency, particularly for rare diseases.5 However, significant concern exists regarding ethical issues and other risks. AI presents potential liability implications arising from automation bias, and patient safety concerns from decreased sophistication in human decision making.3,4,5,6,7 As surgical decision making often carries substantial consequences, any degree of AI automation should first be scientifically interrogated and should be implemented only when there is demonstrable patient and system benefit, ensuring surgical staff roles are preserved.6

When integrating AI within surgical systems, it is important that the views of wider Australian and New Zealand society, a key stakeholder group, are also considered. A 2021 systematic review of the global literature on patient and general public attitudes to clinical AI found overall positive attitudes but many reservations and a preference for human supervision.8 However, an update of this review might produce different results, given how rapidly AI has been taken up by broader society through tools such as ChatGPT. Specifically for surgery, evidence is limited, but studies have found patient acceptability towards AI, apart from fully autonomous surgery.9 Although it is crucial to conduct serial assessments of Australian and New Zealand general public opinion to understand the current state of play, large studies such as the AVA‐AI study provide reliable evaluations of national views.10 Within this large Australian survey, about 60% of respondents supported the development of AI, but this decreased to 27–43% when questions regarding specific health care scenarios were posed.10 A similar study of the Australian and New Zealand general public has not been conducted for the use of AI by surgical services specifically, and this presents an important gap for future research.

Development and implementation

The global AI boom presents opportunities to substantially enhance the quality and efficiency of Australia and New Zealand surgical services, both at the systems level and in the pre‐operative, intra‐operative and post‐operative phases of individual patient journeys. At the systems level, locally developed AI tools, such as the Adelaide Score developed in South Australia, can potentially increase efficiency through providing additional predictive data relating to events that occur in each inpatient admission, such as hospital discharge.11 For individual surgical patients, AI tools can enhance clinical decision making at all time points, such as pre‐operative informed consent and risk assessment, intra‐operative precision and vision, and post‐operative care for improved recovery and follow‐up.12 However, when exploring these opportunities for surgical services, it is imperative that all AI tools are developed, validated and implemented using evidence‐based and internationally accepted approaches, while also being trained with local data. This is challenging but crucial, regardless of the planned use of the AI tool. Currently, relevant evidence for AI use in Australian and New Zealand surgical services is mostly limited to early phase studies that might not reflect real‐world practice. Prospective AI implementation trials within the real‐world clinical context are required, and only after clear benefits are demonstrated for surgical patients and systems, they can be regularly used in Australian and New Zealand surgical services.1 There is currently a paucity of randomised trials regarding AI interventions, although this will change in coming years.13 Further, multiple statements have now been developed for reporting AI research and are listed by the EQUATOR network.14 It is crucial that similar frameworks for AI use are developed specifically for surgical services in Australia and New Zealand, so that the present opportunities can be explored safely, while also ensuring optimal benefit for local patients and systems. Until these are more developed, local surgical technology assessment organisations, such as ASERNIP‐S within the Royal Australasian College of Surgeons (RACS),15 can assist with ensuring adherence to evidence‐based principles during the uptake of AI by Australian and New Zealand surgical services.

Post‐implementation monitoring and audit

After opportunities for implementing AI are pursued by Australian and New Zealand surgical services, an additional challenge is ensuring appropriate and effective post‐implementation monitoring and audit processes. These processes must evaluate ongoing outcome benefits to surgical patients, staff and systems, while also ensuring that local factors relevant to individual surgical services continue to be optimally considered. Adequate infrastructure for audit, retraining (if suboptimally providing benefit or considering local factors), and abandonment (if unresolvable surgical safety concerns are identified during audit) for surgical AI tools should be in place before the broad use of the tools within clinical practice. It is essential that surgical services also correspond and collaborate with local and national audit bodies, such as the Australian and New Zealand Audit of Surgical Mortality of the RACS, when using AI tools that could have significant outcome implications. Surgical patient safety and confidentiality must be maintained, particularly with increasing commercial interests in AI that may potentially conflict with the overall public good.16 As with any surgical innovation, AI must be rigorously audited and its use within the surgical services of Australia and New Zealand must always adhere to evidence‐based principles.16

Engagement with regulatory bodies

The integration of AI within Australia and New Zealand surgical services must be carefully regulated, and this presents an important challenge when clinical opportunities are being pursued. If done too rapidly and with little governance or consideration of necessary protection from profit‐driven commercial entities, unpredicted risks could be conferred to future surgical patients. Surgical oversight bodies such as the RACS play a crucial role for this, but engagement with external regulatory bodies is also required. Novel AI devices may also be required to obtain pre‐market approval from the Therapeutic Goods Administration (TGA) to be listed in the Australian Register of Therapeutic Goods before widespread implementation, which would allow surgical AI devices to be legally supplied in Australia.6 Technically, surgical devices that incorporate AI are classified as software as a medical device (SaMD), regulated by the TGA. The TGA is also a founding member of the International Medical Device Regulators Forum, therefore operating with the goal of remaining consistent with the International Medical Device Regulators Forum regarding the regulation of SaMDs. The Australian Commission on Safety and Quality in Health Care also provides standards for AI and automated decision making in the health care sector.17 From a legislative point of view within Australia, both the Therapeutic Goods Act 1989 and the Therapeutic Goods (Medical Devices) Regulation 2002 specify the conformity assessment procedures, classification rules and principles for regulation in Australia, which also apply to AI in surgical systems. Further, bodies such as the Australian Ethical Health Alliance and the Australian Alliance for Artificial Intelligence in Healthcare also provide guidance for regulating AI that is applicable to Australian and New Zealand surgical systems.18 Overall, there is a need for regulatory authorities to provide a clear framework specifically for the use of AI by surgical services within Australia and New Zealand.

Ethical considerations

Alongside the challenges raised, substantial ethical dangers accompany AI, and caution must be applied when exploring opportunities to integrate this technology to assist surgical care.19 Overriding principles such as “first, do no harm” must always be upheld. AI algorithm performance is influenced by the characteristics of training datasets, and outcome benefits can differ depending on whether public data or real‐world local data are used.20 Further, there may be bias in training data, which risks the AI algorithms perpetuating or exacerbating pre‐existing inequalities (such as race, sex, age or socio‐economic status), leading to discriminatory outcomes.21 Surgeons may treat a specific patient population due to the location of their institution or specialised professional interests, and algorithms trained on datasets derived at a population‐level may perform suboptimally at a local level surgical services should be aware of potential biases in algorithms and limitations of training data and regularly audit AI‐driven systems after local deployment.

AI systems are frequently described as “black boxes” due to the absence of reproducible reasoning underpinning their decision‐making processes.22 This lack of transparency may make it difficult for surgeons to have confidence in AI‐assisted recommendations, particularly in circumstances where there are major differences between specialist surgeon opinion and AI. Furthermore, surgeons may simultaneously encounter difficulty in explaining these recommendations to patients, given that it may only be possible to explain the inputs and outputs of the algorithms rather than the internal processes.22 Australian and New Zealand surgical staff should become familiar with interpreting and transparently communicating the inputs and outputs of AI tools, encourage patients to ask questions and express concerns, and provide information to aid patient‐friendly explanations. In addition, this “black box” intermediary may also make informed consent more challenging, although similar concessions are made when prescribing efficacious medications with mechanisms of action that are not entirely understood.23 Maintaining the shared decision‐making process while using AI algorithms may similarly prove challenging, as they often do not currently account for patient values.24 Surgical staff should be vigilant when using AI‐generated recommendations and ensure alignment with patient interests while also informing patients about factors considered and excluded by AI decision‐support tools.

Surgical decision making is complex, which adds challenges to liability and accountability when AI is used by surgical services.24 Current malpractice guidelines are unlikely to adequately consider the complexities introduced by AI, making it difficult to determine responsibility and assign blame where human and AI factors both contribute to adverse surgical outcomes.25 Surgical staff should refrain from substituting their clinical expertise with AI‐based recommendations, instead using AI as an adjunct tool to increase the quality of their care.26 Given the known effect of automation bias,27 it should be acknowledged that this suggestion may not be feasible in all surgical scenarios. When using AI tools, surgical staff should comprehensively document their rationale for all clinical decision making, particularly any deviations from AI recommendations. Similarly, it is imperative that surgical staff comprehend the ethical implications for patient privacy and confidentiality when AI is integrated within their service. As clinical AI systems are likely to require sensitive patient data, surgical staff must adequately inform patients, including risks such as misuse or data breaches. Patients must be educated about the possibility that certain AI systems may use these inputs for continuous model training or retain their information for future reference.28 Surgical staff must also offer opt‐out processes where feasible. Ultimately, clear policies must be established for the AI handling of sensitive patient information within the surgical systems of Australia and New Zealand.

Future directions and recommendations

The surgical services of Australia and New Zealand currently have an opportunity to become international leaders in the safe, reliable and effective use of AI. Recommendations to seize this opportunity and overcome relevant challenges are provided in the Box. An evidence‐based approach must be maintained, and the significant ethical concerns always addressed. As with any new surgical technology, there must be careful critical appraisal, regulation and post‐implementation monitoring and audit. It is important to emphasise that AI is an assistive technology to help, not replace, surgical staff and patients. Efforts should be made to educate surgical patients and staff in Australia and New Zealand on the use of AI, including its benefits and limitations. There is the need for further guideline statements to provide frameworks specifically relevant to the use of AI by surgical services across Australia and New Zealand, and for broad collaboration between these services during the widespread uptake of AI to ensure safety at a national scale.