Usability challenges in surgical simulator training

Surgical virtual reality simulators have been taken into use in order to improve surgical skills training. Emergence of simulators increases the need for research and knowledge related to usability of medical simulators. In this study usability of laparoscopic surgical simulator was researched experimentally through combined analysis. Data was gathered with heuristic evaluation, questionnaires, and interviews as well as recorded simulator parameters. Results suggest that the surgical simulator could be more efficient learning and training tool if usability issues such as support and error prevention were reconsidered in more detail. There also seem to be grounds for connecting user support into structured simulator training program.


Introduction
Simulator-based training in surgery is evolving to a more common part of resident training. Developing systematic simulator training programs alongside traditional training in hospitals is a new and progressive way to intensify surgical resident training and surgical skills learning. Structured simulator training program developed in Central Finland Central Hospital for intensifying surgeons' skills learning showed that more study is required concerning problems of simulator use. Also introduction of simulators into surgical training programs has not been yet studied. This information is needed in order to provide adequate training support as well as developing more efficient training programs for surgeons.
Usability is usually defined as feasibility [1]; that is how well and easily a person can use functions of a product to perform certain tasks. According to Nielsen [16] usability consist of system's learnability and efficiency of use, memorability, rate of errors and subjective satisfaction of use.
Surgical VR simulation was taken into use in order to improve training efficiency and to reduce time spent in training of basic skills with patients. Term 'simulator' as used in medicine usually refers to a device that presents a patient or part of a patient [5]. Skills training in simulation laboratories have been shown to improve operating room performance [7].
Swift technological evolution of medical simulators has focused on highest possible technical standards, simulataneously discarding end-users needs [21]. Usability evaluation of new virtual reality systems requires also new approaches and perspectives [1]. As a relatively new learning technology, medical simulators need to be studied more in order to resolve user problems and even more importantly, find ways to avoid these problems in future [11].
This experimental study is based on a pilot research project which suggested that there are unsolved problems concerning the learnability and usage of the simulator. We see that the educational aspects of using the simulator as training tool are not considered sufficiently. Simulator user's learnability needs to be analyzed holistically, in order to improve simulators as educational tool and gain the best possible benefit from training. Educational value of the simulator is also dependent on the comfortableness of training, ease of use and it's ability to motivate user to train. Validation studies [14] that have been executed on surgical simulators give only partial view of functionality and effectiveness of simulators. There is also a need for studies on the actual usability of surgical simulators. Nowadays usability is seen as an increasingly important aspect when medical technology is purchased at hospitals. However, there seems to be need for suitable evaluation methods [13,3]. We suggest more detailed usability analysis of the simulator with the performance data gathered from the simulator training combined with experiences from ongoing training program. The purpose of this paper is to describe usability problems and challenges related to the surgical simulator. Paper also discusses the requirements and possibilities to enhance skills learning and training in hospital context. The paper is organized as follows: Section 2 outlines the background issues concerning surgical simulator training and usability. Section 3 describes the study and analysis of the data gathered. Section 4 presents the results. Section 5 discusses the results of the study. In last chapter, concluding remarks are presented 2. Surgical training and usability

Video assisted surgical skills training
The increasing use of video assisted minimally invasive surgery (MIS) has created a need for training simulators which are safe alternative for surgical skills training. Especially video assisted abdominal surgery, laparoscopy, is complex technique and the development of laparoscopic skills need additional teaching and training when compared to traditional open surgical techniques [23,18]. Laparoscopic surgical operation is performed through small incision in the abdominal cavity. The instruments being used are thirty centimetres long and more complex than instruments used in open surgical techniques. For example, physical ergonomics of laparoscopy are known to be challenging due to the awkward position of operating surgeon.

Surgical simulators
Simulators' advantages for surgical training have been studied in the past years. It has been recommended that simulator training should be used in early stage of training, when students are highly motivated and have not yet developed routine working methods [4,22]. Also, surgical complications seem to occur most frequently during the first 10 procedures [10].
There is a common agreement that simulator training should be part of the laparoscopic surgical training and further in curriculum [8]. This study focuses on a simulator that is designed for training laparoscopic techniques. Laparoscopic simulators offer highly flexible and controlled learning situations with various tasks and problems to be solved without external disturbances such as stress and hurry. With the aid of simulator residents are able to practice repeatedly in varying conditions without the risk of harming patients. Laparoscopic simulator training is expected to build skills such as depth perception, eye-hand coordination and instrument control as well as adapting the trainee to the decreased degrees of freedom when operating through small incisions. Further, simulators are used to introduce new equipment and instruments to novice surgeons. [19] Simulators offer possibilities to assess the readiness of resident surgeon to proceed to real patient surgery [6]. Also the experienced physicians from various disciplines can practice procedures such as operations they have not recently performed [2].

Research design
Training program and data collecting was organized in hospital medical skills learning centre. Simulator training was organized as structured training program for surgical residents supervised by specialist surgeons. Participants were given short instructions (30 minutes) before training. Each task and simulator function was introduced by specialist. Next we present research questions and data gathering. Thereafter, we present analysis of data obtained with expert analysis, questionnaires, facilitator comments and simulator parameters.
Research questions: 1. What are the usability problems and challenges of laparoscopic surgical simulator? 2. How do these usability problems and challenges affect to the use of simulator in training?
3. What usability requirements surgical simulators need to fulfil in the hospital context, in order to support the training of surgical skills?
Various data collection methods were used in this study (see table 1.). Questionnaires were used to collect surgical residents' experiences and opinions during the training program. Simulator parameters were collected from each trainee's performance in order to find out the quality of performance and possible problems during it. Participants were instructed to use the simulator and practice skills on five different tasks selected by surgical specialist. All exercises simulated laparoscopic instrument handling in three-dimensional space. The purpose of the training was to familiarize participants to videoassisted surgical equipments and technique. Usability of the simulator was evaluated with the aid of combined methods. Heuristic evaluation was conducted by the usability expert In order to create suitable data gathering method for simulator analysis, heuristics used were gathered from following heuristics or design principles Jakob Nielsen's [17], Ben Shneiderman's [20], IBM's [9] and University of Jyväskylä Käytech project's. Reported heuristics were selected based on their applicability in the simulator evaluation. Usability problems were counted and described in detail. In addition, possible consequences of problems as well as solutions for each problem were evaluated.
Simulator training facilitator was present during usability expert evaluation. Facilitator's role was to offer user support and instruction related to the simulator use. After the heuristic evaluation the facilitator's observations on the learnability and usage of the simulator were recorded during open interview session. Facilitator's comments on user challenges and problems were recorded along with heuristics.
Problems and challenges of the surgical residents' were gathered and further analyzed. The number of trainees who experienced difficulties and specific problems were recorded. Simulator parameters, such as performance, accuracy of movement and the rate of errors, were analysed to reveal performance task failures.

Usability problems of the simulator
Heuristic usability analysis highlighted 42 usability problems in total. A summary of these problems is presented in table 2.
Most usability problems concerned aesthetics and visual design. For example user interface colors, inadequately marked links or icons and unpractical placing of buttons were evaluated as problematic issues. Problems concerning consistency and standards were caused by the fact that simulator user interface was not designed following the standards and conventions of user interface design. The simulator was assessed to offer minimal level and hard to detect guidance. The problems related to system feedback have impact especially on user satisfaction. The simulator does not offer systematic information on errors or when the task is finished. Problems of error prevention and recovery are mostly related to the delicate instruments and calibration which needs extra carefulness in order to succeed. In the area of authenticity and ergonomics problems concerned rigidity of instrument use and impossibility to move or lean the monitor or adjust the height of the simulator. However simulators mobility and feeling of authenticity were evaluated as positive features.
A possibility to use shortcuts would speed up navigation in the monitor user interface. Proper search functions as well as more interactive video functions would also help user and prevent overloading user's memory. Efficiency was reduced by slow loading operations. Positive characteristic was possibility to customize training for each individual user needs. In user interface medical language is mostly used which is familiar to users. However outside the tasks there is also technical language which might be unfamiliar to users.

Usability challenges during training
Problems that took place during the training were collected from participants with the aid of questionnaires as well as recorded simulator parameters. (Table 3). Parameters showed that two out of fifteen users had problems in at least one of five task types. These failures were caused, for example, by calibration problems. Only two users succeeded to perform tasks without any problems visible in parameters. However, nine out of ten users reported in questionnaires having problems in at least one of the tasks. At least two users reported problems in each task types. The problems mentioned in questionnaires and facilitator's interview were combined to the results of the heuristic usability analysis in order to find coherences between these results ( Table  4). The possible consequences of the detected problems are also shown in Table 4. Only few trainees reported a need for instructions even though facilitator and senior surgeons needed to instruct trainees systematically. The guidance offered by the simulator was reported insufficient only by one user from total of 11 users. This is contradictory to the facilitator's observations of users needing hands-on instructions especially in the beginning of their training and after a longer break in training. The facilitator explained that the user training was organized by the manufacturer when simulator was introduced in the hospital. Thereafter user support was offered from the manufacturer mainly through phone or e-mail. In addition user manual in Finnish language was printed and placed alongside the simulator.

Discussion
The study was conducted in order to gain more information concerning usability of surgical simulators, together with increasing attention towards usability aspects on medical technology. Heuristic usability evaluations have been recommended to be performed by a minimum of three evaluators which results finding about 60-75 % of problems [17]. As compensation to the use of only two evaluators other evaluation methods were used. These include such as questionnaires, interview and analysing training performance parameters. Heuristic evaluations have been criticized to be misleading due to their introspective nature and expert evaluators' possible difficulties to understand end users. This criticism encouraged us to use simulator training facilitator's knowledge of end users' ways of working to supplement analysis of usability expert. It is stated that the use of experts of different areas raises the amount of detected problems [15].
The simulator has succeeded on prior validation tests and studies and seemed to also satisfy the users need for training skills outside operating room. Simulator's value for surgical skills training is undeniable. However, due to many usability problems, the training was not as fluent as expected. Many of the heuristic evaluation results are quite ordinary usability problems and should be easily solved.
To compensate user challenges during simulator training, the facilitator was offering hands-on user support and instruction on simulator use. The results show that the user support and the hands on instruction of the simulator use are needed. The support provided by facilitator and specialists was vitally important for trouble-free training.
The simulator user interface offers very little guidance which might be critical if it leads to mislearning. Match between simulator and real world is important aspect of training, however, also motivating users would be useful in this kind of educational simulator. The lack of adjustability is probable to cause muscular pain especially to short and tall users and needs to be improved by for example using platforms. Considering educational simulator use in hospital context, we suggest improvements in order to support surgical skills training and learning. We suggest following requirements for surgical virtual reality simulator: • occasionally it is needed to move the instrument around the screen to get it calibrated, but simulator measures extra movements of the instrument as error in performance parameters. The defect is that the performance parameters are not separating the equipment problems from user's lack of skill.
Actions for improving usability could concentrate on developing more detailed training instructions or help and tutoring systems. The best results, however, could be achieved by developing the quality of user interface ease of use. For example experiences from ship engine room simulator study showed that, many simulator users reported that the main criteria for usability should be simulator's authenticity [12]. Similarly in our study, the satisfaction of simulator authenticity was high which resulted to high motivation in simulator training and less complains on other problems. The good usability means ease of memorizing and use even after training breaks. Intuitive use should be possible during the hectic hospital work environment within lunch breaks or between patient treatments.
We argue that the simulator should not be used without proper backup. Without specialist guidance the risks of learning erroneous working methods increases. Also without proper user support the simulator's functional problems or missing instructions may frustrate users or induce breaking down of the simulator. In addition specialists hardly have time to act as user support or facilitator.

Conclusions
There are challenges and problems concerning usability of the surgical laparoscopic simulator. Based on the results gathered from heuristic analysis and error rates of residents as well as reported problems by users and the facilitator, there is a need for developing userfriendliness of the simulator. We argue that the best laparoscopy simulator training place is in hospital where the residents begin to operate laparoscopies at first time. We see that there are possibilities to increase the efficiency of simulator training and learning as well as user satisfaction and motivation.
Maintaining and offering user support is needed, as empirical experiences have shown,. However, the deficiencies in simulator usability can be compensated with facilitating, tutoring and mentoring. Further experiments are also needed concerning simulator use in training and since there are other laparoscopic simulators these results do not necessarily apply to all of them before further studied.