teaser
Pharmacy-based robots have great potential, with many perceived benefits. But it is necessary to evaluate the benefits in practice, to optimise systems, identify and reduce risks, and plan development
Bryony Dean Franklin PhD
Director
Centre for Medication
Safety and Service
Quality
Imperial College
Healthcare NHS Trust
Professor
The School of Pharmacy
University of London
London, UK
Acknowledgement
The Centre for Medication Safety and Service Quality is affiliated with the Centre for Patient Safety and Service Quality at Imperial College Healthcare NHS Trust, which is funded by the National Institute of Health Research
Pharmacy-based automated dispensing systems (“robots”) are becoming widespread in UK hospitals. The history of their introduction and the different types of system available have been summarised elsewhere.[1,2] More recently, a paper focusing on their implementation has been published in this journal.[3] This present paper discusses the issues to be considered in their evaluation. We focus on the use of robots in UK hospitals, but many of the points will be equally applicable to other countries.
Why do we need to evaluate?
The perceived benefits of pharmacy-based dispensing robots include a reduction in dispensing errors, a reduction in staff time requirements, improved dispensary workflow, reduced prescription turnaround times and a reduction in the floorspace required for medication storage. While all of these benefits are likely to be achievable, the extent of the benefits will depend on how the system is implemented and used, and how it fits alongside other work practices. As with many other technologies, it cannot be assumed that benefits automatically accrue as soon as the system is switched on. Evaluations are therefore needed to find out the extent to which the anticipated benefits are borne out in practice, to identify areas where further optimisation is needed, to identify and minimise any new risks introduced and to assist in the development of business cases for the automation of additional sites.
Approaches to evaluation
The approach to evaluation will depend on the reasons for the evaluation. Evaluations can range from those focusing on one outcome measure of particular local interest to those that span a comprehensive range. The extent of the evaluation will also depend to some extent on the resources available; data on some outcome measures can be easily obtained using existing data sources,[4] while others will require additional resources to collect new data. Cornford[5] presents an evaluation framework that can be used in designing comprehensive evaluations.
A combination of qualitative and quantitative evaluation can be used to understand the full impact of pharmacy automation,[6,7] as has also been used to study the impact of an electronic prescribing and automated dispensing system.[8,9]
Outcome measures
Table 1 presents a summary of the outcome measures that have been used in previous UK evaluations. Only published papers (in either full or abstract form) are represented; it is recognised that there have also been many in-house evaluations that have not been published.
[[HPE.45]]
Other outcome measures that have not been reported in evaluations published to date, but may be important, include the following:
* Time taken to load the robot.
* Changes in stock control, including wastage of out-of-date stock, stock levels and the frequency of products being out of stock.
* Clinical significance of the dispensing errors that occur both pre- and post-implementation.
* Changes in the time taken to complete an annual stock take, if one is conducted.[4]
* Changes in out-of-hours supply in hospitals with offsite on-call arrangements where offsite pharmacists can remotely access the robot.[4]
* Impact on distribution staff workload.[4]
* Robot breakdowns and downtime.
* Time taken to implement and maintain the robot.
* Changes in ward staff satisfaction.[4]
One large gap in the literature is that there has been no economic evaluation of dispensing robots published to date taking into account the costs of purchase, maintenance and installation assessed against the benefits realised.
Whittlesea et al have designed an “evaluation toolkit” for use in Welsh hospitals and give many practical suggestions for the evaluation of pharmacy-based dispensing robots.[4] The toolkit is designed to assess changes in the number and type of dispensing errors and ward distribution errors, changes in turnaround times, cost savings with respect to stockholding,
ordering efficiency and out-of-hours supply, costs of installing and running the system, changes in workload and workload patterns for both dispensing and distribution, ward managers™ and outpatients™ satisfaction, and attitudes of pharmacy support staff to automation.
Methodological issues
There are several important methodological issues that should be considered when designing an evaluation. First is the study design. Most previous studies have used uncontrolled before-and-after study designs. Franklin et al[6] used a controlled study design. A controlled design is useful to control for changes in workload and national changes in practice that may affect the outcome measures concerned, but is not always practical.
Second is the stage at which the post-implementation data should be collected. In most studies, a period of three[13], six[6,10,11] or 12 months[6] has been allowed to elapse before collecting post-implementation data, to allow any initial teething problems to be resolved and ensure that staff are familiar with the system before it is evaluated. One study was longitudinal, and observations and interviews were conducted throughout the implementation and post-implementation periods.[7] Other authors have not specified when the post-implementation data were collected.[15] Groundry-Smith recommends a period of six months.[1] However, there is no evidence to suggest what the right period should be. Collecting post-implementation data at the same time of the year as the pre-implementation data has theoretical advantages in providing some control for seasonal variation in workload, staffing patterns and level of staff experience. Repeated post-implementation data collection periods, spread over many months, would provide the ideal scenario if resources allow and would lead to an understanding of any settling-in effects. Preimplementation data must also be collected before any implementation changes have been made.
Third is the sample sizes and sampling strategies to be used. To determine an appropriate sample size for quantitative data, some understanding of the current measures is needed, together with the size of difference that the study is designed to identify and the extent of variation in measurements over time. Statistical advice may be needed. Consideration should also be given as to whether or not the data collected will include periods such as weekends and evenings.
The fourth issue is whether the evaluation is designed to study the whole post-implementation system, or just those items that are being dispensed by the robot. There are arguments in favour of both approaches, but if using the latter, the study needs to be designed so that only those types of item that are stored in the robot are studied both pre- and postimplementation. This may be hard to determine before the robot has gone live. Finally, some of the more intangible effects – the impact of breakdowns, the benefits in terms of staff feeling that they are working in a department that is keeping pace with change, the time required to implement and maintain the robot – are hard to quantify and may need qualitative study.
Using and interpreting the results
The results of the evaluation can be used to celebrate local successes and to identify any new problems and how to resolve them. Results of the evaluations conducted to date in the UK indicate that sites have achieved some reduction in dispensing errors, increased pharmacy staff satisfaction, either beneficial or neutral effects on prescription turnaround times, increased throughput and a decrease in floorspace requirements. However, differences in the methods and measures used, together with the limited detail given in some published studies, mean that it can be difficult to summarise and interpret some of the literature in this area. Findings should be shared and published so that others may benefit from the learning points identified, and so that an evidence base for the effects of such systems can be built.
Pharmacy-based dispensing robots have huge potential; evaluation is important to help ensure that the anticipated benefits are realised.
References
1. Goundrey-Smith S. Pharmaceutical J 2008;280:599-602.
2. Swanson D. Hosp Pharmacist 2004;11:66-77.
3. Mounsey A, et al. Hosp Pharm Eur 2009;(43):63-5.
4. Whittlesea C, et al. Hosp Pharmacist 2004;11:283-5.
5. Cornford T, et al. J Manage Sci 1994;22:491-504.
6. Franklin BD, et al. Int J Pharm Pract 2008;16: 47-53
7. Oborn E, et al. Int J Pharm Pract 2008;16: 109-14.
8. Franklin BD, et al. Qual Safety Health Care 2007;16:279-84.
9. Barber N, et al. Qual Safety Health Care 2007; 16:271-8.
10. Coleman B. Hosp Pharmacist 2004;11: 248-51.
11. Fitzpatrick R, et al. Pharmaceutical J 2005; 274:763-5.
12. James KL, et al. Int J Pharmacy Pract 2007;15 Suppl 2:B59.
13. Slee A, et al. Pharmaceutical J 2002; 268:437-8.
14. Whittlesea C, et al. Hosp Pharmacy Eur 2005;March/April:17-8.
15. Whittlesea C, et al. Int J Pharmacy Pract 2004;12:R70.
16. Whittlesea C, et al. Int J Pharmacy Pract 2005;13:R90
