Heartbeat: Recovery and restoration – when IT updates don’t go to plan

February 17, 2022

IT – it’s one of those services that we all rely on. We switch on our computers and get on with our jobs, paying little or no attention to what is keeping our data safe and secure – but what happens when things don’t go to plan?

Recently, our IT systems suffered a serious outage which affected over 20 of our IT systems. Many of the systems were recovered within a week and those that required additional testing were recovered the following week. However, some systems are not yet fully recovered including Medisoft, which is the clinical system used in ophthalmology, and the network file shares.

Work continues to recover this information and teams affected have had to work incredibly hard, and in different ways, to continue to provide safe patient care. The Trust’s business continuity plans were enacted quickly as soon as the IT systems became affected. These plans are in place to make sure that we are able to carry on providing the required services, in the event of a serious incident, such as loss of IT systems.

Sharing his insight into the incident, Chief Informatics Officer, Martin Sadler said: “This was an incredibly unfortunate incident, one that should never have happened and was triggered by one of our suppliers pushing out an urgent security update which they then retracted. This action led to a large number of servers becoming unresponsive when we followed their advice and removed the update.

“As a Trust, we have put a lot of time and effort in to building a new infrastructure of scalable and flexible virtual servers. This means instead of separate computers for each server, we have high performance virtual servers that are able to share their resources and allow us to manage our data safely. This outage effectively took one of our virtual servers out of action and affected a range of services.

“Thankfully departments were quickly able to institute their business continuity processes as our IT teams got to work resolving the issue.

Working round the clock and with the support of an international data recovery team we were able to quickly bring many of the affected systems back online.

“Unfortunately, some systems such as the electronic patient record in BMEC were significantly affected and work continues to recover these.

“I would like to take this opportunity to thank colleagues across the Trust who were affected by this outage and who have had to work through business continuity processes on top of the COVID-19 surge to continue to care for patients.

“We are currently conducting an internal review of the impact and response to this outage in a bid to improve our processes. We will also be working with external partners to complete an independent review into the cause of the outage and to review our processes to ensure it never happens again.”

If you have ongoing issues following the recent service outage, please ensure you report these to the IT service desk on 0121 507 4050.