Data centre crash could've been avoided, official says

The $1.6-million data centre crash could have been avoided had risks beenaddressed, says New Brunswick's chief information officer.

"Hindsight is always 20/20. It was avoidable," said Christian Couturier.

"To avoid such a situation you need to be able to measure your risks. You need to know what those risks are and what your mitigation strategies are," he said.

Couturier was commenting on a failure at the Marysville Data Centre which resulted in computer systems going offline across multiple government departments.

An osprey building a nest on a transmission tower knocked out power June 9.

Backup power systems at the data centre failed, which led to the facilitys IT systems hard-crashing, causing IT program outages and data corruption issues for hours and days, and in some departments cases, weeks.

The province released an Ernst and Young report Wednesday which put a $1.6-million price tag on the data centre crash. The report stated the New Brunswick Internal Services Agency was made aware of risks posed by old backup power equipment at the data centre in 2011.

Couturier did not make clear an exact reason warnings went unheeded. He said while there was investment in IT, it may not have always been in the right areas.

"There were decisions and there were lots of IT investments that were made. Were they made as fast as they could? Could they have done a better job of implementing them so that we could mitigate some of the risks of the equipment failure ... I guess hindsight, the answer is yes."

'Never waste a crisis'

Couturier said that the some of the reports recommendations are already being worked on, such as an inventory of all of the programs housed at the data centre as well as a disaster recovery plan neither of which had been done for the facility.

He said the plan is to implement all of the recommendations.

"Theres a saying in my field, you know, [that] you never waste a crisis. So although you want to avoid them at all costs, this one has happened, an incident, and it was not a good one to go through for anybody, the citizens, the providers or the employees. And yes, I am hoping that we can leverage this situation to improve things going forward."

Couturier said a new strategy is being worked on at the Internal Services Agency.

"I think the past is the past. I like to look at what theyre doing now and they'vecertainly showed that they were going to work to improve the situation going forward."

Health files unavailable during outages

The report noted compliance issues in the Department of Health.

According to Bruce MacFarlane, Department of Health spokesman, during the crash, the department was not able to respond to Medicare queries within the normal two-day requirement.

In the review, the department reported 26 "high severity" impacts to its critical services and business. Couturier said in some cases some health databases in hospitals were down for 12 hours at a time.

"You want to give health as efficiently as you can and in a very timely fashion. Nobody likes to be told, 'You go up to the window and, well, we need our systems to come back up for me to tell you about this.'" He said.

The Department of Health has stated that there were no direct impacts to any New Brunswicker'shealth as a result of the outages.

The review was demanded by former premier David Alward after repeated outages and persisting IT issues across government departments, such as inaccessible files and corrupted data.

Its release follows a CBC News investigation in October which revealed more information about the scope of the problems caused by the outage, and which showed government communications officers deliberately tried to limit the information that was released to the public about the outage and recovery efforts.

"Hindsight shows that we could have done things differently," said Couturier.

"We are going to be communicating differently going forward and I think thats very positive, and I think we all welcome that. We want to do that."