Politics

Correctional Service to hire its own IT after being 'drastically' hit by Shared Services's data centre outage

A data-centre outage last October hit Correctional Service Canada (CSC) hard for 20 hours, wiping out email and BlackBerry service. CSC sent a blistering letter of complaint, obtained by CBC News, to Shared Services Canada, saying it planned to hire its own IT professionals because of continuing poor service.

Botched fire alarm test caused 2017 outage, damaging email and systems including one monitoring offenders

Shared Services Canada is the department responsible for the federal government's IT services, including its data centres. The agency has come under fire for an outage on Oct. 3, 2017, that crippled the work of Correctional Service Canada. (Shared Services Canada)

Canada's prison service is so fed up with Ottawa's IT agency that it wants to hire its own computer specialists — which would reverse a 2011 policy that required departments to surrender all their IT work to Shared Services Canada.

The frustrations of Correctional Service Canada (CSC) came to a head last October, when a vital data centre went down for almost 20 hours, cutting off email and more than 80 other key systems.

The Oct. 3, 2017, incident was triggered by a botched fire-alarm test at 6:45 p.m., in which power was cut for three hours because of human error by a third party.

The outage damaged BlackBerry and email systems. So-called "mission-critical" systems were slow to recover. Technicians finally got everything working again by 3 p.m. the following day.

Former Correctional Service Canada commissioner Don Head, left, with Public Safety Minister Ralph Goodale. In a blistering letter weeks before he announced his retirement, Head complained to Shared Services Canada about a data centre outage last October. (Correctional Service Canada)

The then-head of the correctional service wrote a blistering letter to Shared Services Canada (SSC), accusing the agency of being asleep at the switch.

"SSC's inability to identify that CSC's … Data Centre was unavailable for an extended period of time … drastically impacted our ability to effectively manage the issue in a timely fashion," Don Head wrote to Ron Parker, Shared Services president on Oct. 24, 2017.

"Neither notices nor alarms were triggered as a result of this shut down, causing staff to struggle in identifying the problem and formulating a recovery plan."

Public Safety Minister Ralph Goodale spoke to reporters in Saskatoon 1:24

Head, who retired from public service Feb. 3, noted that the hardware that failed was "very old," never having been updated.

"This type of situation has previously been raised with SSC," he said. "For example, past problems relating to a lack of available storage space on CSC's servers, and monitoring of server uptime, could have been more rapidly remediated or avoided with better monitoring by SSC."

Numerous complaints about Shared Services

The letter said the prison service wants to "internalize" some responsibilities that had been handed over to Shared Services when the agency was created in 2011 to take over all federal IT work.

A section of the 2017 federal budget, which came into force in June last year, allows departments to repatriate some IT responsibilities — a measure the Liberal government allowed after getting numerous complaints about Shared Services's poor performance.

CBC News obtained the letter and related documents under the Access to Information Act.

Spokespersons for both agencies say talks are continuing over the transfer of IT work, including who will pay.

Neither notices nor alarms were triggered as a result of this shut down … - Don Head, former commissioner of Correctional Service Canada

CSC's Megan Hooper said managers want to take back control of monitoring, network planning and equipment support, while leaving systems platforms, such as server hardware, under the control of Shared Services.

In a joint statement to CBC, Shared Services Canada President Ron Parker and Anne Kelly, interim commissioner of Correctional Service Canada, said that "CSC's IT services such as email and data centres will continue to be supported by [Shared Services].

"SSC takes its mandate of data centre management and oversight very seriously."

Among the 82 systems knocked out last October, Hooper said, were those used to manage offenders, provide security intelligence and other information to police, and to handle accounting and human resources. All departmental work was either delayed or carried out manually. Landlines were used in place of BlackBerrys and email.

"In the past, unplanned data centre outages have been due to events such as severed network links, air conditioning failure or electrical system failure," she said.

Other agencies and their outages

Neither Shared Services nor CSC would reveal the location of the data centre, used only for prison-service systems, for "security reasons."

Charles Anido, spokesperson for Shared Services, confirmed that talks are underway with CSC to transfer some IT responsibilities. To date, no department has yet taken advantage of the transfer option that was first offered last summer, he said.

"CSC and Shared Services Canada are working collaboratively to move the agency's applications to a modern enterprise data centre that will offer state of the art facilities with multiple redundancies to avoid the service issues encountered in legacy data centres," Anido said.

"There is currently no scheduled closure for this [CSC] data centre."

Ron Parker, president of Shared Services Canada, Ottawa's beleagured IT agency. (Shared Services Canada)

Other federal departments have been hit with outages linked to Shared Services, including Statistics Canada, Canada Border Services Agency and the RCMP.

Shared Services was also slammed by Canada's auditor general in 2016 for poor service and performance, and in 2017 by U.S.-based consultancy firm Gartner Inc. hired for $1.35 million by Ottawa.

The agency was created in 2011 to consolidate IT spread across 43 departments and agencies, including reducing some 500 data centres to seven and creating a single email system — projects still far from complete.

Follow @DeanBeeby on Twitter

About the Author

Dean Beeby

Senior reporter, Parliamentary Bureau

Dean Beeby is a CBC journalist, author and specialist in freedom-of-information laws. Follow him on Twitter: @DeanBeeby

Comments

To encourage thoughtful and respectful conversations, first and last names will appear with each submission to CBC/Radio-Canada's online communities (except in children and youth-oriented communities). Pseudonyms will no longer be permitted.

By submitting a comment, you accept that CBC has the right to reproduce and publish that comment in whole or in part, in any manner CBC chooses. Please note that CBC does not endorse the opinions expressed in comments. Comments on this story are moderated according to our Submission Guidelines. Comments are welcome while open. We reserve the right to close comments at any time.