A Federal Court judge says Canada's spy agency illegally kept potentially revealing electronic data about people over a 10-year period.

In a hard-hitting ruling made public Thursday, Justice Simon Noel said the Canadian Security Intelligence Service breached its duty to inform the court of its data-collection program, since the information was gathered using judicial warrants.

CSIS should not have retained the information since it was not directly related to threats to the security of Canada, the ruling said.

"Ultimately, the rule of law must prevail," Noel wrote, adding, "without it, the actions of people and institutions cannot be trusted to accurately reflect the purpose they were entrusted to fulfil."

At a press conference in Ottawa Thursday afternoon, CSIS Director Michel Coulombe said the agency accepted the court's decision and has "taken immediate actions to respond."

"I deeply regret the court's serious concerns with respects to meeting our duty of candour, and I commit to continuing my efforts with the deputy minister of justice to address this concern," Coulombe said.

"All associated data collected under warrants was done legally," he said. "The court's key concerns relates to our retention of non-threat-related associated data linked with third-party communication after it was collected." 

"CSIS, in consultation with the Department of Justice, interpreted the CSIS Act to allow for the retention of this subset of associated data. It is now clear that the Federal Court disagrees with this interpretation."

Associated data, or metadata, includes information such as telephone numbers and email addresses but not recordings of conversations or the content of those emails. In this case, it is difficult to determine the precise nature of the metadata involved due to heavy redactions to the 126-page court ruling.

Coulombe said CSIS has halted all access to, and analysis of, the data in order to assess the "operational" impact of the ruling and to determine the way forward.

"We are working closely with the Department of Justice to make sure that we meet our obligations with the court," he added. "The trust of Canadians is essential in the fulfillment of our mandate."

Asked how Canadians can be reassured that CSIS is not still retaining metadata about people that it is not authorized to keep, Robert Frater, chief general counsel for Justice Canada, told reporters the agency now understands its limits.

"We've heard the court loud and clear," said Frater, who attended the press conference with Coulombe. "We are taking steps to improve our practices and we will meet that standard."

Data retention and analysis

CSIS crunched the data beginning in 2006 using a powerful program known as the Operational Data Analysis Centre to produce intelligence that can reveal specific, intimate details about people the spy service investigates, the judge writes.

The ruling said the CSIS data analysis grew out of the spy service's concerns in the early 2000s that the information it collected was not fully utilized and should be processed using modern techniques.

Coulombe said his agency did inform its political masters about the data retention and analysis. 

​"The minister, in fact it is in the court ruling, in 2006 we actually wrote to the minister at the time explaining to him the program and successors were also informed about the program," Coulombe said.

In 2006 the minister of public safety was Conservative MP Stockwell Day.

Appearing on CBC News Network's Power & Politics later Thursday, Day was asked twice by host Rosemary Barton if he was in fact informed about the program and the retention of data.

"Well, any data that's ever collected is kept for some period of time, I would assume, and just as he just admitted, he said: 'I thought we told the court, hmmm, I guess we didn't — so I would suggest, if he is suggesting, or anybody, that they thought they told me something inappropriate was going on, yeah, I would suggest they did not."  

Coulombe and Day on who knew what when1:50

'No excuse'

In his ruling, Noel said that just because CSIS could get more intelligence value out of the metadata with modern techniques, it did not mean the agency's mandate, as approved by Parliament had been expanded.

"The evolution of technology is no excuse to flout or stretch legal parameters. When the information collected does not fall within the legal parameters delimiting the agency's functions and actions, it cannot legally be retained."

In the decision, Noel said he considered ordering the destruction of the associated data collected since 2006, but decided against it because, in part, he did not hear legal arguments on the question.

He suggested it may be time to revisit the CSIS Act of 1984, which is "showing its age" in a technologically advanced world.

"Canada can only gain from weighing such important issues once again," Noel wrote. "Canadian intelligence agencies should be provided the proper tools for their operations but the public must be knowledgeable of some of their ways of operating."

In a statement issued late Thursday Public Safety Minister Ralph Goodale welcomed the ruling, saying the federal government would not be appealing the decision.

"I also take very seriously the explicit finding by Justice Noel that CSIS had failed in its duty to be candid with the court," Goodale said in the statement.

"In matters of security and intelligence, Canadians need to have confidence that all the departments and agencies of the government of Canada are being effective at keeping Canadians safe, and equally, that they are safeguarding our rights and freedoms."

CBC is not responsible for 3rd party content
with files from CBC News