Canadian security agents compiled dossiers on former prime ministers John Diefenbaker and Lester Pearson but the secret files were destroyed in the late 1980s, newly declassified records show.

The Canadian Security Intelligence Service, which inherited the highly sensitive files from the RCMP's security branch, says they were discarded to "respect the privacy rights" of the leaders, both of whom died in the 1970s.

John Diefenbaker, right, led the Progressive Conservatives to power in 1957. Lester Pearson successfully ran against Diefenbaker in 1963, forming a minority Liberal government. (Canadian Press)

Two intelligence historians dismissed the CSIS explanation as ridiculous.

"How could destroying these files protect Pearson and Dief's privacy when they were already dead?" asked Steve Hewitt, a senior lecturer in the Department of American and Canadian Studies at the University of Birmingham in England.

"The notion they were destroyed for privacy reasons is just nonsensical," added Wesley Wark, a University of Toronto professor who once prepared a history of Canada's intelligence community for the government.

The confidential files on the two prime ministers — bitter rivals in the 1960s — were among hundreds the RCMP security branch amassed on politicians, senior public servants, judges and other "high profile" persons during the Cold War under what was known as the VIP Program.

Some RCMP files — including extensive ones on Quebec premier Rene Levesque and NDP leaders David Lewis and Tommy Douglas — wound up in the national Library and Archives.

Others were destroyed, including dossiers on Supreme Court Justice Gerald Fauteux, Saskatchewan politician Hazen Argue and federal public servant Arthur Plumptre, indicate the records obtained by The Canadian Press under the Access to Information Act.

Still other files, deemed to have ongoing value, were returned to active intelligence holdings.

CSIS, created in 1984, wrestled with the fate of the top secret files once held by the RCMP Security Service, disbanded following a series of scandals.

In a bid to uncover subversives, RCMP spies monitored a wide variety of groups and individuals, from academics and clergy to peace groups and environmentalists.

In 1988, James Kelleher, then federal minister responsible for CSIS, directed the spy service to comb through the mountain of resulting files.

Of 495,340 files reviewed, 438,218 — or more than 88 per cent — were destroyed. Another 28,820 were archived and 28,302 retained.

Among these were 668 VIP Program files on politicians and bureaucrats at all three levels of government, Hewitt revealed in his 2002 book Spying 101: The RCMP's Secret Activities at Canadian Universities, 1917-1997.

Most of those names remain secret. Security files on individuals are accessible from Library and Archives Canada only 20 years after the person's death, and seeking them out requires a certain amount of guesswork and luck given that many were shredded.

Diefenbaker, a feisty Prairie politician, led the Progressive Conservatives to power in 1957. Pearson, who initially made his mark as a Liberal foreign minister, successfully ran against Diefenbaker in 1963, forming a minority government.

Registers among the newly released records show the files on Diefenbaker and Pearson were reviewed by CSIS in the fall of 1988.

While the fact there was an RCMP file on Pearson recently became public, its ultimate fate was not clear. The 1988 register indicates it consisted of two volumes, including one that contained both documents and photos.

A handwritten note recommending destruction of the Pearson dossier says, "There's nothing significant on file that isn't collected in some published newspaper account."

It has long been known the U.S. Federal Bureau of Investigation, which worked closely with the RCMP, had a lengthy file on Pearson, full of Cold War innuendo and hearsay but no evidence of Communist sympathies.

The RCMP's Diefenbaker file remains even more of a mystery, with a simple notation in the 1988 log saying it had "no archival value."

In each case, the dates covered by the files have been blanked out.

"The Pearson and Diefenbaker files were put together by the RCMP Security Service, presumably during the height of the Cold War," CSIS spokeswoman Tahera Mufti said in a written response to questions.

"That was a time when, as some historians argue, the security community occasionally saw threats that — hindsight being 20-20 — might seem exaggerated to us today."

She noted that it was partly in response to such behaviour that the federal government decided to divorce security intelligence from law enforcement, leading to the creation of CSIS, a civilian agency.

'It was determined that the Pearson and Diefenbaker files did not meet the new standards as set out in the CSIS Act. In other words, there was nothing in these files of sufficient relevance to national security to justify its retention.'—CSIS spokeswoman Tahera Mufti

In directing CSIS to review the old RCMP files, Kelleher wanted clarity on whether they would meet the threshold of practices and authorities outlined in the CSIS Act, the legislation governing the new agency, she added.

"It was determined that the Pearson and Diefenbaker files did not meet the new standards as set out in the CSIS Act. In other words, there was nothing in these files of sufficient relevance to national security to justify its retention," Mufti said.

"Judging from the correspondence of the time, it seems that the exercise reflected an understandable desire to respect the privacy rights of individuals such as Mr. Pearson and Mr. Diefenbaker. Accordingly, the records indicate that the files were destroyed in early 1989, not long after CSIS received the ministerial direction."

The CSIS records provide insight as to why some records were sent to the archives.

An October 1988 note on whether to preserve the file on Lewis points out that "the very early volumes are of particular interest" as they include some early speeches, notably one at Ottawa's Chateau Laurier hotel in 1939.

The decision to destroy the Pearson and Diefenbaker material seems calculated to "avoid the embarrassment that would accompany the revelations around such files," said Hewitt.

"They did preserve other files of historical value that seemed to have more substantive material, but any material held regarding a prime minister of Canada is of potential historical significance."

Wark said the file review was done in a "wrongheaded fashion" — a problem that persists today because CSIS has become a custodian of historical records.

"They're not trained historians and they're not trained archivists. They're records managers," he said.

The newly released memos suggest "very minimal" interplay between CSIS and the national archives in reviewing the RCMP files, Wark said.

"No one seems to have thought that maybe it would be a good idea to get an expert view on this," he said.

"We just don't know what was in those files. And we're left wondering whether the right judgment was made. And I think the odds on the right judgment having been made at the time by people who weren't trained to make that judgment is pretty slim."

Wark said a new partnership should be struck between federal agencies, Library and Archives Canada and an external historical advisory committee to ensure the best decisions are made in future.