More than 80 lawyers have written an open letter to Citizenship and Immigration Minister Jason Kenney challenging his assertion that he was not involved with the decision to grant Conrad Black a permit to live in Canada after the former media mogul served jail time in the U.S.

In an open letter to Kenney, the immigration lawyers say it is "not credible" that Black would have been granted a temporary resident permit "without any input" from Kenney himself.

Prime Minister Stephen Harper defended Kenney calling the allegations against his cabinet minister "spurious" when questioned by reporters following an announcement on cleanup funding for Lake Winnipeg in Gimli, Man., on Thursday.

"Kenney took every step to ensure that this matter was handled independently by public servants," said Harper.

The prime minister added that it was "not in the government's interest to intervene in this matter, in any way, shape or form."

But the lawyers believe Kenney must have had some involvement in the controversial decision to grant the temporary permit to Black, who renounced his Canadian citizenship and served jail time in the U.S. for obstruction of justice.

Echoing the prime minister's remarks, Kenney's office maintains "neither the minister of immigration nor his staff were involved in processing this file."

"In fact, Minister Kenney directed his department multiple times to specifically exclude him and his staff from any deliberations regarding this application," said Ana Curic, director of communications for the immigration minister, in a statement to CBC News.

"This decision was made by independent public servants, based on Canadian law," she said.

Kenney's office 'made a mistake'

One of the lawyers who first expressed doubt at the minister's lack of involvement in the decision to grant Black a temporary resident permit, saw a complaint filed against him.

Lawyer Guidy Mamann saw the immigration minister's office file a formal complaint against him with the Law Society of Upper Canada.

The complaint filed by Kasra Nejatian, the director of strategic planning for the minister, was dismissed in July by the law society which said Mamann had done nothing wrong.

In an interview with CBC News Network's Power & Politics, Tom Flanagan, a professor at the University of Calgary and former adviser to Harper said Kenney's office "made a mistake" by filing a formal complaint against Mamann.

Although Mamann hasn't provided any evidence to support his claim, Flanagan told guest host Hannah Thibedeau "you can't start complaining every time somebody does that. So Jason [Kenney] has brought this on himself."

Mamann said he saw the complaint as an attempt by the minister to have a chilling effect not just on him but also on other lawyers.

The complaint was intended "to shut me up and people who think like me," Mamann said.

Micheal Vonn, a lawyer and policy director for the B.C. Civil Liberties Association, thinks this is part of a wider pattern by the federal government to muzzle its critics.

In an interview with CBC News, she cited the instance when Public Safety Minister Vic Toews depicted anyone who was against the government's surveillance bill as "child pornographers" to the attacks against senior diplomat Richard Colvin when he spoke out about the transfer of Afghan detainees by Canadian troops.

"There is something going on here that should be deeply concerning to Canadians," she told CBC News.

The New Democrats also weighed in on the matter.

In a press release issued on Thursday, the NDP said, "When confronted with criticism about using personal attacks to silence critics, the Conservative reaction is to double-down and launch new, more personal attacks."

In open defiance, the lawyers are challenging Kenney to take them before the law society for agreeing with Mamann. "If you believe that our statement violates the Law Society of Upper Canada Rules please feel free to report us to the Law Society."

"We find the attempt by you and your officials to muzzle freedom of expression to be reprehensible. We will not succumb," the lawyers wrote.

Mamann said he isn't opposed to the fact that Black was granted a permit per se. He is opposed to what he claims was ministerial intervention on the part of the minister in Black's case, an assertion that Kenney is denying.

"It's inconceivable ... that the one permit that is going to be most scrutinized is not going to be subject to some sort of ministerial review," said Mamann.

In June, records obtained by CBC News Network's Power & Politics under the federal Access to Information Act suggested there was no political interference to fast-track or approve Black's application.

Some of the same lawyers who have signed this open letter are also involved in a legal challenge trying to stop the Conservative government from deleting a massive backlog of 280,000 immigration applications.