Prime Minister Justin Trudeau is on the receiving end of three formal complaints filed with the federal commissioner of official languages after speaking only French despite English questions at a town hall meeting Tuesday night in Sherbrooke, Que.

Nelson Kalil from the Office of the Commissioner of Official Languages of Canada confirmed that three complaints had been received as of Wednesday evening. 

Trudeau said earlier in the day that in retrospect, maybe he should have answered in English when asked questions in that language at the Sherbrooke meeting.

The prime minister at first defended his decision to respond to questions only in French, including one about anglophone access to mental health services, when asked about it at a news conference this morning.

"I will always defend official bilingualism. I believe deeply in it, but I understand the importance of speaking French and defending the French language in Quebec," he said.

"That is something I will continue to do while respecting minority language rights across the country."

Trudeau added that he was "surprised" to get so many questions in English in Quebec's Eastern Townships.

He also pointed out that he answered a French question in English at a recent town hall meeting in Peterborough, Ont.

But Trudeau switched gears a little later, when a reporter asked another question on the topic.

He then said he maybe could have answered partly in English and partly in French and that, on reflection, it would have been a good thing to do.

'Very insulting' to anglophone minority

Trudeau's decision to speak only in French struck a nerve with members of the anglophone minority in Quebec's Eastern Townships, which now makes up roughly six per cent of the population in the region east of Montreal. 

Judy Ross, who asked the question about mental health services, said she found Trudeau's response "very insulting."

"I really felt disrespected," she told CBC News.

Watch the exchange:

Trudeau answers English question in French because 'we're in Quebec'0:52

Gerald Cutting, president of the Townshippers' Association, which represents the local English-language community, said he was shocked by Trudeau's stance.

"It was as if someone had just walked up to me and hit me in the stomach. I lost my breath," said Cutting, who was at the event.

"During the whole time that the prime minister was in the meeting, he did not say one word of English, not even in his opening remarks."

Cutting said French-speaking people around him also appeared puzzled by Trudeau's actions.

"People afterwards said, 'Would he have ever thought of going into Manitoba and answering a French Manitoban who asked a question in French, and say, well, we're in Manitoba and therefore we should speak in English?'''

The head of the Quebec Community Groups Network, which represents English community organizations in the province, also criticized Trudeau.

"Even staunch defenders of the French language like René Lévesque would never have made such a misstep," president James Shea said in a statement.

Politicians take jabs, as Conservatives struggle

Kevin O'Leary at the Manning Conference

Kevin O'Leary entered the crowded Conservative leadership race on Wednesday, a day after the French-language debate. (Adrian Wyld/Canadian Press)

Politicians also took shots at Trudeau for his French-only position at the event.

Jason Kenney, a former federal Conservative cabinet minister under Stephen Harper, was one of the first to slam Trudeau. 

He wrote on Twitter that it was an example of the "sort of arrogance that undermines support for bilingualism."

The town hall event came on the same day as the French-language Conservative leadership debate in Quebec City, during which many of the candidates struggled to deliver their messages.

Businessman Kevin O'Leary, a unilingual anglophone from Montreal, entered the crowded Conservative leadership race Wednesday, the morning after the debate.

Anthony Housefather, Liberal MP for the Montreal riding of Mount Royal, said Trudeau's performance at the town hall should be seen as an anomaly, considering his "long history of standing up for official bilingualism."

"I contacted his office to make sure they were aware that he needed to rethink the way he handled the situation yesterday, and I was very pleased that this morning that he said that in retrospect he would've done it differently," Housefather said.

With files from Melissa Fundira, Steve Rukavina and The Canadian Press