Conservative MP John Williamson is facing heavy criticism after comments he made about the Temporary Foreign Worker program over the weekend where he referred to "whities" and "brown people."

Williamson told delegates at a conference in Ottawa that it makes no sense to pay "whities" to stay home while companies bring in "brown people" as temporary foreign workers.

By Saturday, the New Brunswick Southwest MP had apologized "unreservedly" on Twitter for his "offensive and inappropriate language."

"I believe different parts of Canada have different labour needs.  With respect to my region, I believe employers in my district need to work to fill job vacancies by prioritizing Canadians for available jobs," he said.

Williamson declined further comment to CBC News on Monday, saying, "I don't think there is any explanation for the words I used."

John Williamson

New Brunswick Southwest Conservative MP John Williamson has apologized after referring to "whities" and "brown people" in response to a question about the Temporary Foreign Workers program on Saturday at a conference. (CBC)

"This is the worst mistake I've made as an elected member and also over my 20 years of writing and commenting on public policy. I am deeply disappointed in myself," he said in an emailed statement.

Tobique-Mactaquac Conservative MP Mike Allen told reporters that it was important that Williamson apologized for the remarks.

"It’s always good when someone apologizes for saying something dumb," he said.

But the criticism has not halted, including from his fellow Tory MPs.

Tory MP Deepak Obhrai, the parliamentary secretary for foreign affairs and international human rights, also had tough words for his colleague on Twitter.

He said he was "very disturbed" by the comments.

"Foolish statement damages all of us. Years of hard work down the drain," he tweeted.

Williamson is a former director of communications for Prime Minister Stephen Harper.

The MP said in a statement that the comments are the biggest mistake of his political career.

The comments were resonating in his riding on Monday.

Esthela Pyett, the president of the New Brunswick Filipino Association, said she was "hurt" by Williamson’s comments.

Pyett said the MP has helped many Filipino residents in the riding, but he clearly chose the wrong words at the conference.

"I think that's what he meant by the browns are coming here to work and they really work hard. And at the same time they pay taxes," she said. "Not all temporary foreign workers are brown, by the way."

'Seeing red'

Williamson was also criticized by the riding’s Liberal candidate for the upcoming federal election.

Karen Ludwig said the MP’s language is totally inappropriate.

"When Mr. Williamson believed he was only speaking to Conservatives, he shamefully sought to divide Canadians," she said in a statement. 

"This is completely unacceptable. Our riding needs attention from Ottawa, but not like this."

Constituent Judy James said his colour comments insulted temporary foreign workers and locals. 

"The colour I'm seeing right now is red," she said. "He's basically discriminated against every single New Brunswicker by those statements, every single one. He didn't miss any of us."

James plans to remind other voters of his comments come election time. 

"Because this is a Conservative community. If one person is racist and we support this person, then that means we're all racist. And that is very upsetting to me."

Vern Faulkner, editor of the St. Croix Courier in Williamsons's hometown, said the the riding is so staunchly Conservative that the party could run a ham sandwich and win.

He wondered why they'd risk hanging onto a now controversial politician. 

"The story is going to play out in the urban ridings of Toronto or Vancouver, Montreal, southern Ontario — where the ethnic vote that the Tories have been courting some time and with considerable expense," he said.