Public servants who got in trouble for wearing "Stephen Harper Hates Me" buttons to work are fighting back.

Several employees at the Canada Revenue Agency who were told to remove the buttons by their managers have filed grievances through their union to fight the order.

The buttons were made by members of the Public Service Alliance of Canada (PSAC), the largest union representing federal public servants, and have been circulating around the country. Some workers wore them at May Day rallies to mark International Workers Day and they were available at PSAC's national convention in Ottawa at the end of April.

Robert Campbell, president of the Union of Taxation Employees (UTE), which falls under the PSAC umbrella, said some CRA employees pinned the buttons to their jackets and wore them to and from work in the following weeks. They complied when advised to remove them.

Management got wind of the buttons and the CRA's collective bargaining, interpretation and recourse section sent a notice to all regional human resources representatives and the message trickled down to CRA offices across the country.

According to the union, the notice said the buttons are "considered to be derogatory and damaging to the employer's reputation" and that if employees are wearing them they should be advised to take them off immediately.

The union doesn't see the buttons as offensive but rather, accurate.

Employees at the Canada Revenue Agency who wore this button to work were told to remove them and are now filing grievances. (CBC News)

"We think it pretty well summed it up – Stephen Harper hates public servants and there were buttons created," said Campbell, who himself wore a button.

He said preventing employees from wearing the buttons violates their civil rights.

"We're allowed to protest and make a statement," he said. "If we walked in and said the income tax system is bad … I could see the problem.

"But as a Canadian citizen, I can't say I'm not happy with the way the government's treating me? That doesn't seem like it's a fair thing."

Buttons have caused problems before

Grievances are filed when union members believe their rights under their collective agreement have been violated. Depending on the agreement there can be multiple stages to the grievance process. Adjudication before a third party can be the final stage.

The union gave suggested wording for the grievance to members and it said that by directing the employee to remove the buttons the CRA violated the collective agreement and the Public Service Labour Relations Act. The act says employees are free to join the employee organization of his or her choice and participate in its lawful activities.

The UTE said it is working with PSAC to see what other potential actions it can take to challenge "these arbitrary and oppressive actions" by the CRA.

This isn't the first time the union has had a conflict with management over the wearing of buttons. Campbell said "Stop Harper" buttons have also had to be removed, and buttons in support of the unions weren't allowed to be worn either. He's not optimistic the union will win the fight over the "Harper hates me" buttons, but he says it will try.

The public service is supposed to be impartial and the Public Service Employment Act lays out rules for engaging in political activity. It says that employees are responsible for assessing whether an activity could impair, or be perceived to impair, their ability to perform their duties in an impartial manner.

A political activity includes "displaying political material to support or oppose a political party or candidate."

National Revenue Minister Gail Shea's office said it doesn't comment on internal staffing matters.

Edith Bramwell, who works as legal counsel with PSAC and often deals with issues of political activity, said the union fully supports the principle of a politically neutral public service.

At the same time, however, "We don't think there should be unreasonable limits on people's rights to political expression," she said.

Bramwell said freedom of political expression and an impartial public service are both important elements of a democracy and they need to be balanced.

Public service 'under attack'

"So it's a question of when and where do you reach a point where you have to take from one side of the balance and give to the other," she said.

Bramwell said for the average public servant who is just going about their job at their office – not a head of a department or other senior level manager – there should be no limitations on their political expression.

"I would say this latest incident is part of a trend. We see these crackdowns coming in an environment where the need for a viable public service is being questioned and where public service workers really are under attack," she said.

The ongoing cuts to the public service and the Conservatives' attitude toward collective bargaining indicate that the government doesn't recognize how important the public service is, Bramwell said.

Parks Canada employees were recently warned by letter that they're not allowed to criticize the government or agency. They were told only designated people can speak to the media about budget cuts and that employees are to follow the code of ethics.

CBC News asked CRA for a comment Thursday on the button controversy and the department provided one Friday afternoon.

"All CRA employees are subject to a strict standard of conduct defined in the agency's code of ethics and conduct, which requires employees to behave ethically and with good conduct when acting in their professional capacity on behalf of the CRA," the emailed statement said.

"The CRA has the authority as employer to manage the workplace as established in its collective agreements and the Canada Revenue Agency Act ... to ensure that employees respect their obligations," it said.