Prime Minister Stephen Harper wouldn't let himself be drawn into a battle Friday over the Canadian flag at Quebec's national assembly, referring to it as an old quarrel.

Quebec newspaper Le Soleil is reporting that the Parti Quebecois has asked the Speaker of Quebec's legislature to remove the Canadian flag.

Speaking to reporters in Quebec City, Harper threw up his hands and said, "What can I say?" after hearing the question.

"I can say that our priority, for the population of Quebec and the rest of Canada, is the economy, and I think that is the real priority of Quebeckers is the economy, not these old quarrels. I have no intention of participating in...quarrels," Harper said.

The Quebec Liberals call the request "shocking and totally unacceptable."

The PQ removed the flag from the legislature last September when its members were sworn in.

Restoring Quebec City's armoury

Harper was in Quebec to visit the Citadelle and discuss plans to completely restore a nearby armoury that burned down in a 2008 fire. The armoury will become a "multifunctional facility," housing the museum and a conference auditorium for international events, Harper said. It will also be used as a workplace for federal employees.

Most of the building, which was built in 1884, collapsed in the fire, leaving only a brick wall and two towers standing. The building was under renovation at the time.

In 2010, Josée Verner, then-federal intergovernmental affairs minister, said Ottawa would have the armoury rebuilt by 2016.

The final plans wouldn't be completed before 2012 and construction was unlikely to begin until 2013, Verner said at the time. It was expected to cost $3.5 million to prepare the site.

The remaining parts of the façade will be integrated into the new design, she said.

The armoury was designed by architect Eugène-Étienne Taché, who also designed the national assembly. Opened in 1887, it is considered to be the second most important military building in Quebec, after the Citadelle. The armoury is home to the Voltigeurs, an army unit that fought in the war of 1812.

The Citadelle, home to the 2nd Battalion Royal 22nd Regiment, is the only historic site in Canada still garrisoned by a regular military unit.

with files from CBC News