Mayor Rob Ford says he's done everything he can to get a rainbow flag supporting gay rights taken down from Toronto's City Hall.

"The bottom line is, again, this is about the Olympics, this is about supporting our athletes," said Ford outside his office on Tuesday.

"I've done everything I can to get the Canadian flag back up," he said.

The rainbow flag was raised to coincide with the Opening Ceremony at the Olympics in Sochi, Russia. It is a protest of the anti-gay laws in that country, and a show of support for LGBT Russians.

Ford said "it's irrelevant" to him whether individuals are "gay or straight" and that the Canadian flag represents all the people in the country.

The mayor also said the majority of phone calls he has received from residents on this issue have been supportive of his position.

There is currently a Canadian flag flying at Toronto City Hall, as is required by the city's flag protocols. The rainbow flag is flying to the east of the clamshell-like building, where an auxiliary Canadian flag usually flies. Community groups can request to have flags raised on that pole from time-to-time.

Ford said he has written to the city manager to get the flag removed, but there is nothing he can do because it is a matter of protocol. The 519 Church Street Community Centre made a request to fly the flag outside city hall for the duration of the Olympics. The mayor has no authority to remove it.

During Ford's discussion of the rainbow flag, protesters outside his office yelled that he was a "homophobe." He then turned around to leave, telling media to "get rid of the protesters" or else he would not speak.

Ford has posted a Canadian flag in the window of his office as gesture against the rainbow flag.

The Olympics "is about being patriotic to your country, this is not about someone's sexual preference," he told reporters on Friday of last week.