A woman is led away Sunday after being arrested in a morning raid on a residence at the University of Toronto's downtown campus. About 70 people were taken into custody without incident on suspicions they contributed to disorder in protests against the G20 summit. (Christinne Muschi/Reuters)

Police and anti-G20 protesters engaged in tense and sometimes violent confrontations in several parts of downtown Toronto for a second day on Sunday, leaving at least 604 people under arrest.

About 500 people were surrounded by hundreds of riot police early Sunday evening at the intersection of Queen Street West and Spadina Avenue, where several standoffs had occurred Saturday on the first day of the two-day G20 summit.

Police said they decided to box in a large group of protesters who were making their way on Queen, heading for Peter Street, because there were militant Black Bloc members donning masks while weapons were found along the way.

But only a small number in that boxed-in crowd were arrested for a breach of the peace and taken to the makeshift detention centre in the city's east end, said Toronto Staff Sgt. Jeff McGuire.

Most were detained at the intersection in a heavy downpour for several hours until Toronto police Chief Bill Blair ordered them released without any charges at about 9:40 p.m. Several people told local media they were innocent bystanders who had been waiting for buses, walking their dogs and minding their own business.

"It was unfortunate they found themselves in the situation," McGuire said. "But the officers had a right to detain them."

East-end showdown

Earlier Sunday, the G20 detention centre in a former film studio on Eastern Avenue in the city's east end was a flashpoint between police and demonstrators on two occasions.

At mid-afternoon, police in riot gear began partly surrounding a group of bicyclists who had staged a protest through downtown demanding more rights for cyclists. For a while outside the centre, the cyclists called for the release of detainees, saying, "Let them go, let them go."

Tensions eased around 6 p.m. ET after police asked the protesters to open the street to allow the release of some of the prisoners. The CBC's Amber Hildebrandt reported that the crowd of cyclists and protesters on the north side of the street cheered each time a person was released.

Three people react after being released Sunday from a temporary detention centre related to G20 summit demonstrations. (Jacques Boissinot/Canadian Press)

Previously at the same location, police fired at least half a dozen rubber bullets and arrested several people. The confrontation began when an estimated 150 demonstrators started staging a peaceful gathering while police in riot gear looked on.

At one point, plainclothes police arrived, entered the crowd and began to arrest several people.

"They knew who they were looking for," the CBC's Bill Gillespie reported. "These are trained police snatch squads using intelligence on finding suspected troublemakers."

At the same time, police formed a line in front of the crowd, telling the protesters to "move back." They then opened fire with rubber bullets, Gillespie said.

The crowd began to move away from the detention centre area, returning north to Queen Street East, he said.

The Integrated Security Unit, which is handling overall policing issues for the summit, told CBC News there was no immediate information on whether rubber bullets had been fired, said spokeswoman Catherine Martin.

According to the ISU, the number arrested in G20-related incidents had risen to 604, with 253 arrests reported Sunday.

Along with the standoff in the east end, another emerged in the west-end Parkdale neighbourhood.

A Starbucks outlet at John Street and Queen Street West remains boarded up on Sunday after being vandalized in anti-G20 violence on Saturday. (CBC)

About 80 people were detained and some were seen being strip-searched in front of Parkdale Community Legal Services on Queen Street West. About 40 of them had been preparing to board a bus bound for Quebec when the police surrounded them, freelance journalist Rebecca Granofvsky-Larsen told CBC News.

On Sunday morning, dozens of people were arrested at the University of Toronto.

Seventy people were rounded up after police said they found makeshift weapons, including bricks, and black clothing hidden in bushes. It's believed the bricks were to be used by vandals who had caused widespread damage Saturday.

'I was there to peacefully protest'

Several handcuffed people were seen being taken into waiting police buses or, in at least one instance, a court services vehicle.

One man dressed in black told CBC News: "I was there to peacefully protest."

Const. Rob McDonald told reporters it was his understanding that people from across Canada had been arrested: "They were found in possession of bricks and other items that could compromise the safety of the citizens of Toronto."

Four other people were arrested in the early morning after they were caught coming out of a sewer in the financial district on Queen Street West between Yonge and Bay streets.

Toronto police spokesman Sgt. Tim Burrows told CBC News that the four were arrested at 2:25 a.m. ET "while leaving a maintenance hole cover, after being in the underground infrastructure of the tunnels."

A G20 protester takes pictures of riot police. (Ryan Remiorz/Canadian Press)

Burrows said no explosives were found and "the security plan is well intact."

All day, a heavy police presence continued in the downtown area near the convention centre, a day after dozens of businesses, as well as police cars and other vehicles, were damaged.

Amnesty International Canada is calling for an independent review of the G8/G20 security measures.

Alex Neve told CBC News there were some "very troubling human rights dimensions" over how security was used in the past several days.

The human rights group is concerned by the "heavy, heavy police presence" in the city and by the talk about "new weapons and unclear laws" that "really led to quite a considerable chill."

"There were many people who either participated in the protests on Saturday with the sense of nervousness and fearfulness, but also many who stayed away," he said.

With files from The Canadian Press