There was a major setback Thursday for former residents of the Mount Herbert Orphanage in Prince Edward Island.

Fifty-seven former residents of the orphanage filed a class action suit in 2002 alleging they suffered physical and sexual abuse there.

The Supreme Court of Canada decided Thursday that although the province gave grants to the orphanage, it didn't own or run it and therefore wasn't responsible for most of the children who lived there.

However, the court said the province was responsible for 14 children who became wards of the government while living at the orphanage between 1956 and 1964.

The orphanage, run by the P.E.I. Protestant Children's Trust, closed in 1976.

The ruling upholds one by the P.E.I. Court of Appeal in January 2009.

"I conclude that the Court of Appeal was correct in finding that the agreed facts and legislative record did not support a finding of vicarious liability of the province for the acts of the home's employees," the court ruling read.

The Mt. Herbert Orphanage was closed in 1976 and demolished in 1998. (CBC)

The decision means 14 of the former residents will be able to go ahead with their multi-million dollar lawsuit against the province.

The remaining 43 will only able to sue the Protestant Children's Trust, which holds limited assets, believed to be about $1 million.

Lawyers for the former residents of the orphanage argued the province had a level of parental responsibility for the children and there had been legislation connected to the establishment of the orphanage, but the Supreme Court largely sided with the province.

"The home was not a child welfare agency under the legislation, the children were not foster children or wards of the province and the legislation created no role for the province in the operation of the home, for the care of the residents, for directing their care, or for ensuring that no harm came to them in the course of their care by the representatives of the home," the decision read.

Province could have responsibility for some

Hardy Broome wants to see the case against the province continue. (CBC)Hardy Broome, who is the lead litigant in the case and is one of the 14 who were wards of the government, was put in the orphanage in 1956 with his three brothers when he was little more than a baby. Broome told CBC News Thursday he and his brothers were beaten and often locked in the coal cellar or in a cage in the attic.

He was at Mt. Herbert for 14 years. One of the matrons at the orphanage took him into her own home until he turned 18.

Broome said he would be telling his lawyers to carry on with the case against the province.

"I think they're responsible for the 14. They're the ones that put us in there," he said.

"It's still going to carry on, as far as I can see."

Broome said the option was considered to launch the case with only the 14 who were under the care of the province, but he rejected that idea. He said all the residents went through the same experience.