The family of a 71-year-old man who was reported missing and was found dead the next day vow to fight for answers as to why Ontario Provincial Police let him go after he was pulled over while he was missing.

Graydon Scott told relatives on Thursday morning he was driving from Ottawa to Midhurst, Ont., near Barrie, to visit his son James. When he didn't show up that evening, his family filed a missing person report with Ottawa Police.

Graydon Scott missing found dead OPP April 25 2014

Graydon Scott was reported missing by his family on Thursday evening. (Ottawa police)

Later Thursday evening, an Ontario Provincial Police officer stopped Scott for speeding in the Bancroft area. Scott had asked the officer for directions to a hotel and was directed to one, said OPP spokeswoman Sgt. Kristine Rae.

The following day at 2:30 p.m., someone called police after finding Scott's body in his vehicle on a dead-end road north of Bancroft. It was about a 40-minute drive from where Scott had been pulled over by OPP the previous evening.

OPP told the family they believe he died as a result of health reasons, not as a result of a motor vehicle accident. A coroner's investigation to determine the cause of death is under way.

'He should not have been released'

Alan Lew, Scott's brother-in-law, said Scott should not have been let go by OPP without speaking to anyone.

"They told us that the officer made an attempt to phone Ottawa police, and I understand that he got no response, no answer. That's all we were told. And that he also tried to call Graydon's son, James, and left a message. And I know, speaking to James, James says he did not receive any message. And at that point the officer let Graydon go," Lew said.

"He should not have been released. ... Somebody should have been contacted or he should have been detained because of that missing person's report."

The officer notified Ottawa police and Scott's family, according to OPP spokeswoman Sgt. Kristine Rae. But James Scott, whose home phone was the contact number for police, disputes this and said no one attempted to reach him and no message was left.

SIU won't investigate

James and Michael Scott, two of Graydon's four sons, want the Ontario SIU to investigate why they weren't contacted and why their father was let go.

"I don't know if that police officer could have saved his life but I do feel like at the very least he wouldn't have been on his own in the pitch black confused and not knowing what's going on," said James Scott.

The province's Special Investigations Unit, which handles cases of death, serious injury or sexual assault involving police, has told CBC News they will not be investigating because they say it falls outside their mandate.

Michael Scott said they don't accept that answer.

"We won't let this go until we get some questions answered," he said.