Former MPs who weren't re-elected on Oct. 19 are returning to Ottawa this week to clean out their offices and say goodbye to staff, but defeated New Democrat Peter Stoffer is facing a bigger challenge than most.

"You're in what was my office — 8,000 baseball caps, 9,000 pins, 5,000 buttons, a lot of them political, and a whole bunch of other stuff I've accumulated over 18 years," the veteran politician told Rosemary Barton, host of CBC News Network's Power & Politics, in an on-site interview airing Tuesday.

That's going to require a lot of cardboard packing boxes — and that's not including his pool table.

"I don't like blank wall space anyway, so I thought I'd fill [the office] up," Stoffer said of his storied section of real estate on Parliament Hill.

The colourful collection of paraphernalia will end up going to charity, he said.

'I didn't think when Mr. Harper left, he'd take me with him'

Peter Stoffer

Peter Stoffer's iconic office was a colorful addition to Centre Block on Parliament Hill. (Hallie Cotnam/CBC)

Stoffer, who's known for organizing the "All Party, Party" — an annual non-partisan fundraiser for various charities — and was repeatedly voted "Most Fun MP to Work For" by the Hill Times newspaper, admitted he didn't think he'd be clearing out his office after the election.

"On the doorsteps this time, there was a definite attitude for change for Mr. Harper," he said. "I just didn't think when Mr. Harper left, he'd take me with him."

Stoffer had originally planned to retire before the election was called, but decided to run once more in his riding of Sackville-Preston-Chezzetcook. He was defeated by Liberal Darrell Samson.

"It didn't work out, so I didn't get the chance in the House to say goodbye...but I accept the voters' decision with great grace and humility," he told Barton.

Although Stoffer said he believes voters are never wrong, he said was still surprised at the results for the NDP across the country.

Peter Stoffer

Former NDP MP Peter Stoffer is packing up his legendary Ottawa office after serving as an MP from Nova Scotia since 1997. (CBC)

"I don't think it was necessarily a vote against the NDP," he said.

"A guy called me up in tears, sort of like a Catholic parishioner confessing to a priest, and he said 'Mr. Stoffer,' in between sobs, 'I had your sign on my lawn but I voted for [Justin] Trudeau to get rid of Harper.'"

"I think that's what happened to a lot of us — Megan [Leslie], Jack Harris, Ryan [Cleary] and many others," Stoffer continued, listing the names of prominent NDP candidates who lost their seats in Atlantic Canada.

But Stoffer doesn't think he could have convinced voters to choose the NDP over the Liberals.

"I could have done old Maritime politics and got everybody a bottle of rum and graveled their driveway, but it wouldn't have worked," he said.

"They wanted change, complete change, and a lot of them went back to traditional voting patterns of voting Liberal or Conservative."