The Canada Revenue Agency says full service has been restored on all of its online systems as of Sunday.

A release from the CRA said that "individuals, businesses and representatives are now able to file returns, make payments, and access all other e-services available through the CRA's website, including all our secure portals."

"Our systems are back online. We apologize for the delay and the inconvenience it has caused to Canadians. That said, the delay was necessary. We could not allow these systems back online until we were fully confident they were safe and secure for Canadian taxpayers," said CRA Commissioner Andrew Treusch.

The agency is also extending the filing deadline for tax returns and promised to resume e-services by the end of the weekend for all federal departments using software vulnerable to the Heartbleed bug.

"The Minister of National Revenue has confirmed that interest and penalties will not be applied to individual taxpayers filing their 2013 tax returns after April 30, 2014 for a period equal to the length of this service interruption," the CRA said in a statement Saturday.

This means individual tax returns for 2013 filed by May 5 will not incur penalties or interest.

Several department sites had been shut down this week.

The directive issued late Thursday ordered the immediate disabling of public websites, calling it a precautionary measure until the "appropriate security patches are in place and tested." The government did not say how many departments are involved.

The Heartbleed bug is caused by a flaw in OpenSSL software, which is commonly used on the Internet to provide security and privacy.

The bug is affecting many global IT systems in both private and public sector organizations and has the potential to expose private data.

It's not clear the extent to which other federal government systems were affected. Employment and Social Development Canada, which handles things like Employment Insurance and Canada Pension Plan, Social Insurance Numbers and passports, had issued a statement Wednesday saying its web applications do not use OpenSSL and are therefore not affected.

"We have also worked with Shared Services Canada to confirm that none of our other connectivity solutions were affected," the department said in a statement emailed to CBC News.

CRA services affected included tax-filing systems E-file and Netfile, as well as access to business and personal account data stored by the system and new applications for a record of employment. 

In the meantime, Canadians using commercial tax software may need to change their passwords, as some programs such as Turbo Tax were affected by the bug.

The Canadian Bankers Association had said online banking applications of Canadian banks were not affected by the bug.

With files from the Associated Press