Oslo police said the explosion was caused by a bomb, the NTB news agency said.
A Reuters correspondent counted at least eight injured people.
The blast blew out most windows on the 17-story building housing Stoltenberg's office, as well as nearby ministries including the oil ministry, which was on fire.
Heavy debris littered the streets and a tall plume of brown smoke rose over the city center. The tangled wreckage of a car could be seen near the blast site.
The Reuters correspondent said the streets had been fairly quiet in mid-afternoon on a Friday in high summer, when many Oslo residents take vacation or leave for weekend breaks.
© Scanpix Norway / ReutersDebris is seen on the street after the powerful explosion.
"We are trying to stay focused, and we are focused, on the rescue operation," he told the BBC.
The BBC also said there were local reports that two people had been arrested.
Norwegian newspaper Aftenposten said that Stoltenberg was uninjured, and was not in the building at the time of the explosion, which happened around 3:30 p.m. (9:30 a.m. EDT).
Stoltenberg, speaking about 12 p.m. ET, confirmed he was safe and said all government ministers also appeared to be unhurt.
"This is very serious," he said. He said it was too early to say if the blast was a terrorist attack.
'Lingering smell' like 9/11
Olaf Furniss, a freelance journalist in Oslo, told BBC News that people he had spoken to feared it was a terrorist attack.
The BBC reported residents as saying there was a smell of sulfur in the air and asked Furniss about this.
"I was in New York three weeks after 9/11, there was still a lingering smell [and] I would compare it to that, it's very similar to that," he replied.
Furniss told the BBC that he was in a cafe when the blast went off, but he had not realized how big it was until he went outside.
© ESRI
"It exploded - it must have been a bomb. People ran in panic and ran. I counted at least 10 injured people," said another witness Kjersti Vedun, who was leaving the area.
NATO member Norway has sometimes in the past been threatened by leaders of al-Qaida for its involvement in Afghanistan. It has also taken part the NATO bombing of Libya, where Moammar Gadhafi has threatened to strike back in Europe.
However, political violence is virtually unknown in the country.
"There certainly aren't any domestic Norwegian terrorist groups although there have been some al-Qaida-linked arrests from time to time," David Lea, Western Europe analyst at Control Risks, said. "They are in Afghanistan and were involved in Libya, but it's far too soon to draw any conclusions."
John Drake, a senior risk consultant at London-based consultancy AKE said the attack may not be "too dissimilar to the terrorist attack in Stockholm in December which saw a car bomb and secondary explosion shortly after in the downtown area."
"That attack was later claimed as reprisal for Sweden's contribution to the efforts in Afghanistan," he said.
© Thomas Winje Oijord / AFP - Getty ImagesAn injured man is treated at the scene after an explosion near the government buildings in Norway's capital Oslo Friday.
A journalist with the NRK media organization, Ingunn Andersen, told The Associated Press news agency that the main office of newspaper VG was damaged in the blast.
"I see that some windows of the VG building and the government headquarters have been broken. Some people covered with blood are lying in the street," he told the AP. "It's complete chaos here. The windows are blown out in all the buildings close by."
Tom Erik Sundbye said in a tweet that he was just over 4 miles away and "my office building was shaking."
Lise Sand expressed her shock in a series of messages on her Twitter account.
"I've never seen an explosion like this," Sand wrote. "I can't believe how well we felt it. We're pretty far away."
"Our windows shook, and we could actually feel the shake," she added. "The whole area is evacuated. Sirens everywhere."
Craig Barnes, a British man who was in the center of Oslo when the blast happened, told the U.K.'s Sky News that he had "put my foot down" on the accelerator of the car and got to a friend's house.
"It's absolute chaos, there are many people injured," he told Sky.
"There's debris over at least half a kilometer (546 yards) ... total chaos," Barnes added. "It's absolutely mad."
Well the first thing I ask is why here, why now? And because most of us believe that terrorism is a manufactured beast of the 'Powers that Be' unleashed on innocent civilians for nepharious purposes, then who are these people trying to 'set up', protect or distract from?
Whoever, or whatever it is - they seem to be going to great lengths to do it.
It can't be Rupert Murdock, surely? Even though he probably is one of their staunchest supporters.