The most radical right-wingers number 32,000 in the country, said BfV chief Thomas Haldenwang, adding that 13,000 are considered potentially violent.
“Right-wing extremism and right-wing terrorism are currently the biggest danger for democracy in Germany,” he said at a press conference.
His agency also placed under formal surveillance the far-right AfD party's most radical faction Flügel (The Wing) which now has about 7,000 members.
Founded in 2013, the AfD has grown and shifted further right over the last seven years.
Railing against Chancellor Angela Merkel's decision in 2015 to keep Germany's borders open to refugees, the AfD has scooped up a significant number of votes from those unhappy with the government's migration policy.
It is now the largest opposition group in the Bundestag, Germany's lower house of parliament.
The AfD's most radical wing, the Flügel, was founded by far-right firebrand Björn Höcke, who has sparked outrage with statements on Germany's Nazi past.
Höcke has called Berlin's Holocaust monument a “memorial of shame” and urged a “180-degree shift” in the country's culture of remembrance.
Haldenwang said security officials believe the Flügel violates “characteristic features of the free democratic basic order, human dignity, democracy and the rule of law”.
Recent attacks in Germany meant it was more important than ever to monitor groups that may be inciting hatred, he added.
A gunman with apparent far-right beliefs killed nine people at a shisha bar and a cafe in the city of Hanau in February, while two people were killed in an attack targeting a synagogue in Halle in October.
In June, pro-immigration politician Walter Lübcke was found dead at his home in the state of Hesse, with a far-right sympathiser confessing to the attack.