Three of the men, aged between 19 and 29, have confessed, and two are still being questioned, police spokesman Horst Spitzhofer said.
He added that the men are not believed to be part of the right-wing scene but had been drunk, and had decided to target a pro-asylum seeker campaign.
They tore down and ripped up several posters in support of asylum seekers in Dorfplatz, near the town hall at 2:00am on Monday morning.
They then got into an argument with two supporters of the local "We are asylum" campaign and threatened the men, aged 32 and 46. Witnesses prevented the scuffle from getting out of hand.
Later that morning, at 5:00am, two of the men tried to break into the asylum seekers centre but were prevented by the same men they had fought with earlier who were staying at the home. The vandals kicked in a glass panel on the front door and slightly injured the 32-year-old man.
Some witnesses told Austrian broadcaster ORF that the men had been shouting right-wing slogans outside the home.
A few days ago hundreds of Alberschwende residents gathered to protest against the deportation of five Syrian refugees from the community of just over 3,000 people.
Vorarlberg governor Markus Wallner condemned the attack. "Anyone who thinks he can express his opinions on asylum through violence, will have to answer to the police and the prosecutor," he said, adding that this was a "zero tolerance" issue.
He said that although the men were very drunk at the time, this was no excuse. They will be charged with criminal damage, trespass, and assault.