Projections showed the European Union-sceptic FPO winning for the first time and with its highest ever vote share after a campaign dominated by immigration and worries about the economy.
"Voters put their foot down today," Kickl said, reading the projected win for his party as a "clear statement that things cannot go on like this in this country".
The most closely watched projection, by pollster Foresight for broadcaster ORF, put it on 28.9 per cent based on a count of 82.8 per cent of ballots, with Chancellor Karl Nehammer's ruling conservative Austrian People's Party (OVP) second on 26.3 per cent.
Chancellor Karl Nehammer has rejected a coalition with the FPO if Herbert Kickl remains its leader. (AP PHOTO)
The FPO would need to form a coalition with one or more parties to secure a parliamentary majority and build a stable government but when party leaders held a discussion on ORF on Sunday evening, no potential partners were forthcoming.
"I do not want you in government and I stand by that," the leader of the fourth-placed liberal Neos, Beate Meinl-Reisinger, said, addressing Kickl directly.
"I simply believe it would not be good for our country."
Of the four parliamentary parties other than the FPO, three have long ruled out a coalition with it altogether.Â
Only the ruling OVP has left the door open to one although Nehammer has said his party will not join a government with Kickl in it, which he repeated after results came in.
The leader of the Social Democrats, Andreas Babler, also called for an alliance against the FPO.
Kickl said they should question their democratic credentials.
"I think there's something you haven't understood, and this goes for Karl Nehammer as well as you," he responded to the Neos' Meinl-Reisinger.
"You can continue your attacks in my direction but you're forgetting something. I stand here only as an ambassador and as an advocate for many, many voters in this country."
The FPO, which is critical of Islam and pledges tougher rules on asylum seekers, won an Austria-wide vote for the first time in June when it beat the OVP in European Union elections.
The party is a long-established political force but Sunday's projected result, if confirmed, would be its best yet in a parliamentary election - beating the 26.9 per cent it scored in 1999.
In its election program, titled "Fortress Austria," the Freedom Party calls for "remigration of uninvited foreigners" for achieving a more "homogeneous" country by tightly controlling borders and suspending the right to asylum via an "emergency law".
with AP and DPA
