Furore over the presence of neo-Nazi protesters at a Melbourne anti-trans rights rally has spilled over into federal parliament, with Opposition Leader Peter Dutton rebuffing a fiery tirade from Attorney-General Mark Dreyfus.

The clash comes after an ongoing push by Victorian Liberals leader John Pesutto to expel MP Moira Deeming from the parliamentary party following her attendance at the anti-trans rights rally on Saturday.

Attorney-General Mark Dreyfus launched an attack on Opposition Leader Peter Dutton during question time over what he alleged was a failure to condemn the neo-Nazi protesters.

Mr Dreyfus — who is of Jewish ancestry — described the events on the steps of Victorian Parliament as "abhorrent" which, he said, brought to the fore memories "of ghettos, of deportations and mass murder that touched my own family".

The Attorney-General noted Mr Pesutto's prompt move to expel Ms Deeming and attacked Federal Opposition Leader Peter Dutton over his lack of a public response.

"What have we had from those opposite, in particular their leader? Complete silence, Mr Speaker. We all know that bigotry and hatred breed in silence," Mr Dreyfus said.

"He has failed to condemn the display of the Nazi salute on the steps of the Victorian parliament. He has been invisible since the weekend, he has done no media."

Since the anti-trans rights "Let Women Speak" event, organisers have moved to distance themselves from the neo-Nazi group, and the Victorian government has vowed to ban Nazi salutes.

Opposition leader staunchly rejects attack on morals, backs anti-Nazi legislation

In response to the accusations, Mr Dutton vehemently denied any assertion that he endorsed the events of the weekend and was quick to condemn any glorification or use of Nazi symbols.

"I would support any legislation in this parliament that you choose to move, noting that you've not chosen to move any legislation to make illegal in our country the display of any aspect of Nazi glorification," Mr Dutton said.

Mr Dutton referred to the treatment of people of the Jewish faith by those with aligned with Nazism as "an abomination".

"It is equally condemned, and that it would be used for political purposes in this place is a very, very poor reflection on you if I might say," Mr Dutton said.

Mr Dutton pointed to his own record as the former minister for defence and minister for home affairs as evidence of his stance on far-right groups.

"I supported every decision, in fact encouraged to the nth degree, the director-general of ASIO to use every resource at his disposal to make sure that those who would seek to propagate this hatred should be charged according to the law, and I won't take a morals lecture from [Mr Dreyfus], or indeed [Anthony Albanese]," Mr Dutton said. 

"[The Prime Minister] should get up and make the statement himself."

Mr Albanese condemned the anti-trans rights rally and the appearance of neo-Nazis in a radio interview on Tuesday.

"There's no place in Australia for Nazi salutes and people basically paying tribute to Nazis who were responsible for the Holocaust," Mr Albanese told Nova 100 radio.

Victorian Liberal leader continues push to expel MP

Earlier on Tuesday, the Victorian Liberals held a meeting following Opposition Leader John Pesutto's announcement that he would move to expel Ms Deeming from the party.

A vote on her expulsion is due to take place next Monday, and a motion raised in Tuesday's meeting to delay that vote failed.

Mr Pesutto said the party room discussion during this morning's meeting was not "an easy conversation".

"These are not easy issues for any party to have to grapple with, but what the party room does understand, I can say to all Victorians, is that they can understand why I feel it necessary to take this action," he said.

"Leadership carries with it great responsibilities and that includes taking tough decisions from time to time."

He said he was confident that he would have the support of the party room for next Monday's vote.

Mr Pesutto also confirmed he had not spoken to Mr Dutton about the matter.

Ms Deeming maintained in a Facebook statement that she had done nothing wrong, and denied any connection to the neo-Nazis who attended the rally.

"I completely reject the beliefs of National Socialists (Nazis) and I have seen first-hand the impact that the Holocaust had on a family member," Ms Deeming said.

"If Daniel Andrews had not repealed the 'move on' laws, they could have been removed. The 'move on' laws need to be strengthened and I also welcome moves to ban the Nazi salute."

Anti-trans rights activist strongly denies far-right links

Anti-transgender rights activist Kellie-Jay Keen-Minshull — who organised Saturday's rally — also denied her group had any links to neo-Nazi groups.

When Mr Pesutto explained why he was moving to expel Ms Deeming, he claimed there were speakers at the event with "known links with Neo-Nazis and white supremacists".

The Victorian opposition leader alleged Ms Keen-Minshull had "done videos" with figures such as former Klu Klux Klan leader David Duke and neo-Nazi Richard Spencer.

Ms Keen-Minshull has vehemently denied being a white supremacist and laughed off suggestions she was a sympathiser of the Klu Klux Klan.

"It's absolutely repulsive that somebody in a position like the leader of the opposition would spew such vile lies about somebody," Ms Keen-Minshull told 3AW radio on Tuesday morning.

The British activist — also known as Posie Parker — said that assertions Mr Pesutto made were completely false.

"I've never done anything with those men, ever, in my life, ever," she said.

"This is what happens when women speak up about our rights: We get accused of all sorts of really revolting things. This is apparently one that, for some people, lands."

Ms Keen-Minshull said she would likely not have time to pursue legal action against Mr Pesutto over his comments.