"The Rwanda scheme was dead and buried before it started," Starmer said in his first press conference on Saturday.
"It's never acted as a deterrent. Almost the opposite."
No asylum seeker has been deported under the scheme, described by critics as an "Alice in Wonderland adventure that was both absurd and inhumane".
Scores of people, reportedly as many as 200, who were due to be deported have recently been released on bail by immigration judges because there was no longer a realistic prospect of removal within a reasonable timescale.
Starmer has said he will curb Channel crossings by hiring specialist investigators and using counter-terror powers to "smash the criminal gangs" behind the flow of migrants into the UK, but how this will work in practice remains largely unknown.
The financial implications of walking away from the deal and the total bill to the taxpayer are not yet known.
Earlier this year Rwanda's President Paul Kagame suggested British taxpayers' money could be repaid if the deal failed, though his view of the change in UK leadership is uncertain.
Yolande Makolo, a spokeswoman for his administration, later said the country had "no obligation" to return any of the funds but if the UK requested a refund "we will consider this".
Starmer told reporters in a wood-panelled room at 10 Downing Street that he was "restless for change," but would not commit to how soon Britons would feel improvements in their standards of living or public services.
His Labour Party delivered the biggest blow to the Conservatives in their two-century history on Friday in a landslide victory on a platform of change.
His new government takes on the massive challenge of fixing a heap of domestic woes and winning over a public weary from years of austerity, political chaos and a battered economy.
Prime Minister — UK Prime Minister (@10DowningStreet) @Keir_Starmer chaired his first Cabinet this morning.He outlined the government's mission to restore Britain to the service of working people. pic.twitter.com/M7d8e9k8DzJuly 6, 2024
"We have a huge amount of work to do, so now we get on with our work," Starmer said.
He said it had been the honour of his life to be asked by King Charles III to form a government in a ceremony that officially elevated him to prime minister.
Among a raft of problems they face are boosting a sluggish economy, fixing a broken health care system, and restoring trust in government.
In his first remarks as prime minister after the "kissing of hands" ceremony with Charles at Buckingham Palace, Starmer said he would get to work immediately, though he cautioned it would take some time to show results.
"Changing a country is not like flicking a switch," he said as enthusiastic supporters cheered him outside 10 Downing Street.
"This will take a while. But have no doubt that the work of change begins — immediately."
Keir Starmer has met with King Charles in a meeting that officially elevated him to prime minister. (AP PHOTO)
He will have a busy schedule following the six-week campaign, heading out Sunday to visit each of the four nations of the UK - England, Scotland, Wales and Northern Ireland - that he said had each voted in favour of Labour.
His first visit will be to Edinburgh on Sunday, where Starmer will say he wants to "turn disagreement into co-operation" with the SNP north of the border.
"People across the United Kingdom are bound by shared beliefs. Fundamental values of respect, service and community which define us as a great nation," he said.
"That begins today with an immediate reset of my government's approach to working with the First and deputy First Ministers, because meaningful co-operation centred on respect will be key to delivering change across our United Kingdom."
He will then travel to Washington for a NATO meeting Tuesday and will host the European Political Community summit July 18, the day after the state opening of parliament and the King's Speech, which sets out the new government's agenda.
with PA
