Almost 60 people have voluntarily left the center on buses that had been waiting outside to take them to new facilities, according to a statement Friday from Papua New Guinea (PNG) police.
The operation, codenamed "Helpim Friends," would use buses and trucks to transport the refugees to their new centers "without (the) use of force," Manus Provincial Police Commander Chief Inspector David Yapu said in the statement.
He added police would deal with any situations which arose during the transfer "in a professional manner."
On Thursday, PNG police told the refugees remaining in the formerly Australian-run center they had two days to get out or they would be forced to leave.
"If necessary, force may be used to relocate those who refuse to move voluntarily for your own sake," said a statement distributed to refugees.
Speaking to CNN, refugee Aziz Adam said the authorities hadn't told them anything Friday before they moved in and "started to break things."
Around 600 refugees have been without food and water for over a week after services were shut off at the Manus Island center, which officially closed on October 31.
The refugees, detained in the center for years, say if they leave they risk being attacked by locals who don't want them living in their town.
Unlike the existing facility, which is located on a secured site far from residential areas, the proposed new facilities are located in the town of Lorengau.
The site's closure follows a PNG Supreme Court ruling last year, that the refugees -- who were not allowed to leave the Australian-run immigration center -- were being deprived of their personal liberty.
Fences, structures torn down
Papua New Guinea police and immigration officials tore down the fences surrounding the camp on Thursday ahead of the planned move. On Friday, authorities moved in to remove shelters within the property.
"Police and immigration destroyed our shelters. Inside the rooms is very hot without power for fans. We built these shelters to provide shade & cover from tropical sun & rain," refugee Behrouz Boochani wrote on Twitter from inside the center.
Police and immigration workers dismantling the shelters on Friday have thrown away water being stored by refugees, according to the refugees, and photos posted on social media.
The Manus Island Regional Processing Center was originally built by the Australian government in 2001, together with another center on the small Pacific country of Nauru, to process asylum seekers who arrived by boat.
Even if those processed on the island were found to be refugees, they were told by the Australian government they would never be settled in Australia.
"This is a humanitarian crisis completely of Australia's own making," Dr Barri Phatarfod of Doctors for Refugees told CNN.
"People are dying. The time to score political points at the expense of people's lives has well and truly come and gone. There have been nine deaths in offshore detention center, and six on Manus Island. They cannot continue to play with people's lives."
Australian activists, protestors speak out
Australian activists are continuing to work to help the refugees. On Monday, lawyers for Boochani will appeal to the PNG Supreme Court to grant travel documents to refugees and asylum seekers who wish to leave the country.
Australian Greens Senator Nick McKim arrived at the Manus Island detention center for the second time on Friday, writing on Facebook the situation was "teetering on the edge of tragedy."
"Lives are at risk, and they are at risk purely because of the choices (Immigration Minister) Peter Dutton and (Prime Minister) Malcolm Turnbull have made," he said on his official social media.
Meanwhile, small but eye-catching protests over the Manus Island crisis have continued across Australia in recent days, including a blockade of Customs House in Melbourne.
On Thursday, protesters unfurled a banner on top of the Sydney Opera House which read, "Australia: World leaders in cruelty."