"If they are now threatening (to go to UN institutions), they must know something simple: they will pay a heavy price," Tourism Minister Uzi Landau told public radio.
Palestinian Authority president Mahmoud Abbas on Tuesday said he had begun steps to join several UN agencies, abandoning a pledge to freeze such action for the duration of peace talks - which end in just four weeks.
Abbas announced a request to join "15 UN agencies and international treaties."
"The demands (for membership) will be sent immediately" to the relevant agencies, he said.
The documents Abbas signed, officials said, included the Geneva Conventions - the key text of international law on the conduct of war and occupation.
Palestinians hope it will give them a stronger basis to appeal to the International Criminal Court and eventually lodge formal complaints against Israel for its continued occupation of lands seized in the 1967 war that they want for their state.
"This is not a move against America, or any other party - it is our right, and we agreed to suspend it for nine months," Abbas said of the decision.
Hamas, the Gaza Strip's ruling party, welcomed the move by Abbas,
The Palestinians had repeatedly warned that they could resume their action through international courts and the UN over Israel's settlement expansion on occupied territory in the West Bank and in annexed east Jerusalem.
"One of the possible measures will be Israel applying sovereignty over areas which will clearly be part of the State of Israel in any future solution," said Landau, a member of the hardline Yisrael Beitenu faction.
Landau's remarks were referring to areas of the West Bank populated by Jewish settlers which Israel hopes to retain in any future peace deal.
Israeli settlements are illegal under international law.
An Israeli government official, who would not be named, said Abbas's announcement had thrown everything up in the air.
"Is this Israel's partner? Is this a partner for peace?" he asked.
"Everything has changed now, is there even a deal now? We don't know," he said, referring to the proposal which was being discussed with Kerry.
Israel could also hurt the Palestinians economically by acting "to block financial aid to them," the minister added.
Abbas made his announcement just hours after Israel reissued tenders for hundreds of settler homes in east Jerusalem, as Washington was working around the clock to resolve a major dispute over Palestinian prisoners.
Israeli NGO Ir Amim described the tenders as "a poke in the eye of both the Palestinians and the Americans," army radio said.
And Hagit Ofran, from Israeli's Peace Now NGO, accused the housing ministry of "trying to forcefully undermine the peace process... and John Kerry's efforts to promote it."
Israel refused to free a fourth and final group of 26 veteran Palestinian prisoners on Saturday, which would have completed an agreement that had brought the sides back to the table.
On Monday, the Palestinian Authority gave Kerry a 24-hour deadline to come up with a solution to the prisoner row, warning that failure to do so would see them turning to UN bodies to press their claims for statehood.
"America must compel Israel now to follow through on its agreement to release the fourth group of prisoners. We will be watching these efforts and hope they don't fail," Palestinian Prisoners Affairs Minister Issa Qaraqe said.
"But our position is clear: we want the release of the fourth group according to the agreement, and without that we won't accept any other obligations or conditions," he said.
The standoff came soon after US Secretary of State John Kerry left Israel on Tuesday after a lightning visit.
He had been due to fly back to the region on Wednesday for talks in Ramallah with Abbas but he cancelled his visit following the Palestinian leader's announcement, while attempting to remain optimistic.
"It is completely premature tonight to draw... any final judgement about today's events and where things are," he said in Brussels.
"My team is on the ground meeting with the parties even tonight," he said. "We urge both parties to show restraint."
US efforts have been focused recently on getting the parties to agree an extension to the end of the year.
A US proposal to continue talks was to include a limited "freeze" on settlement construction, with Israel adopting "a policy of restraint with (West Bank) government tenders" but would not include annexed east Jerusalem.
Sources close to the negotiations had said Washington was also mulling a proposal to free Jonathan Pollard, who was arrested in Washington in 1985 and sentenced to life in prison for spying on America on Israel's behalf.
But White House Jay Carney said before the Tuesday afternoon developments that President Barack Obama had not made any decision on Pollard.
Separately, a spokesman for the US Justice Department said Pollard had waived his right to attend a meeting of a parole board that could have re-examined his ongoing detention.
5 Broken Cameras on PBS will help others to understand the situation.