© PAProfessor Alexis Jay is the national inquiry's fourth chair.
The Shirley Oaks Survivors Association, which represents 600 victims of alleged sexual abuse, said it has lost confidence in the inquiry's leadership. The group's withdrawal is yet another blow for the inquiry, which is on its fourth chairwoman in just two years and has seen a number of senior lawyers resign in recent months.
The group represents former residents of children's homes run by Lambeth council who claim they were abused by paedophiles over several decades. It said it believed the inquiry was a "botch job" and chair Professor Alexis Jay was the wrong person to lead it. Its representative, Raymond Stephenson, told the BBC's Radio 4 Today programme the inquiry was "failing tragically" and "failing publicly". He added: "They need to get rid of Alexis Jay, who has been parachuted in by the Home Office.
"She hasn't done anything wrong, she's just not the right person. We have to make a decision based on what we feel."
Chuka Umunna MP, whose Streatham constituency has former Shirley Oaks' residents, said: "I've very reluctantly come to the conclusion that we need a new chair of the inquiry.
"For a long time people have had concerns about the appointment of the current chair, whether they have the heft and forensic capacity to lead an inquiry of this type.
"A High Court judge needs to do this. That's what the survivors want. There is no point carrying on with the wrong person in the job simply because we have already changed the person in that position several times."
The Shirley Oaks case represents one of the 13 strands of the inquiry being investigated by the Independent Inquiry into Child Sexual Abuse.
Professor Jay is the inquiry's fourth chair after New Zealand judge Dame Lowell Goddard quit during the summer. Further frustration arose for the Home Affairs Select Committee when she refused to give evidence in person about her decision to leave the post.
Professor Jay's background as a social worker has not instilled confidence in the survivors, said Mr Umunna, and this issue has been "swept under the table".
This week the Prime Minister said she had full confidence in the inquiry.
Home Secretary Amber Rudd said: "The independent inquiry has a vital role to play in exposing the failure of public bodies and other major organisations to prevent child sexual abuse.
"We must learn the lessons of the past and we owe it to victims and survivors to get behind the inquiry, and its chair Alexis Jay, in its endeavour."
Comment: This latest setback for the UK child sex abuse inquiry comes just days after ITV News learned that the inquiry had postponed its investigation into the late Labour peer Lord Janner.
What links these stories, and is the real crux of the abject failure in providing justice for the victims, is one of intent.
The inquiry into historic child sex abuse in the UK is "failing tragically" by design. There is NO REAL INTENTION to sincerely investigate these vile crimes against innocent children by the very 'Establishment' that has always colluded to protect the depraved monsters who stalk the Establishment corridors of power.
The inquiry is an illusory smokescreen to create the impression that justice is being served, when in reality it is all part of their modus operandi. For more information, read: UK 'Establishment': Unmasking psychopathic faces - Pedophilia and murder in VERY high places