Those officials, who asked not to be identified discussing sensitive issues, say China is becoming a harder target and more opaque, just as the demand for insights into Xi's decision-making is soaring and tensions with the United States are heating up over issues from Taiwan to high technology.
That reality comes after officials in both the Trump and Biden administrations said they were surprised by Beijing's rapid moves to consolidate control of Hong Kong, project military power across the South China Sea, limit probes into the origins of Covid-19, undercut Chinese companies going public in the US and ramp up hacking against adversaries.
The current and former officials emphasise that America's spy agencies have long struggled to provide the insights policymakers demand on China. The hurdles facing the US intelligence community are both deep-seated - Beijing did significant damage to American spy networks in China before Xi's presidency - and basic, including a continuing shortage of Mandarin speakers.
Comment: It's rather telling as to the state of the US that it has apparently been unable to find enough staff capable of learning Mandarin.
"Our human intelligence has been lagging for decades," former national security adviser John Bolton said in an interview, when asked about China. "I never feel I have enough intelligence. I'm always willing to hear more. I'm never satisfied. No decision maker should be."
As the Biden administration seeks to shift more of its foreign policy strategy towards countering China, Central Intelligence Agency director Bill Burns has announced the creation of a China Mission Centre to hone the agency's focus on "an increasingly adversarial Chinese government".
Some of the people interviewed by Bloomberg said that such announcements were more symbolic than substantive and needed to be backed up by greater spending and staffing to have credibility.
CIA officials declined to comment.
Several of the current and former officials say US intelligence shortfalls are worsening, a problem that comes as the 68-year-old Xi seeks to cement his legacy alongside former leaders Mao Zedong and Deng Xiaoping at a key Communist Party meeting in Beijing this week. That gathering, one of the last hurdles before Xi likely secures a third term as head of the party next year, takes place with the US having little insight on some basic issues, such as who his eventual successor is likely to be. And it comes after some high-profile intelligence flubs on other topics, including the failure to foresee the Taliban's rapid takeover in Afghanistan.
Comment: This isn't true. It was widely known that the US-propped up, corrupt, Kabul government would fall and that the Taliban would be readily accepted by Afghans. And, almost immediately, a US destabilization campaign began in Afghanistan through its proxy army ISIS-K: Pepe Escobar: Afghanistan: between pipelines and ISIS-K, the Americans are still in play
Criticism of the intelligence community's insights on China weigh most heavily on the CIA, which has primary responsibility for recruiting spies and saw its network severely damaged more than a decade ago by Beijing's counter-intelligence efforts.
Those efforts were detailed extensively in 2017 by The New York Times, which said as many as a dozen US sources were executed by China, with others jailed, in what represented one of the worst breaches ever of American spying networks.
The creation of the CIA mission centre was denounced as a "typical symptom of the cold war mentality" by Chinese foreign ministry spokesman Zhao Lijian. The US "should view China's development and China-US relations in an objective and rational light and stop doing things detrimental to mutual trust and cooperation", he added.
Comment: Somehow the US has managed to convince its citizens that this kind of talk from the Chinese is 'adversarial'?
But for the leading consumers of intelligence in the Biden administration - a group that includes the senior-most officials with access to the highly classified President's Daily Brief - a stronger pivot to China cannot come soon enough. Last week, the Pentagon said it now saw China's nuclear arsenal growing faster than forecast, the latest in a series of stepped-up assessments of Beijing's global ambitions.
CIA looks beyond hardware
Xi's sweeping efforts to change China's domestic politics and consolidate his control also have taken a toll on American intelligence. The shift from a system of "collective" leadership under former presidents Jiang Zemin and Hu Jintao towards one dominated by Xi means the CIA has had to go from focusing on the inner circles of seven or even nine top leaders to, effectively, just one.
Comment: And this is just one of the advantages of having a competent and decent leader. Notably Russia is another country that has managed to evade US attempts to destroy it, and it too has, for the most part, relied on a single leader for the last 20 years.
Even before Xi, China's political system was highly secretive and organised using a "stove-piped" system where information flows up to top leaders but rarely is disseminated widely inside the system. Chinese academia, the media and civil society organisations are all closely controlled by the government, further compounding the challenge of reporting on the country.
Consumers of intelligence often failed to recognise the severity of these challenges, former US officials explained, and might have unrealistic expectations for what conclusions could be drawn from any raw intelligence collected in the field.
Xi's broad anti-corruption campaign, which has punished more than 1.5 million officials, has also led to greater scrutiny of Chinese officials' income, making payments to potential sources far more problematic, two former officials said.
Comment: Judging by the complaints of the CIA, the anti-corruption campaign seems to have been pretty effective.
Why China's a 'hard target'
Despite China's history as a "hard target" for the CIA to penetrate, the agency exists precisely to overcome such challenges, whether it is deciphering the leadership of al-Qaeda or Kim Jong-un's regime in North Korea.
What's more, the agency was capable of providing significant insights into the upper reaches of the Chinese political system as recently as a decade ago, one former intelligence official said. Its ability to penetrate the Chinese leadership has ebbed and flowed over time, but the agency's current ability to do so is more limited, the person said.
Another former official said that if he were sitting in the White House Situation Room today, his priority requests of the intelligence community would centre on projections for China's build-up of its navy, cyber and artificial intelligence capabilities; Xi's plans for Taiwan; and better intelligence on Beijing's strategy for the South China Sea. This person said the Trump White House also lacked good intelligence on China's strategy towards Vietnam, India and North Korea.
Comment: The US needs this intelligence, not because it doesn't know what China's intentions are - for example, China has stated quite openly that it intends to "peacefully" reunify with Taiwan - the US needs intelligence that will enable it to thwart those plans.
The frustrations of administration officials echo public assessments from Congress.
A partially redacted House Intelligence Committee report from September 2020 concluded that US spy agencies were failing to meet the multifaceted challenges posed by China and were too focused on traditional targets such as terrorism or conventional military threats.
"Absent a significant realignment of resources, the US government and intelligence community will fail to achieve the outcomes required to enable continued US competition with China on the global stage for decades to come, and to protect the US health and security," according to the report.
Comment: There you have it, the US isn't concerned that China will attack it, it's rightly concerned that China threatens the corrupt US hegemony.
The report also cited America's foreign policy focus on the Middle East and the "war on terror" as reasons the intelligence community came to treat "traditional intelligence missions as secondary to counterterrorism".
Conflict over Taiwan
A leading concern now is the question of whether Xi would invade Taiwan, or possibly seek to take smaller islands controlled by Taiwan, a move that would be seen as a significant test of Western resolve.
Comment: Taiwan is also officially known as the Republic of China (ROC).
General Mark Milley, chairman of the Joint Chiefs of Staff, last week reiterated his view that China was unlikely to take Taiwan by force within the next 24 months. And China's state media have sought to quiet online speculation that a conflict with Taiwan may be imminent.
"Xi has sent contradictory signals on Taiwan," said Bonnie Glaser, director of the Asia programme at the German Marshall Fund of the United States. "It is difficult to disaggregate which signals Xi intends for the party elite, the general domestic audience, Taiwan audiences or the United States."
For now, the intelligence community's analysis relies more on inductive reasoning about whether an invasion would align with Xi's stated objectives than on raw intelligence on the Chinese leader's views, according to the people.
Comment: Note that in the last two decades China hasn't invaded any country. The US on the other hand...
Former officials explained that recovering from China's dismantling of the CIA's network in China involves a multi-year process that includes the recruitment and onboarding of new assets, followed by systematically increasing the asset's access to sensitive information. That is probably still under way, the people said.
In addition, CIA officers in China face daunting challenges posed by China's burgeoning surveillance state, which has blanketed Chinese cities with surveillance cameras and employs sophisticated facial recognition software to track threats.
Comment: "The United States has 15 CCTV cameras [for] every 100 individuals, followed by China with 14".
In an interview with National Public Radio in July, Burns said the agency was looking into how to deal with "ubiquitous technical surveillance" and other "very advanced capabilities on the part of the Chinese intelligence service".
Problem-solving outside China
Burns also has hinted at one potential fix for the agency's problems.
The CIA chief told NPR that the agency was considering whether to deploy China specialists in locations outside China, following the approach used to counter Soviet influence in the Cold War.
One of the former officials said the effort was being undertaken partly in the hope that overseas destinations prove a more fertile recruitment environment than the closely surveilled streets of Beijing.
Comment: It's highly likely that the US already has assets operating in countries surrounding China. However, note that the US is brazenly admitting that it hopes to subvert citizens of other countries in its attempts to destroy the progress of another.
But that strategy is more of a long-term fix. In the short term, officials are having to brace for more of the rapid moves that have distinguished Xi's leadership in recent years, without knowing what they may be.
Bolton, who served under former president Donald Trump, said that meant officials would have to play the hand they were holding now, making the best use of what they had, even if that information had gaps that were widening over time.
"There comes a point when you have to make a decision," he said. "You're not going to have complete intel. Live with it."