I first tried the infamous Atkins diet 20 years ago, not as a New Year's resolution but as an experiment in participatory journalism. I was a correspondent for the journal Science, investigating the questionable research implicating saturated fat in heart disease. An economist at the Massachusetts Institute of Technology told me that his collaborator's father had lost 200 pounds on the notoriously fat-rich Atkins diet and that he had lost 40. "If you're investigating fat," he said, "you have to try Atkins."

I wasn't obese, but I was pushing middle age and gaining two pounds a year despite all my best intentions. I was eating a very low-fat diet, as per the dietary zeitgeist of the era, and working out religiously, largely motivated by the belief that if I burnt enough calories, I could eat to satiety and still stay lean.

It wasn't working. Getting ever fatter seemed my fate - the best I could do.

So I did my Atkins experiment. I lived on eggs and bacon, steak and chicken and fish and cheese and butter and copious green vegetables. (On Atkins, I joked, bacon and butter are health foods.)

I lost 25 pounds in three months. I even cut back my workouts because I was losing weight effortlessly while eating what seemed like copious amounts of food. The experience was both gratifying and fascinating - precisely what you want from such an experiment.

It came with a catch, though: I didn't know if I was killing myself.

I've never renounced my Atkins diet and have spent the intervening decades researching and writing books on nutrition, obesity and chronic disease. Can a diet that allowed me to achieve and maintain a healthy weight so effortlessly be an unhealthy diet, one that will shorten my life? I find that hard to believe, but that's only one informed opinion among a cacophony of supposedly authoritative voices. The question is among the most important and controversial in all of medicine.

The story of my experiment is known in medicine as an anecdote or anecdotal observation, implying evidence that's inherently unreliable. Anecdotes are noteworthy when they recount an event that is contrary to our expectations. But they don't tell us whether the same thing, good or bad, would happen to anyone else under the same circumstances, let alone everyone else. We can only guess. Anecdotes are often the basis of quackery - but not always. Much of reliable medical knowledge emerged initially from anecdotal observations. Medical science can be thought of as a process that begins with such observations and, through relentless testing of hypotheses, eventually generates truth.

Here's another anecdote - a variation on the theme. This one was told to me last July by Sue Wolver, a physician and associate professor of medicine at Virginia Commonwealth University. I was interviewing Dr. Wolver about her clinical experience prescribing Atkins-like diets to treat the obesity and diabetes that dominate her patient population.

One day last winter, Dr. Wolver said, she received a phone call from a colleague who had just diagnosed diabetes in a patient, a 24-year-old woman. The patient had been unaware that she was sick. But she was very sick. Her hemoglobin A1c - a measure of blood-sugar control and thus severity of diabetes - was 10.5. Physicians consider levels above 6.5 to be diabetic. Over 10, according to American Diabetes Association guidelines, and the patient should be started on insulin therapy.

"Do you think she'd ever get off insulin?" Dr. Wolver asked rhetorically. "Never. So my colleague said to me, 'I know you have a long waiting list, but can you see this patient? She's in my office, scared to death, crying.' I saw her the next morning. I explained to this young lady what she had to do, how she had to eat, and she started that day.

"I just saw her for her three months follow-up. Her hemoglobin A1c was down to 6.1 - no longer in the diabetes range. She had lost 25 pounds. When I told her she was no longer diabetic, she was crying. I called my colleague over, and she started crying. I was crying. I literally felt like I had cured cancer. This girl has her whole life in front of her and it is not going to be spent on insulin, managing a chronic disease."

Such anecdotes are virtually incompatible with the conventional wisdom on diet and health - and they have been for half a century. In an era when obesity and diabetes epidemics worldwide are "slow-motion disasters," as former WHO director-general Margaret Chan described them, the diet Dr. Wolver prescribed to her young patient to reverse her diabetes clashes conspicuously with our widely held beliefs about healthy eating.

For starters, it's fat-rich and carbohydrate-restricted. As commonly prescribed and consumed, it features plenty of animal products, including red meat, bacon and butter, which means it's rich in animal fats and saturated fat. It includes plenty of leafy vegetables but is not "mostly plants" - as food activist Michael Pollan suggests - nor in any conventional way balanced. Whole grains, beans and legumes are not recommended. Of fruit, only berries make the cut. And it does not restrict calories; eating to satiety is advised.

This way of eating is now commonly known as a low-carbohydrate, high-fat (LCHF) diet. At its most extreme, it excludes almost all carbohydrates and is known as a ketogenic diet - keto for short. Many popular diets - paleo, South Beach, Dukan, Protein Power, Sugar Busters, Whole30 - are variations on the LCHF theme. And they're all widely considered by academic authorities to be fads, perhaps quackery. U.S. News and World Report, which publishes annual authoritative diet reviews, rates these diets the least healthy imaginable.

The hypothesis underlying LCHF thinking is that obesity is not caused by overeating - not calories in minus calories out - but by hormonal/regulatory phenomena triggered by easily digestible carbohydrates and sugars. By this logic, carbohydrates are uniquely fattening because they elevate levels of the hormone insulin, and insulin signals our bodies to store fat while inhibiting the use of fat for fuel. Carbohydrates literally make us fatter, and that makes us want to eat them to excess.

Hence, for a diet to successfully prevent or reverse obesity, it has to reduce insulin levels and so restrict these foods specifically. To fix any unresolved problem, let alone end an epidemic, the essential step is to correctly identify and remove the cause. And because obesity and the common form of diabetes, type 2, are so closely associated, carbohydrates are also the likely cause of diabetes. It's a simple hypothesis with a long history. Considerable textbook science supports it. But it has never been rigorously tested.

Anyone can do what I did, though, and experiment for themselves - see whether it does for you what it did for me. We need no clinical trials to tell us if a way of eating helps us attain and/or maintain a healthy weight. We need the trials to know if we're likely to live longer for doing so - do the benefits outweigh the risks? - or whether other diets, which might better suit our tastes, are safer still.

Twenty years ago, when I began my investigative reporting on this subject, perhaps a dozen North American physicians were openly prescribing these diets to their patients. A handful of the researchers I interviewed admitted to using these diets - "It's a great way to lose weight," a renowned Stanford University endocrinologist told me, "that's not the issue" - but they would not prescribe them for their patients. The risk was too great. Hence their thinking: Eat Atkins until losing the excess pounds, then stop, regain the weight, repeat.

Now, thousands of physicians around the world are eating this way and telling their patients to do the same. One significant difference is that they now believe these diets are inherently healthy, partly because research supports this position, partly because of their experiences - personal and clinical. They say they have little desire to return to a more conventional way of eating.

In October, the Huffington Post published a letter signed by more than 100 Canadian physicians, publicly acknowledging that this is how they eat and the dietary approach they prescribe to patients. "What we see in our clinics," they wrote: "Blood sugar values go down, blood pressure drops, chronic pain decreases or disappears, lipid profiles improve, inflammatory markers improve, energy increases, weight decreases, sleep is improved, IBS symptoms are lessened, etc. Medication is adjusted downward, or even eliminated, which reduces the side effects for patients and the costs to society. The results we achieve with our patients are impressive and durable."

With the conventional dietary guidelines, they added, none of this happens: "Patients remain diabetic and still need medication, usually in increasing dosages over time. Don't we say that type 2 diabetes is a chronic and progressive disease? It doesn't have to be this way. It can actually be reversed or put into remission. Of the patients that we treat with a low-carb diet, most will be able to get off the majority or all of their medications."

I've been cataloging these stories for my next book, so far interviewing more than 100 physicians and dietitians, all of whom have had what journalist Malcolm Gladwell calls a "conversion" experience in precisely this context. They were struggling with their own weight or frustrated, as Dr. Wolver was, that their patients were becoming fatter and more diabetic with each passing year. They were telling their patients to eat low-fat diets - mostly plants - control portion sizes and exercise. They were following that advice themselves. It wasn't working.

So they did what I would hope any intelligent person would do, and certainly my physician in these circumstances: They went searching for a better approach.

"Maybe it's not that my patients aren't following my advice," as Dr. Wolver described her thinking. "Maybe my advice stinks."

When they read about LCHF diets, they opted to self-experiment. When they discovered that the diet worked for them, they suggested cautiously it to their patients. When it worked for their patients, they became passionate about it.

These physicians repeatedly told me they could not "unsee" what they had witnessed - how well their patients were doing. For this reason, they can sound like evangelists ("I literally felt like I had cured cancer"). As more than one doctor told me, their discovery of a dietary means, one that was easy to follow, of preventing and treating obesity and diabetes - the disorders that overwhelm their practices - had made them excited again about practising medicine. They now believe they can make their patients healthy, which is why they became doctors in the first place, rather than doing what medicine had become to them: prescribing pills, treating symptoms, managing disease.

That medical organizations still consider LCHF diets to be fads, of borderline efficacy and likely to cause long-term harm, is less compelling to these doctors and dietitians than their own eyewitness testimony. For their patients and themselves, the benefits are clear and easy to quantify. Every time Health Canada and the U.S. Department of Agriculture proclaim in their dietary guidelines that a healthy diet must include grains and fruits, that meats should be lean, fat avoided, saturated fats replaced by polyunsaturated vegetable oils, it directly conflicts with what these physicians are seeing in their clinics and their lives. It makes their job, as they now see it, harder.

One way to think of this half-century-old controversy is as a conflict between hypothesis and experience. On the one hand, we have our ideas about the nature of a healthy diet, and on the other, we have what these physicians observe in their clinics and what happens to ourselves when we try different diets. The conventional wisdom on nutrition is dominated by the hypothesis that saturated fats cause heart attacks by raising cholesterol levels, specifically the "bad cholesterol" in low-density lipoproteins (LDL). Hence, eating polyunsaturated fats instead, from corn, soy or canola oil, will make us live longer. A related hypothesis is that red and processed meats cause chronic disease, so if we eat all- or mostly plant diets, we will live longer.

Physicians and dietitians are expected to base their diet advice on these hypotheses, but they get no meaningful feedback on their validity and efficacy. Should a patient or client die at 70 or live to 90, regardless of any changes in cholesterol levels, they will not know what role the low-fat diet played. By the same token, should I die tomorrow from a heart attack (knock on wood) or in my 100th year, my next of kin will not know if my unconventional high-fat diet shortened my life or lengthened it. (Critics of my nutrition work will insist my diet killed me prematurely, but they'll be guessing.)

Physicians embracing the LCHF philosophy believe these conventional healthy-diet hypotheses are refuted daily in their practices. After all, many of them and their patients lived by these guidelines while they got progressively fatter and sicker. Some had been vegetarians, even vegans, but the LCHF diet was the one that eventually worked.

More importantly, health can be quantified by numerous measures, not just LDL cholesterol. It also seems a reasonable bet that being healthier in the short term translates to health in the long term. Excess weight, poor blood-sugar control (i.e., prediabetes or diabetes) and high blood pressure are all major risk factors for heart disease - and all improve on LCHF diets.

So here's another way to ask the critical question: Can a diet that has so many beneficial effects be unhealthy because it elevates LDL cholesterol or allows for the conspicuous consumption of a processed meat such as bacon? An acquaintance of mine puts this question in the proper perspective by asking why her friends never criticized her diet when she weighed 380 pounds, but now, having lost 140 pounds in less than a year eating the LCHF way, they caution her about the supposed dangers of eating bacon.

Not that bacon or meat of any kind are required in LCHF diets, but as fat-protein combinations, they can be eaten freely. With a few exceptions - avocados and vegetable oils - plant-based foods come with carbohydrates as a significant source of available energy. Lower-carb, higher-fat versions of plant-based diets can be consumed, but they are unlikely to be as effective as the omnivorous or carnivorous alternatives.

I would find this conflict between hypotheses and clinical/personal observation easier to accept had the evidence supporting the hypotheses actually held up over the years. But quite the opposite has happened: the more research, the less compelling the evidence.

In 1965, for instance, based on a hypothesis that had yet to be tested, The New York Times quoted a Harvard nutritionist who said that for a physician to prescribe LCHF diets was "the equivalent of mass murder." In 1988, after the publication of $200-million-plus studies - the results of which were contradictory - the U.S. surgeon-general was blaming two-thirds of the two million annual deaths in the United States on the overconsumption of fat-rich foods and advertising the "depth of the science base" as "even more impressive than that for tobacco and health."

Now, decades later, the most recent unbiased review of this evidence - from the Cochrane Collaboration, an international organization founded to do such unbiased reviews - has concluded that clinical trials have failed to demonstrate any benefit from eating low-fat diets and so, implicitly, any harm from eating fat-rich foods. The 2015 review describes the evidence as only "suggestive" that avoiding saturated fat specifically may avert a single heart attack and says it's even "less clear" whether this would lengthen anyone's life.

Put simply, if this were a legal case, saturated fat would easily be acquitted. Any unbiased judge might even toss it out of court. This puts the history of LCHF diets into an entirely different perspective: remarkably effective medical interventions - both as treatment and preventive measure - that were rejected as quackery not because they were harmful but because they clashed with naive biases about the causes of obesity (gluttony and sloth) and the nature of a healthy diet.

Mainstream nutrition and obesity authorities will still insist they were right all along about the role of saturated fat and carbohydrates by denigrating LCHF diets as "fads," unworthy of serious consideration. But they can be considered fads only by ignoring the history of obesity and nutrition research and a fair amount of anthropology as well. My reporting suggests that these authorities are simply unaware of this history.

All human diets before agriculture were effectively carbohydrate-poor, and many, particularly in northern latitudes, were fat-rich. And until the industrial revolution, virtually all human diets were absent the highly refined grains and sugars that the LCHF philosophy specifies as the primary causes of obesity and type 2 diabetes.

As early as 1825, French culinary expert Jean Anthelme Brillat-Savarin, whose The Physiology of Taste is among the most famous books ever written about food, was arguing that carbohydrates cause obesity. In the 1860s, the LCHF diet became widely known as a Banting diet, after British undertaker William Banting, who wrote the first bestselling diet book based on his LCHF conversion experience. Variations on LCHF then spread from England to the European continent, embraced by German medical authorities as the most effective diets for reversing obesity.

By the early 1950s, physicians from some of the best medical schools in the world - Harvard, Stanford and Columbia, for instance - were publishing articles in the academic literature advocating for LCHF diets to treat obesity, advising obese patients to eat "as much as [they] like" of meat, fish, fowl, eggs, animal fats, cheese and green vegetables, while avoiding all carbohydrate-rich foods and beverages, particularly "all sweets." This is what Hilde Bruch, the leading mid-20th-century authority on pediatric obesity, believed. It's what Benjamin Spock taught in six editions and almost 50 million copies of Baby and Child Care, the bible of child-rearing from the 1950s onward. "Rich desserts," Dr. Spock wrote, and "the amount of plain, starchy foods (cereals, breads, potatoes) taken is what determines, in the case of most people, how much [weight] they gain or lose."

As for diabetes, the disease was once thought of as a carbohydrate-intolerance disorder because diabetics are unable to metabolize carbohydrates. The conventional dietary treatment was mostly to all meat. "Every attempt was made to avoid all carbohydrate," wrote University of Michigan physician Louis Newburgh, who published articles in 1920 and 1922 documenting the remarkable efficacy of LCHF diets in treating his diabetic patients. But the timing was off.

The 1921 discovery of insulin was life-saving for patients with the acute form of the disease, known now as type 1. Physicians quickly came to think of diabetes as an insulin-deficiency disorder to be managed over the patient's lifetime with insulin therapy and a diet that included liberal carbohydrates to balance out the insulin. Dr. Newburgh's LCHF diet seemed unnecessary and overly restrictive, what with the wonders of drug therapy.

By the 1960s and the rise of the hypothesis that dietary fat clogged our arteries, physicians were simply unwilling to contemplate high-fat diets as a treatment for anything, particularly disorders they believed could be brought under control if the fat people who suffered from them would merely exercise the self-control these lean physicians believed they did.

With Christmas dinner fast approaching - immediately followed by the season of diets and resolutions - the same question invariably returns: Is it really possible that I or anyone else will live longer as a lean person with bacon and butter than obese and diabetic without them? I think it's a good bet - I hope so - but my reporting and experience has biased me.

Neither anecdotal evidence nor the existing clinical-trial data can settle this. Nor can they tell us whether a vegan or vegetarian diet, absent refined grains and sugars, might be healthier still - perhaps ideally so. I doubt it, but again, I am biased. For some, such diets might certainly be easier to sustain, if for no other reason than they are consistent with valid ethical concerns about eating animals.

Many of these questions of diet and health simply require definitive experiments to settle - randomized, well-controlled trials. This may require the level of societal commitment that we use to justify spending billions of dollars on other scientific endeavours (atom smashers, for instance, or space telescopes) less directly relevant to human health. Until then, and before more guidelines based on failing or poorly tested hypotheses are promulgated, maybe the nutrition experts could actually talk to doctors and their patients - the eyewitnesses to the efficacy of LCHF diets - and learn why they find their anecdotal and clinical experiences so compelling.

Gary Taubes is the author of Why We Get Fat: And What to Do About It, The Case Against Sugar and Good Calories, Bad Calories: Fats, Carbs, and the Controversial Science of Diet and Health.