The following analogy is based on one used by colleague Prof Retsef Levi. Suppose we want to find out whether runners in a 40 km (that's 26-mile) marathon are more likely to finish the race if they are given a special vitamin drink during the race. To do this we set up a drinking station at 20 km where runners can pick up the drink if they wish. Suppose 200 runners start the race and we observe the following:
It seems clear that taking the drink increases the chances of completing the race. But this may not be the case.
The problem is that in marathons many runners drop out before the 20 km mark. That means that such runners not only fail to complete the race but also fail to reach the drink station (and so are recorded in the 'no drink' category). This means the summary data above may be 'hiding' something like the following data on such runners:
This analogy is important because there has been much recent furore over conflicting government reports about whether the Covid vaccine is safe and effective for pregnant women and it turns out that similar statistical 'illusions' may be at play when comparing pregnancy outcomes of 'vaccinated' against 'unvaccinated' women. This is especially true of outcomes relating to whether the pregnancy resulted in the birth of a healthy baby. Ultimately what people really want to know is: if there are 100 pregnant women, how many end up with a healthy baby? Many of the studies over-complicate things and fail to answer that question. Imagine if we simply make the following replacements to the above marathon example:
- Starting marathon runners --> newly pregnant women
- Runners who complete the race --> those who delivered a healthy baby
- Vitamin drink at 20km --> vaccine at 20 weeks into the pregnancy
Of course, this is all hypothetical and over-simplified because pregnant women might get the option of vaccination at many different stages both before and during pregnancy and we have to take such information into account in order to arrive at a suitable risk assessment. But it turns out that this 'survivor bias' problem is still highly relevant to the real-world studies and data.
I have already explained in a previous article why the UKHSA's regular vaccine surveillance reports are likely to be overestimating the safety of the vaccine because, instead of comparing the 'never vaccinated' with different categories of vaccinated women, they lump together the never vaccinated with those vaccinated pre-pregnancy into a single 'no dose in pregnancy' category and compare these with the 'one or more doses in pregnancy' category. Here is an example of one of their graphs from the most recent report (and note this does not tell us anything about the chance a pregnant woman will give birth to a healthy baby since it deals only with still births** and not miscarriages in early pregnancy).
Indeed, those claiming that the vaccine is safe and effective for pregnant women point to studies such as this which even seem to show the miscarriage and still birth rates of vaccinated women to be lower than that of unvaccinated women. But these studies do not properly factor in the stage during pregnancy when the vaccine was taken.
To understand the scale of the problem, consider the following hypothetical scenario:
Using the UKHRA classification, suppose that we observe 200 pregnant women of whom 100 have 'no doses in pregnancy' and 100 have 'one or more doses in pregnancy'. For simplicity we will refer to any miscarriage** or still birth** outcome collectively as a 'foetal death'**. Suppose we observe the following hypothetical data in this group:
But we need to take account of the confounding effect of women who are first vaccinated late in pregnancy. For simplicity, instead of considering the normal three trimesters of pregnancy we will consider just two stages 'early' (0-20 weeks) and 'late' (21-40 weeks), and we also need to properly distinguish the never vaccinated from those vaccinated at least one once before or during pregnancy, i.e. we need data for the following separate categories of vaccinated:
- A: never vaxxed
- B: vaxxed pre pregnancy only
- C: vaxxed both pre pregnancy and early pregnancy (i.e. 0-20 weeks) or early pregnancy only
- D: vaxxed both pre and/or early (i.e. at least once before 20 weeks) and late (i.e. after week 20)
- E: vaxxed late only (i.e. after week 20)
The following hypothetical data produces the same aggregate results (shown in bold) as the above table (there are no tricks or sleight of hand - the excel spreadsheet is here).
While the aggregated results for 'no dose' in pregnancy (26% ) against 'one or more dose' (18%) is unchanged, look at the key results highlighted in yellow which were totally obfuscated from the aggregate data:
- The never vaccinated (A) have a lower early foetal death rate 20% than either of the two vaccinated categories they can be directly compared with, namely those vaccinated pre-pregnancy only (B) 24% and those vaccinated pre and/or early pregnancy only (C) 26%. The never vaccinated also have a lower overall foetal death rate 24% than either of the two categories B and C (32% and 34% respectively).
- The never vaccinated (A) have a lower late foetal death rate 5% than each of the different categories of vaccinated, (B) 11%, (C) 12%, (D) 11% and (E) 7%
guarantees a statistical illusion of effectiveness even in a placebo. Similarly, those suffering an adverse reaction (including death) shortly after first vaccination are also often put in to the unvaccinated category which guarantees a statistical illusion of safety even in a placebo.
Conclusion: While the example uses only hypothetical data, it illustrates the limitations of the real-world studies claiming safety and efficacy of vaccines for pregnancy failures. For rigorous safety assessment, data on pregnancy outcomes require proper categorization of the vaccinated and unvaccinated and must include the data for each pregnancy phase (ideally 3 trimesters, but at least 'early' and 'late'). Most real-world studies claiming vaccine safety are based only on aggregated data, and as such any safety claim is likely to be a statistical illusion that may be hiding strong evidence of lack of safety.
** Note that the terms 'miscarriage', 'still birth' and and 'foetal death' - all have different precise definitions timing wise, and the precise definitions vary between jurisdictions. It is best perhaps to consider them collectively as "deaths in the unborn".