New studies show the complexity and importance of HIV epidemiological modelling
13 August 2012. Related: Epidemiology.
Nathan Geffen, CSSR
In Isaac Asimov’s Foundation science fiction series, which is set far in the future, mathematical modelling of human society has reached such a sophisticated level that the protagonists can predict the fall and rise of their civilization thousands of years into the future. The series’ main mathematician is even able to determine a set of interventions that will shorten the period of barbarism between the collapse of his society and the rise of the next one.
Alas, in 2012, we are barely able to predict the future trajectory of a single disease in one country a few years into the future.
It might seem unlikely that mathematical models can generate controversy that ultimately even involves the president of a country. Yet when the Actuarial Society of South Africa (ASSA) published their AIDS models in the early 2000s, public debate followed because contrary to the AIDS denialist views of then President Mbeki, the ASSA models showed that millions of people in South Africa had HIV, that hundreds of thousands of people were dying annually of AIDS, life-expectancy had plummeted and that, in the absence of antiretroviral treatment (ART), the worst was yet to come.
Mathematical models of epidemics estimate important information such as the prevalence, incidence and effect on life-expectancy of an epidemic disease at time-points for which there have been no direct measurements. They usually tell us about the state of an epidemic now and in the future under different scenarios.
It is seldom that people outside the field have a good understanding of the details of mathematical models. Models often seem mystical. Perhaps this is why there is a myth that epidemiological models are just mathematics and not based on real-world measurements.
But this is not true. Good models have a large number of parameters that have to be set using the best available peer-reviewed data, for example the risk of a sexual act resulting in HIV transmission, or the effect of ART on a person’s infectiousness. Once all the parameters are set models must be calibrated against reliable epidemiological data, so that their outputs match what is known about the epidemic. This is analogous –albeit much more complex– to calibrating a scale before you stand on it. You make sure that the scale points to zero, so that when you stand on it, it will not understate or exaggerate your weight. Likewise, a good mathematical modeller will make sure that if the countrywide HIV epidemic was measured in a reliable survey to be, say 9%, in 2001, that when the model is run, it calculates close to 9% prevalence for 2001. If it does not, then the modeller has to revisit the model’s parameters and calculations. It is hard work and good modelling is a highly skilled undertaking.
There is also a belief that models are entirely dependent on the assumptions and biases of the people who develop them, which is partially true, and therefore have no value, which is not true. A household budget is a mathematical model, and most readers of this article have no doubt done one. If done carefully, they are based on real-world measurements and they predict future expenditure quite well most of the time. Most of us find them very valuable. At the risk of hyperbole, the difference between a household budget and the most sophisticated mathematical models of the HIV epidemic is merely complexity.
A comparison of models
Today there are new controversies in HIV demographic modelling, albeit much more interesting and rational than the AIDS denialist response to the models of the early 2000s. In 2009 Reuben Granich and his colleagues published the results of their model in the Lancet. They predicted the HIV epidemic in South Africa could be eradicated by 2050 if universal voluntary testing and immediate treatment for all people with HIV were introduced.  But using different treatment-uptake assumptions, Wagner and Blower used the same model and reached different conclusions. They published the results of their model in a paper titled, “Voluntary universal testing and treatment is unlikely to lead to HIV elimination”. 
PLoS Medicine has published a set of articles that look at the cost and effectiveness of using antiretroviral treatment to reduce HIV incidence. These papers debate the assumptions, methodologies and conclusions of mathematical models and consequently the affordability and benefits of treatment as prevention.
One of these papers was co-written by the developers of 12 different epidemiological models, including the Granich one. Jeffrey Eaton, Timothy Hallett and colleagues explain, “Each of these models has predicted dramatic epidemiologic benefits of expanding access to ART, but models appear to diverge in their estimates of the possibility of eventually eliminating HIV using ART, the cost-effectiveness of increasing the CD4 threshold, for treatment eligibility, and the benefits of immediate treatment compared to treatment based on the current World Health Organization eligibility guidelines. Directly comparing the models’ predictions is challenging because each model has been applied to a slightly different setting, has used different assumptions regarding other interventions, has been used to answer different questions, and has reported different outcome metrics.” 
The aim of the research described in the paper was to systematically compare the 12 models by standardizing a set of antiretroviral treatment scenarios and reporting a common set of outputs. The intervention scenarios were consistently implemented across the different models with the purpose of controlling “several aspects of the treatment programme and isolat[ing] the effects of model structure, parameters, and assumptions …”
Three variables were controlled across the models and systematically varied: CD4 threshold for starting treatment, proportion of people eligible who access treatment and retention of people on treatment.
PLoS Medicine’s editor explained the methodology, “To exclude variation resulting from different model assumptions about the past and current ART program, it was assumed that ART is introduced into the population in the year 2012, with no treatment provision prior to this, and interventions were evaluated in comparison to an artificial counterfactual scenario in which no treatment is provided. A standard scenario based on the World Health Organization’s recommended threshold for initiation of ART, although unrepresentative of current provision in South Africa, was used to compare the models.”
The methodology of the twelve models varied greatly. For example:
- They used two different modelling methods. Four used microsimulation. In these models, each individual in a population is simulated. Random events that affect their risk of HIV infection are applied to them. This is the most computationally intensive of the modelling methods. Microsimulations can take hours or even days to produce results. The remaining eight models, “stratify the population into groups according to each individual’s characteristics and HIV infection status and use differential or difference equations to track the rate of movement of individuals between these groups.”
- Ten of the models explicitly provide for both sexes and heterosexual HIV transmission.
- Six of the models include age, but the extent “to which age affects the natural history of HIV, the risk of HIV acquisition, and the risk of HIV transmission varies amongst these.”
- One model simulates the HIV epidemic in Hlabisa, KwaZulu-Natal, while the remaining models deal with the national South African epidemic.
The models were compared under three different CD4 cell count thresholds: CD4 count ≤ 200 cells/mm3, ≤350 cells/mm3 and treatment for all irrespective of CD4 count. The proportion of eligible individuals who eventually initiated treatment was also varied as follows: 50%, 60%, 70%, 80%, 90%, 95%, and 100%. So was the percentage of people retained on treatment after three years, excluding those who died, as follows: 75%, 85%, 95%, and 100%.
Outcomes of the different models
The estimates of adult male HIV prevalence in South Africa in 2012, if there was no ART, ranged from 10% to 16% across the models. Female prevalence ranged from 17% to 23%. Male incidence ranged from 1.1 to 2.0 per 100 person-years and female incidence ranged from 1.7 to 2.6.
Under the scenario where no treatment is provided, the models varied in their predictions about the future trajectory of the epidemic, ranging from almost no change in HIV incidence to a 45% reduction in incidence over the next 40 years. All the models predicted that treatment would reduce incidence by a large percentage over the no treatment scenario. Their estimates varied, but by a narrow range. For example, if 80% of HIV-positive individuals started treatment a year after their CD4 count drops below 350 cells/mm3 and 85% remain on treatment after three years, the models’ estimates of the drop in incidence ranged from 35% to 54% lower eight years after the introduction of ART compared to not providing ART at all. The number of person-years of ART per infection averted over eight years ranged between 5.8 and 18.7. As expected, the further into the future the models went the more they diverged. This scenario, incidentally, reflects current WHO treatment guidelines coupled with the UNAIDS definition of universal access.
Effect of ART on incidence in South Africa
The authors then did a separate analysis using seven of the models to determine the effect of the actual ART rollout in South Africa on incidence by comparing it with a no-treatment scenario. Models either used their own existing calibrations of the number of people on ART in South Africa or were calibrated using estimates of the number of adults starting and on ART in each year from 2001 to 2011.
All of the models predicted that ART has reduced incidence. The estimates ranged from 17% to 32% lower in 2011 than in the absence of ART. Interestingly, the models give widely different estimates of the effect of ART on prevalence. For example a model by Jeff Eaton and colleagues, as well as Leigh Johnson’s STI-HIV model, estimate that prevalence is 8% higher than it would have been without treatment, while the Granich model and another by Christophe Fraser and colleagues calculates that massively reduced incidence results in no net change in prevalence at the current point in time. It is worth recalling that an increase in prevalence does not mean a failed response to the HIV epidemic. On the contrary, the only way prevalence can decrease is if more people die than are infected. Since ART keeps people alive, it is unsurprising that several models predict an increase in prevalence. Incidence, not prevalence, is the measure of the success of prevention efforts.
Test and treat
The impact on incidence of a CD4 threshold of 200 cells/mm3 versus 350 cells/mm3 versus treatment with very high rates of HIV screening and removal of CD4 eligibility–the latter known as the test and treat approach–were also compared across the 12 models, but there were not consistent results here. Some models showed that that moving from 200 to 350 would not make a substantial difference, but that moving from 350 to treating everyone did, while others found that moving from 350 to treating everyone made little difference.
In an intervention treating all HIV-positive adults with 95% access and 95% retention, only three of nine models predicted that HIV incidence would fall below 0.1% per year by 2050, the virtual elimination threshold proposed by Granich and colleagues.
Explaining the differences between models
The authors put forward three hypotheses to try and explain the differences between their models. These were (1) differences in the fraction of transmission that occurs after people become eligible for ART in the no-treatment scenario, (2) differences in how effective ART is at reducing transmission and (3) different assumptions about what happens to patients who drop out of care. These hypotheses were tested and only accounted for some of the differences in model outcomes.
Although the models estimates diverge, collectively they help policy makers and provide tentative estimates of how successful antiretroviral treatment will be at reducing incidence. Also, they were not all designed to answer the identical questions.
The effectiveness of treatment as prevention is a question that will have to be answered more definitively with clinical trials as well as observational studies of actual practice. Over the next few years, cluster controlled trials in South Africa, Zambia, Tanzania and Botswana will hopefully provide these answers.
In the same PLoS Medicine collection, there is an interesting debate between Kimberly Powers, William Miller, and Myron Cohen on the one hand, and Brian Williams and Christopher Dye on the other, followed by a commentary by Christophe Fraser. One side argues that widespread treatment will have a massive effect on HIV incidence, while the other argues that it will be compromised by the high transmission rate during primary infection. Interestingly, Myron Cohen who was the lead investigator on HPTN 052, the trial that showed a 96% reduction in HIV incidence amongst sero-discordant couples if the HIV-positive partner was put onto early treatment, takes the view that high levels of transmission during early infection will “compromise the effectiveness of HIV treatment as prevention.” 
The future of mathematical modelling
It is important to realize that disease modelling at the level of sophistication seen in these models is a relatively new field, made possible by the tremendous increase in computer power over the last few decades. Microsimulations in particular stretch the capabilities of even today’s computers and computer programmers. The widely different methodologies and assumptions used should be seen as the pioneering efforts in a new science. Hopefully over time, and as the predictions of models are compared to what actually happens, modellers will be able to identify techniques that are robust and standardize the science. Just as the 95% confidence interval and the correspondence of a p-value less than 0.05 with significance are part of a standard part of medical statistics today, similar standard concepts might emerge in the modelling field. And just as R and STATA are standard software tools used by the vast majority of medical statisticians, so there will hopefully one day be standard tools for both deterministic and microsimulation mathematical models.
An effort to standardise modelling is already underway. Wim Delva and colleagues have published an article in the same PLoS Medicine collection which summarises extensive discussions between mathematical modellers. They describe nine principles for mathematical modellers and those who depend on HIV models to make policy decisions.  The (edited) principles are:
- The model must have a clear rationale, scope and objectives.
- The model structure and its key features must be explicitly described.
- The model parameters must be well-defined and justified.
- The way the model has been calibrated must be explained and justified.
- The model’s results must be clearly presented including uncertainties.
- The model’s limitations must be described.
- The model must be contextualised. In other words previous studies must be referenced and the similarities and differences must be explained. Differences in the results of the model and previous models must be described.
- The model must provide epidemiological impacts that can be used for health economic studies.
- Models must be described in clear language.
These principles surely apply to all disease modelling, not just HIV.
Disease modeling is important. Models help us understand the relative contribution of different factors to the present state of the epidemic and they give us some understanding of what will happen in the future under different interventions. They are valuable for making policies with short and medium-term impacts. Longer-term projections, such as to the year 2050, are less useful given that so many unpredictable technological and demographic changes are likely to occur over such a long time. Besides their practical value, disease modelling is a fascinating theoretical field with some elegant mathematics and computer algorithms.
- Granich RM et al. 2009. Universal voluntary HIV testing with immediate antiretroviral therapy as a strategy for elimination of HIV transmission: a mathematical model. Lancet 373: 48–57. doi:10.1016/S0140-6736(08)61697-9.
- Wagner BG and Blower S. 2009. Voluntary universal testing and treatment is unlikely to lead to HIV elimination: a modeling analysis. doi:10.1038/npre.2009.3917.1
- Eaton JW et al. 2012. HIV Treatment as Prevention: Systematic Comparison of Mathematical Models of the Potential Impact of Antiretroviral Therapy on HIV Incidence in South Africa. PLoS Med 9(7): e1001245. doi:10.1371/journal.pmed.1001245
- Cohen M et al. 2012. HIV Treatment as Prevention: Debate and Commentary—Will Early Infection Compromise Treatment-as-Prevention Strategies? PLoS Med 9(7): e1001232. doi:10.1371/journal.pmed.1001232.
- Delva et al. 2012. HIV Treatment as Prevention: Principles of Good HIV Epidemiology Modelling for Public Health Decision-Making in All Modes of Prevention and Evaluation. PLoS Med 9(7): e1001239. doi:10.1371/journal.pmed.1001239.
Thanks to Leigh Johnson, Jeffrey Eaton and Timothy Hallett for their advice.