Monday, May 23, 2016

ABC in Helsinki

Last week I went to a conference on ABC (Approximate Bayesian Computation) that was held on the Silja Symphony cruise ship between Helsinki and Stockholm.  Thank you to Michael Gutmann and the other organisers for the opportunity to go on an exotic nordic odyssey with a stimulating scientific programme!

I came to the conference as something of an outsider, as I have not used ABC in my research so far.  The main reasons I am interested in the area is that my supervisor Richard has done some work on ABC, with applications mainly focused on population genetics.  And a lot of people who work with differential equation models are attracted to it.

I was interested in answering the following questions.  When is ABC the preferred method compared to other statistical methods?  And what situations is the use of ABC is to be avoided?

The short answer to the first question is that ABC is used when the likelihood of the model is not available.  This may be because the likelihood is an intractable integral.  For example, in population genetics the likelihood can only be obtained by integrating over the space of phylogenetic trees.  As an aside, there is an alternative method for the population genetics example, which is the pseudo-marginal method.  It is not clear to me whether ABC or pseudo-marginal methods are preferable for this example.

As well as situations where the likelihood is intractable (another example of this is the transition density in nonlinear SDEs), there may be situations where it is not possible to write down an expression for the likelihood at all.  I have not encountered any situations like this myself, although I am told that a lot of agent-based or rule-based models fall into this category.  In Reading there are people using these kind of models to describe population dynamics of earthworms and the interaction with pesticides.

Several of the conference talks (Richard Everitt, Richard Wilkinson and in an indirect way, Michael Gutman) focused primarily on the use of ABC in expensive simulators, such as climate simulators or epidemiology models.  These talks mainly confirmed my view prior to the conference that expensive simulators should be avoided wherever possible because there are severe limitations on what you can do for parameter inference.

On a more positive note, there were examples in the talks where ABC was made tractable in models with 1 or 2 unknown parameters by combining ABC with other methods such as bootstrapping (RE) and Gaussian Processes (RW and MG).   The basic idea, is that if we knew the probability distribution of the model's summary statistics as a function of the unknown parameters, we would be done.  Tools like bootstrapping and Gaussian Processes can be used to estimate the summary statistics' probability distribution.  They can also quantify uncertainty in estimators.  This is not ideal since the relatively small sample size for these estimators means that the error in the estimators may be quite large and difficult to quantify accurately.  However if you are only interested in classifying parameter sets as plausible vs non-plausible, or you only need estimates of mean parameter values, you may not need that many samples.

It is unclear to me what you would do when you have an expensive simulator with more than 2 unknown parameters.  I am not sure that the methods presented in the conference would work that well in that setting, without (i) a lot of computing power, and/or (ii) strong assumptions on the probability distribution of the summary statistics as a function of the parameters.

One area that was not covered at the conference but that I would like to know more about is what to do when you have a deterministic simulator, like an ODE model.  I have come across this situation in the literature, for example in the work of Michael Stumpf's group, where ABC has been used.

Suppose that a deterministic simulator is a perfect model of the process we were interested.  Then repeated samples from the true process should yield exactly the same data.  Furthermore, we should be able to identify a parameter set that perfectly reproduces the experimental results.  (Or multiple parameter sets if the model is non-identifiable.)  In practice, a far more common situation is that repeated samples from the true process give varied results even though the process model is deterministic.  With a deterministic simulator there is no way of accounting for this variability.  Two options in this situation are (i) introduce some stochasticity into the model, for example observation noise, (ii) define ranges for the summary statistics so that parameter values are accepted if the deterministic simulation results in summary statistics that fall within the pre-specified ranges.

If we choose option (i), I would then suggest using a likelihood based approach, and if that didn't work, then trying ABC.  If we choose option (ii), this fits within the ABC framework.  However in the standard ABC framework, the effect of reducing the tolerance on the posterior should diminish as the tolerance goes to 0.  I.e. for small tolerances, dividing the tolerance by 2 should have a very small effect on the credible regions for the parameters.  If you have a deterministic simulator, the effect of reducing the tolerance will not diminish.  I haven't got all this worked out rigorously, but intuitively it seems like dividing the tolerance by 2 will always decrease uncertainty by a factor of 2.  Furthermore the limit (tolerance = 0) is singular with respect to all non-zero tolerance values.

A quick google scholar search shows that Richard Wilkinson has thought about these issues.  The gist of what he is saying seems to be that rather than treating the tolerance as a parameter that controls the accuracy of the approximation, it can be thought of as a parameter in a probabilistic model for the model error.  If anyone knows of any other work that discusses these issues, please let me know!  As you can see I am still rather a long way from being able to provide practical guidance on the use of ABC.

Monday, May 9, 2016

Parallels between finance and academic research environments

I recently went to an event titled 'Heart of the City: What is Really Wrong with our Financial System?' where John Kay and Joris Luyendijk were speaking.  John Kay is an economist and columnist for the Financial Times.  Joris Luyendijk is an anthropologist by training who has done some financial journalism for The Guardian.

It was interesting learning about the workings and culture in the City, and I also found myself wondering about parallels in other walks of life.

John Kay emphasised that traditionally financial services have provided useful services by enabling capital accumulated by one individual or corporation to be borrowed and used for investment by another individual or corporation.  In recent years this idea of banking as a service has been lost as banks have sought to maximise short-term profits by trading worthless bits of paper for a lot of money and taking a cut each time (this is somewhat simplistic - John Kay explained it better!)

While it is useful to establish what is going on the financial sector, it is also important to understand why this is happening.  What are the causes for the unhealthy changes seen in finance?  Joris Luyendijk's answer to this was that liquidity has replaced of loyalty.  By loyalty he was referring to the loyalty that the employees of city banks have for the companies they work for.  And by liquidity he was referring to the volume of transitions between banks made by city workers.  His view was based on a fascinating collection of evidence obtained from interviewing over 200 city workers, which he wrote up as a book, "Swimming with Sharks."

For example, it is apparently very easy to lose your job in the city if you do not conform to what your manager asks of you, or if you miss the ambitious profit targets that banks set.  People are terrified of losing their jobs and this means that they are willing to compromise on almost anything, whether that is working extremely long hours, or not blowing the whistle on other's malpractice.  This leaves very little time for thinking about whether the bank's end-product has genuinely been of service to the bank's customers.

For me, all this raises some interesting questions for academic research.  Should academic research be thought of as a service?  If so, who is it a service for?  I have met several academics who see their work primarily as finding out about interesting things for their own sake.  This view is sometimes justified by saying that historically many important advances have come about by exactly those type of people being free to pursue their interests.  Personally I think it is useful to think of academic research as serving some purpose, as this helps to direct where effort should be applied.  However I do acknowledge that it is not always possible to identify in advance which discoveries will lead to the greatest impact.

In terms of liquidity, there are some shortcomings to the current academic research system.  Academics can often be under a lot of pressure to produce short-term results in the form of publications in high-profile journals, so that they can win research funding and permanent positions.  This can sometimes get in the way of longer-term research goals.

However I think on the whole I think there should be loyalty and liquidity in academia.  It is a good thing for academic researchers to move between research groups, picking up different tools and perspectives in different places.   And it is also good  for academics to have long-term affiliations to a group or a particular academic sub-community to allow for relationships to develop and for paths to be fully explored.