Mark Transtrum > Differential Geometry and Sloppy Models
## Differential Geometry and Sloppy Models## BackgroundAs a new graduate student looking for a project, I was introduced to Sloppy Models by my then-to-be advisor, Jim Sethna. Jim's group had been studying what they called "Sloppy Models," which are multi-parameter models with an extreme insensitivity to large-scale fluctuations in many parameter combinations. A good example of a sloppy model comes from systems biology. Protein signaling models often have many unknown parameters corresponding to things like reaction rates and Michaelis-Menten constants describing the complex interactions of a network. In fact, Jim's group had found that systems biology models were almost universally sloppy. They also observed sloppiness in such diverse models as quantum monte carlo variational wave functions, radioactive decay, insect flight, particle accelerators, and many more. For more information about sloppy models, you can visit Jim's webpage here, which has a lot of interesting information about sloppy models. ## Is Sloppiness Intrinsic?What intrigued me about sloppy models, was their apparent ubiquity in multi-parameter modeling. Surely there must be some unifying principle tying these very different models together.
On the other hand, there seemed to be a very convincing
argument (to me anyway) that sloppiness couldn't be anything
profound. Sloppiness is identified by considering the
sensitivity of the model behavior to changes in the
parameters. This is measured by calculating derivatives of
the model with respect to parameters, assembling these
derivatives into a matrix (i.e. the Fisher Information Matrix)
and calculating its eigenvalues. If the eigenvalues span
several orders of magnitude (typically more than 6) the model
is identified as sloppy. However, suppose we take a sloppy
model with parameters θ
Although Jim's group identifies sloppiness from the
eigenvalues of the Fisher Information (which depend on the
parameterization) Jim speculated that these models also had
common
## Discoveries
Studying sloppy models using differential geometry turned out
to be very insightful. There were, in fact, several intrinsic
features that sloppy models had in common. The models'
manifolds are typically Not only did we observe bounded manifolds in our sloppy models with similar curvatures, we were able to show, using theorems from interpolation theory, that such boundaries and curvatures should be a very general feature of multi-parameter models as long as the model predictions are relatively simple. This leads us to think of models as generalized interpolation schemes. This perspective helps explain the ubiquity of sloppy models. Model behavior can typically be constrained with just a few parameter combinations corresponding to the main features of the behavior. The remaining parameters combinations are irrelevant. ## Practical ApplicationsThe problem described above sounds very vague and abstract. However, it turns out to have many useful applications. As a graduate student, I worked to improve algorithms for fitting models to data. Jim's group had found that existing algorithms would struggle to find good fits. Their experience suggested the problem was closely tied to sloppiness: the insensitivity of the model to some parameter combinations seemed to give the algorithms fits. Often, they would require a lot of human guidance in order to converge. Since these models are often very large and expensive to compute, the entire fitting process could take a long time. Our goal was to use the parameterization-independent approach of differential geometry to motivate algorithms to improve the fitting process. More generally, understanding the differential geometry of sloppy models tells us a lot about other, related problems. It can tell us a lot about the cost surface which in turn can be used to select methods of sampling in an MCMC calculation. It can be used to design experiments to more accurately infer parameters. It leads to a method of coarse-graining models with many parameters to describe simple emergent behaviors. Many of these applications we did not anticipate, but taken collectively suggest that thinking about modeling as a geometric problem is a very powerful and useful approach! Last Modified: 14 August 2012 |