Crowdsourced Simulation

One of the challenges in creating a simulation model of a society is that it is complex, on several fronts:

1. There are many systems—finance, government, healthcare, education, etc.—each with its own domain of expertise.
2. There is no one correct way to model human behavior—anything we model in a computer has to be a gross simplification of realty. And we don't even know exactly how minds generate decisions.
3. Models are often created from a particular point of view, and can have unintentional (or intentional) biases and/or omissions
4. Models can have mistakes. Mistakes can be easy to spot when the correct behavior of the model is clear, such as simulating the mechanical stresses on a bridge, but not so easy to spot when modeling, for example, how people will respond to a change in tax policy.

I propose addressing these challenges via crowdsourcing. That is, in the spirit of Wikipedia and open-source software, providing a platform on which many people can contribute to a model.

Levels of contribution

There are several levels at which people could contribute to a model, listed below in order of increasing requirements on the skills of the contributor:

• Testing: Take an existing model and explore how it responds. This may entail examining metrics currently provided by the model, or adding new metrics.
• Scenarios: A scenario provides a set of initial conditions and external factors for a model. Scenarios may be forecast-oriented, in which case the initial conditions are likely to represent the present, and the external factors represent a possible future. Scenarios might also be education oriented, in which case the initial conditions and external factors might represent a period in real world history, or might be fictional to illustrate a point or test the model.
• System Structure: Simulation models are often made up of interacting objects, also known as agents or actors. These actors have some type of distribution in space (and perhaps time) for a given simulation. For example, the mortgage model has a population of households and banks, as well as a single national market for mortgages. An example of a structural change would be to change this model to have a separate mortgage market for each state.
• Behaviors: The actors in a model have a specific set of behaviors, such as buying and selling homes, and refinancing to get better terms or take money out of the home equity. To such a model, someone might want to add a behavior to the household that modeled life events (births, college, marriages, new cars, vacations, etc.) and use these life events to trigger refinancing actions that draw on home equity.
• Actors: Changing the arrangement or number of actors was mentioned above in “System Structure”. Another way to modify a model is to create new kinds of actors. For example, one might want to add a secondary mortgage market (where banks buy and sell mortgages from other banks).

Each of these levels of contribution requires tools with specific capabilities. I’ll explore ideas around these tools in future posts, and also welcome comments.

Version Control

One challenge that a crowdsourced simulation model poses is version control. The typical open-source software project is convergent—there may be multiple solutions proposed to a particular problem (like file system structure), but a single solution is chosen and built upon by others.

Wikipedia is different— any particular article is constantly changing, and while there is one current version, past versions are available as well. This presents challenges when trying to build something upon Wikipedia content, as the content can change underneath your edifice.

I’m finding it a bit more challenging to think of an example where a crowdsourced effort branches and is not convergent, but this is certainly conceivable in simulation models. For example, there could be two groups with different views on politics (socialism versus capitalism) that build different behaviors into their models. Or, there could be multiple domain specific models (e.g. mortgages and healthcare) which each dive into detail for their specific domain while simplifying or ignoring the other domain.

It seems like we will need to pick one of two approaches to “version control”

• Convergent: Some curatorial body will work to integrate the best thinking into a single version of “truth”. Or perhaps a small set of domain specific versions.
• Embracing diversity: In this case, there would be little or no effort at integrating separate models built on the same platform. Perhaps there would be a way to integrate the response of many different models to a specific scenario. One would still want some level of quality control when choosing models whose results are integrated.

The Power of Crowdsourcing

I believe a crowdsourced approach to a model of society is essential.

• Different domains each have their own experts,
• Different approaches to any single aspect of a model each have pros and cons
• Many sets of eyes can vet a model, uncovering biases, omissions and errors

There are some additional advantages that, in my experience, are very important if not essential is using models to effect changes in behavior:

• Transparency: People must understand a model before they can trust it. A model that can be tested and changed is transparent.
• Engagement: Interacting personally with a model, whether simply changing a few external factors or parameters, or building new behaviors, enhances people’s interest the model. It gets them thinking about it, talking about it, contributing to it and ultimately acting in the real world.
• Understanding: With engagement and interaction comes understanding. One of the best ways to learn is by “doing”, and one of the best ways to “do” is by building a dynamic model.

I welcome comments on crowdsourcing a simulation model, on this blog as well via email (karakots@gmail.com). Have you ever seen this? Have you tried it? Would you participate? How would you like it to work?