Crowdsourced Simulation

One of the challenges in creating a simulation model of a society is that it is complex, on several fronts:

1. There are many systems—finance, government, healthcare, education, etc.—each with its own domain of expertise.
2. There is no one correct way to model human behavior—anything we model in a computer has to be a gross simplification of realty. And we don't even know exactly how minds generate decisions.
3. Models are often created from a particular point of view, and can have unintentional (or intentional) biases and/or omissions
4. Models can have mistakes. Mistakes can be easy to spot when the correct behavior of the model is clear, such as simulating the mechanical stresses on a bridge, but not so easy to spot when modeling, for example, how people will respond to a change in tax policy.

I propose addressing these challenges via crowdsourcing. That is, in the spirit of Wikipedia and open-source software, providing a platform on which many people can contribute to a model.

Levels of contribution

There are several levels at which people could contribute to a model, listed below in order of increasing requirements on the skills of the contributor:

• Testing: Take an existing model and explore how it responds. This may entail examining metrics currently provided by the model, or adding new metrics.
• Scenarios: A scenario provides a set of initial conditions and external factors for a model. Scenarios may be forecast-oriented, in which case the initial conditions are likely to represent the present, and the external factors represent a possible future. Scenarios might also be education oriented, in which case the initial conditions and external factors might represent a period in real world history, or might be fictional to illustrate a point or test the model.
• System Structure: Simulation models are often made up of interacting objects, also known as agents or actors. These actors have some type of distribution in space (and perhaps time) for a given simulation. For example, the mortgage model has a population of households and banks, as well as a single national market for mortgages. An example of a structural change would be to change this model to have a separate mortgage market for each state.
• Behaviors: The actors in a model have a specific set of behaviors, such as buying and selling homes, and refinancing to get better terms or take money out of the home equity. To such a model, someone might want to add a behavior to the household that modeled life events (births, college, marriages, new cars, vacations, etc.) and use these life events to trigger refinancing actions that draw on home equity.
• Actors: Changing the arrangement or number of actors was mentioned above in “System Structure”. Another way to modify a model is to create new kinds of actors. For example, one might want to add a secondary mortgage market (where banks buy and sell mortgages from other banks).

Each of these levels of contribution requires tools with specific capabilities. I’ll explore ideas around these tools in future posts, and also welcome comments.

Version Control

One challenge that a crowdsourced simulation model poses is version control. The typical open-source software project is convergent—there may be multiple solutions proposed to a particular problem (like file system structure), but a single solution is chosen and built upon by others.

Wikipedia is different— any particular article is constantly changing, and while there is one current version, past versions are available as well. This presents challenges when trying to build something upon Wikipedia content, as the content can change underneath your edifice.

I’m finding it a bit more challenging to think of an example where a crowdsourced effort branches and is not convergent, but this is certainly conceivable in simulation models. For example, there could be two groups with different views on politics (socialism versus capitalism) that build different behaviors into their models. Or, there could be multiple domain specific models (e.g. mortgages and healthcare) which each dive into detail for their specific domain while simplifying or ignoring the other domain.

It seems like we will need to pick one of two approaches to “version control”

• Convergent: Some curatorial body will work to integrate the best thinking into a single version of “truth”. Or perhaps a small set of domain specific versions.
• Embracing diversity: In this case, there would be little or no effort at integrating separate models built on the same platform. Perhaps there would be a way to integrate the response of many different models to a specific scenario. One would still want some level of quality control when choosing models whose results are integrated.

The Power of Crowdsourcing

I believe a crowdsourced approach to a model of society is essential.

• Different domains each have their own experts,
• Different approaches to any single aspect of a model each have pros and cons
• Many sets of eyes can vet a model, uncovering biases, omissions and errors

There are some additional advantages that, in my experience, are very important if not essential is using models to effect changes in behavior:

• Transparency: People must understand a model before they can trust it. A model that can be tested and changed is transparent.
• Engagement: Interacting personally with a model, whether simply changing a few external factors or parameters, or building new behaviors, enhances people’s interest the model. It gets them thinking about it, talking about it, contributing to it and ultimately acting in the real world.
• Understanding: With engagement and interaction comes understanding. One of the best ways to learn is by “doing”, and one of the best ways to “do” is by building a dynamic model.

I welcome comments on crowdsourcing a simulation model, on this blog as well via email (karakots@gmail.com). Have you ever seen this? Have you tried it? Would you participate? How would you like it to work?

Skill Set for Simulation Designers

This article in the NY Times (ht Bill Parks) provides a concise and cogent description of the skills of successful simulation designers.

The profession draws on expertise in a number of areas and does not fit neatly into any single category... Simulation “overlaps engineering, math and computer science, but it isn’t the same as any one of those,” .... [S]kills include … an aptitude for “conceptualizing the world,” he said. Developing a simulation requires enough native intelligence to view a problem abstractly, research the issues and tease out the myriad key elements. Then they must be incorporated into a model in which they are poked, prodded and tweaked to reach useful solutions.

Two other very important skills for those who use simulations are noted.

- First, as one pares down the real world to its essential elements for a specific model, an equally important skill is understanding the of limits the model—what is left out, what is not well understood, and what is “uncalibrated” (this is, not numerically reconciled with real-world data).

- Second, one needs to be very clear in communicating the correct way in which to interpret results.

Too often there is too much confidence placed in the results of a model, as illustrated in this article, also in the NY Times (ht Franz Dill). In this case, an underestimation of the spread of the Swine Flu, which was officially declared a pandemic by the WHO last week.

Disappointment and back-pedaling occurs when too much confidence is placed in a model. This lowers the credibility of modeling, and also dampens the desire to apply resources to improving the model.

A model should not be an end in itself, but a conversation. It is only through the crucible of testing, critique, discussion and refinement that a model can converge on reality. Key to this process is engaging others to use a model themselves, understand how that model works, and be able to suggest (and ideally implement) changes to improve the model. This is an important goal of the platform we’re developing.

Economic Model Update

Though I have not posted for a while, a lot of work has been done on the model.

The first step was to build the simulation engine, which has been architected and implemented by Steve Noble. This engine is the platform within which we create populations of the objects: households (people), banks and homes, as well as the markets, government and external factors.

The next step was to create the behaviors for the various objects. We currently have a mortgage market up and running, in which the households request, and banks offer, mortgages. On a given simulated day, each bank evaluates all current household requests for a mortgage, based on the bank’s lending policy, and each requesting household evaluates any mortgages for which it is qualified. Each household and bank has its own unique way to evaluate a mortgage. I’ll describe these in a future positing.

We don't have houses in the model yet—we simply have the households request mortgages and, if the request is filled (the mortgage is preapproved), then the household has a probability of actually buying a house at the preapproved amount of the mortgage. The buying household then pays the down payment to the seller, and pays any points on the loan to the bank. The buyer then makes mortgage payments until the loan is paid off. The current loans are simple fixed interest rate loans.

While the addition of houses will allow for a supply-and-demand variation in housing prices, the current simple model already provides interesting behavior with respect to the business models of the banks. Banks make money on loan interest paid, but to make loans a bank needs deposits and has to pay interest on the deposits. These interest payment seat into the loan profits. The banks are competing for the households’ loan business, so they can’t charge too high an amount for loan interest. But if the deal on the mortgage is too “good” from the perspective of the borrower then the bank will get lots of business but lose money.

This example highlights that even when there is no risk of default in the loans, the banks still must balance offering attractive loans with making enough money to pay interest and turn a profit. If there is just one bank it is easy, but when there are multiple banks competing for the borrowers’ business it is not clear ahead of time which bank will win.

We have some more work to do to finish this first simulation, and plan to put it online within the next two to three weeks. Watch this space for an announcement when we do!

Meanwhile, here's a screenshot of the work in progress...

Market Signals & Financial Engineering

One of the side effects of building dynamic models is that you learn new things about the system you were modeling. I’m not an economist by training, so while what I’m learning is new to me, I’m not claiming it is new to everyone. Please do keep in mind, though, that an important goal of this modeling work is to help society, most of whom are not experts in economics (or energy or healthcare or …) develop an understanding of how society works and how to make it work better.

This positing at a blog I follow, Franz Dills’s The Eponymous Pickle, pointed to an article Re-thinking Risk Management: Why the Mindset Matters More Than the Model. This got me thinking about how financial firms think about risk.

The Path to the Insight

Since we will be modeling “structured finance” products like Mortgage-Backed Securities (MBS) and Credit Default Swaps (CDS), I need to understand what these are, how they work, and why they are desirable to the parties involved. This led me to the concept of price signals in markets.

An example of a price signal is the price of gas. When it is high, it sends a signal to drivers to drive less in the short term, and consider more efficient vehicles and public transportation in the long term. It is possible for government to send the same signal by a gas tax, even if the price of imported oil is low. The key point is that in a market, signals influence behavior.

Investments have two key signals, price and risk. These are traditionally unified—if an asset is perceived to be riskier, buyers of the asset will demand a higher return to offset the risk (think of “junk bonds”). The risk signal is set by a credit rating agency such as Moody’s, which assigns a rating such as AAA for the safest investments (such as a money market fund) to BBB for a junk bond.

In the article Regulatory Malfeasance and the Financial System Collapse, economist Joseph R Mason, Senior Fellow at the Wharton School , writes that it is important to understand “the terms and triggers of securitizations to recognize perverse incentives apparent in selling the AAA securities but keeping the risk.”

What structured finance products do, whether intentionally or not, is to obscure the risk signal associated with an investment. This occurs because risk is separated out from the underlying investment. When a MBS is insured by a CDS, the risk is apparently moved from the firm holding the MBS to the firm holding the CDS. But the decoupling or risk and return is an illusion, because the MBS and CDS are both linked to the risk of the underlying real asset—the mortgage. If the mortgage fails, the MBS loses value, and the CDS is supposed to pay. But why did the mortgage fail? Because the economy has problems, and these problems also affect the ability of the holder of the CDS to pay when the mortgage fails. So, while the risk signal of the MBS was changed via financial engineering, the underlying risk remained with the MBS.

The Insight

The insight is that financial risk obeys a kind of “conservation of matter” law. That is, “risk can neither be created nor destroyed”. So, while financial engineering can move risk around, it behooves anyone considering purchasing a structured finance product to understand exactly where the risk is being moved to, and to ensure that the destination of the risk is truly unconnected with the structured finance product being purchased. My hunch is that you can't ever guarantee decoupling, and that structured finance is essentially just an exercise in obfuscation. Once we have the model running, I look forward to tracking the flow of risk.

A note on Financial Engineering

The practice of creating structured finance products is called “financial engineering”. I am trained as an engineer, and grew up viewing engineering as synergistically combining elements from a palette of technologies to create a new capability. These new capabilities add value to the economy, because they either solve a problem that has value or enable some new activity or product, which also has value.

Financial engineering, in contrast, seems to only move things (e.g. risk) around without creating new value. One may argue that securitizing debt (e.g. MBS) allowed many more institutions to sell debt to individual people in the form of mortgages and credit cards, which expanded homeownership and consumer spending. But it seems to me that primary motivation behind securitizing debt was to have new financial products to trade, with a commission being made on every trade. Hosting the trading of assets is a great business model, because “the house always wins” in that every trade generates a commission, whether the buyers and sellers win or lose. In other words, securitization of debt doesn't add any real value to the world; it just allows the institutions that facilitate the trades to siphon value out as commissions.

That doesn't sound like engineering to me.

Modeling the Meltdown; Part 7: Five Easy Pieces

We currently envision five steps to creating the full mortgage meltdown model. As we implement the model, we’ll likely learn new things that revise how we go about these steps. Our initial plan is outlined below.

1. Mortgage Core: Households, financial firms (banks), homes and mortgages. Also the government and regulations it applies to the mortgage and banking system.2. Employment: Adds businesses that produce goods. Allows for the modeling expenses (buying goods) and income (employment) for households. (The new elements added in each step are highlighted in yellow).3. Investment: Adds a stock market. Each firm issues stock, and households and firms invest in that stock.4. Structured Finance: Adds the secondary debt market, where mortgage and non-mortgage debt is resold to other financial firms. These other firms securitize the debt they buy into structured finance products such as Mortgage-Backed Securities (MBS) and Collateralized Debt Obligations (CDO). Via a market for these vehicles, financial firms buy and sell these structured finance products.5. Financial Insurance: Additional financial firms, such as AIG, create insurance products such as Credit Default Swaps (CDS) to insure the MBS and CDO.
In the next post, we'll start designing the specific behaviors and relationships for the first step, the Mortgage Core model.

Modeling the Meltdown; Part 6: Transparency

One of the goals of our modeling effort is transparency. We want anyone to be able to understand how the models work, so that they can either be confident the model is correct and/or help improve the model. This series of posts is designed to help with transparency by illustrating the process by which we’re creating the model.

Assumptions
An important aspect of transparency is identifying assumptions, which typically means what is left out of the model. Here are some assumptions that we are starting with in this model:

Rent

• Don't model who owns a rented home
• Don't model change in rental rates during tenure of renter
• Assume sufficient supply of rentals

Homes

• No family owns more than one home
• Only one mortgage per home
• Ignore commercial real estate
• We’ll ignore growth in number of households and new home construction

Home Sales

• Ignore realtor’s fees when buying & selling a home

Mortgages

• Assume no “interest only” or “negative amortization” loans.
• Price appraisals not implemented yet
• Not modeling cost of marketing a mortgage to buyers/refinancers
• Assume no fraud (borrowers lying to lenders)
• Terms for a pre-approved mortgage do not vary while buyer is shopping for a house.

Savings

• All savings accounts can be withdrawn from at any time

Goods

• A single good is produced, (though it has variable consumption)
• No supply chain
• Ignore accounts receivable and payable

Stocks

• Stock price does not consider future earnings potential

Government

• No Taxes

As we identify additional simplifying assumptions, we will add them to this list. Also keep in mind that these assumptions can be explicitly addressed in future versions of the model, after we have the basics implemented.

Building in Steps
The full model will be rather complex—but this does not mean that it will be opaque. To help reduce the complexity, we will start simple and add to the model in steps. Each step will build on the previous step with an incremental amount of functionality. These steps are described in the next post.

Modeling the Meltdown; Part 5: Financial Model Basics

Because we are building an economic model, all proactive agents, and the government, will have a financial model which includes the two standard elements:

• Income statement: how much the agent is making and spending over a given period of time
• Balance sheet: the amount of assets and debts (stuff) an agent holds at any specific point in time

Here is a more detailed view of the income statement and balance sheet, in terms of the elements we are using in this model (I’ve omitted the structured finance elements for the moment to keep it simple):

Income statement

• Income
  • Salary (households), sales (firms), debt interest (financial-firms)
    
  • Stock dividends
    
  • Interest paid on cash deposits
  
  
• Expenses
  • Goods purchased (households)
    
  • Salaries paid to employees (firms)
    
  • Cost to produce a good (e.g. buying mortgages)
    
  • Debt servicing (principle + interest)
    
  • Paying interest on deposits (firm)
    
  • Rent
• Profit = Income – expenses

Balance Sheet

• Assets
  • Cash
    
  • Investments
    
  • Home value (Households)
    
  • Equipment value (Firm)
    
  • Goods inventory
  
  
• Liabilities
  • Debt balance (secured & unsecured)
  
  
• Equity = Assets – Liabilities
  • For a household, equity is the net worth
    
  • Firms will be considered 100% publicly owned, though they can buy their own stock. The fundamental determination of stock price will be equity/# of shares. There will also be a market effect on the stock price

Credit Worthiness
An important aspect of this model will be the ability of one agent to evaluate the credit risk of another agent. The balance sheet and income statements make this possible, because in general a credit rating is better (the risk that the debt will not be repaid is lower) when

• profit as a percent of income is higher
• the ratio of assets to liabilities is higher
• these ratios have been stable or increasing over time
• the agent has not defaulted (left debt unpaid) in the past

Next, we'll look at how the elements of this specific model will interact.