Complex Systems Modeling and Networks Summary of NECSI Course CX202 11 February 2009 John M. Linebarger, PhD Sandia National Laboratories Interactive Systems.

Complex Systems Modeling and Networks

Summary of NECSI Course CX202

11 February 2009

John M. Linebarger, PhDSandia National Laboratories

Interactive Systems Simulation & Analysis (06344)

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration

under contract DE-AC04-94AL85000.

Outline

• Overview• What are Complex

Systems?• Complex Networks

– Metrics– Types of Networks

• Modeling and Simulation of Complex Systems– How to Model– Types of Models

• Further Reading• Class Project• Overview of Design for

Tractable Analysis (DTA) framework

Overview

• I attended a Complex Systems Modeling and Networks (CX202) course from 23-27 June 2008– I used all 32 of my FY 2008 Strategic Education Initiative

(SEI) hours on the course; the balance was covered by the Systems Analysis LDRD.

• The course was held at the New England Complex Systems Institute (NECSI) on the campus of the Sloan School at MIT, in Cambridge, MA

• The course prerequisite is CX201, Complex Physical, Biological and Social Systems– I fulfilled the prerequisite by reading Making Things Work

by Yaneer Bar-Yam, and Linked by Albert-László Barabási, before the course began

• The follow-on course is CX203, Methods for the Study of Complex Systems, which is mathematically-oriented

Overview (cont’d)

• Over 30 attendees from all over the world, predominately PhD students and academics– Only three or four people from “industry”– My project group consisted of a Norwegian PhD student, a

South Korean health care researcher (working in the Boston area), an Iranian PhD student (studying in Maryland), and a Bulgarian PhD student (studying in France)

• The course was divided into two parts – The first part was complex network analysis, taught by Dan

Braha of NECSI and UMass Dartmouth – The second part was modeling and simulation of complex

systems, taught by Hiroki Sayama of NECSI and SUNY Binghamton

• Project groups met each evening, often until quite late, to work on their course project

• Projects (and the associated papers) were presented to the class on Friday, followed by the final exam for the course

NECSI Course Participants

What are Complex Systems?• Complex systems are characterized by emergent, self-

organizing, collective behavior– Such emergent collective behavior often arises due to the

combination of simple individual behavior patterns

• The components of a complex system are interdependent• Complex systems exhibit multiscale variety

– The structure of the system differs depending on the level at which it is viewed

– A common pattern is a mixture of competition and cooperation at different levels of the system

• Complex systems arise due to evolutionary, not deterministic, processes– Engineered systems should be implemented this way too

• Examples of complex systems include the health care system, the education system, and recent military conflict

From Making Things Work by Yaneer Bar-Yam

Another Characterization of Complex Systems

• Complex systems contain many constituents interacting nonlinearly

• The constituents of a complex system are interdependent

• A complex system possesses a structure spanning several scales

• A complex system is capable of emerging behavior

• Complexity involves an interplay between chaos and non-chaos– “Complex systems dance on the edge of chaos”

• Complexity involves an interplay between cooperation and competition

From “Chaos, Complexity, and Entropy” by Michel Baranger

Wicked Problems

• Every wicked problem is essentially unique• There is no definitive formulation of a wicked problem• Wicked problems are never solved• A wicked problem is not understood until after the

formulation of a solution• Solutions to wicked problems change the problem itself• Solutions to wicked problems are not true or false, but

better or worse• Stakeholders have radically different world views and

different frames for understanding the problem• Conclusion

– Wicked problems are a proper subset of complex systems

– All wicked problems are complex systems, but not all complex systems are wicked problems

Dancing on the Edge of Chaos• Two fundamental assumptions of most

systems analysis are that the system is continuous, and that if it is decomposed far enough it becomes simple

– It can then be analyzed without loss of fidelity

• These assumptions may be driven by the predominant analytical tool of 20th century physics, which was calculus

– For both derivatives and integrals, if you zoom in far enough a curve becomes a straight line

– Thus a calculus problem is transformed into a geometry problem, which has been well understood since Euclid

• However, chaotic systems are often discontinuous, and the decomposition of chaotic systems generally yields more complexity

– e.g., fractals, Sierpiński triangles• “The arrogance of analysis”• Yet chaotic systems often are simple,

not in terms of their decomposition, but in terms of the rules that govern their behavior

– e.g., “boids,” where simple rules lead to complex flocking behaviors

Complex Networks• The interdependencies of complex systems are often modeled by

networks• Networks can be analyzed using several important metrics from

graph theory– Directed (nodes and arcs) vs. undirected graphs (nodes and

edges)– Number of nodes (N): The number of nodes in a network– Degree (k): The number of edges or arcs incident on a node

• Average degree is denoted as <k>• Can distinguish between in degree and out degree

– Node density is actual connections between nodes divided by potential connections that could exist between nodes

• Note that real-world networks are quite sparse, and have an average density of 10-4

– Characteristic path length (L): The average length of the shortest path in the network between two nodes

– Clustering coefficient (C): Number of actual links between neighbors of a node divided by the number of possible links between those neighbors

• This can be thought of as local density, the probability that two neighbors know each other

• The clustering coefficient is a measure of modularity

Random Networks

• For years (since the late 1950’s) the dominant model of networks was the random network, otherwise known as the Erdős-Rényi model, even though it is rarely seen in nature

• Random graphs have a fixed number of nodes which are connected randomly by undirected edges. The number of edges depend on a specified probability.

• The seminal paper is “On Random Graphs,” by Erdős and Rényi (1959)

Small-World Networks

• Adding a small number of links to a regular ring network, in which only the nearest neighbors are connected, can have dramatic effects

• The characteristic path length becomes quite small with only a small number of such cross-neighborhood links

• Small-world networks are pervasive in nature: Friend networks, power grids, predator-prey networks, and even neural networks of the brain.

• The seminal paper is “Collective Dynamics of ‘Small-World’ Networks,” by Watts and Strogatz (1998)

Scale-Free Networks

• Scale-free networks are so-called because they have no characteristic scale, or average number of links per node

• Instead, they are characterized by numerous hubs that hold together highly connected regions of the network

• However, direct connections between hubs are rare (for reasons of resilience)

• Scale-free networks are also pervasive in nature: Disease propagation, terrorist networks, and the World-Wide Web

• The seminal paper is “Emergence of Scaling in Random Networks,” by Barabási and Albert (1999)

Scale-Free Networks (cont’d)

• Scale-free networks seem to follow a power law distribution (which is a variant of a Pareto distribution, or the 80/20 rule)

• Earlier models (such as Erdős-Rényi) described networks using a bell curve, with many nodes with the same number of links and no highly connected nodes

• The scale free model describes networks as having many nodes with only a few links, and a few nodes with many links

• Power law functions appear as straight lines on a log-log plot

Most nodes have the same number of links

No highly connected nodes

Many nodes with few links

A few nodes with many links

# of links (k) # of links (k)

# o

f n

od

es

with

k li

nks

# o

f n

od

es

with

k li

nks

Bell Curve Power Law Distribution

Scale-Free Networks (cont’d)

Pk ~ e-k Pk ~ k-γ

Scale-Free Networks (cont’d)

• In a more formal sense, a scale-free network is characterized by two properties– Continuous growth (the

other network models did not consider growth)

– Preferential attachment (e.g., “the rich get richer”)

• The number of nodes with exactly k links follows a power law for any value of k– P(k) ~ k-γ

– Often 2 < γ < 3• However, many real-world

networks actually exhibit “stretched exponential” behavior, becoming exponential past a threshold– Cost of adding new links– Aging

From Faloutsos, M., P., & C., “On Power-Law Relationships of the Internet Topology” (1999)

Topology is Destiny!

• Characterizing a network quantitatively gives insight into the dynamics (performance) and vulnerabilities of the network

• Random graphs have a short path length (good), but also a small clustering coefficient (not good)

• Regular ring networks have a large clustering coefficient (good) but long path length (not good)

• Small-world networks are a balance between those two extremes, but do not have the robustness of scale-free networks

• Scale-free networks (which are similar in some ways to small-world networks) are robust against random attacks (good) but not against targeted attacks (not good)– The hubs are vulnerable when directly targeted– The network can become disconnected fairly rapidly in this kind

of attack• The moral of the story: When it comes to networks (and to

everything else in life), you can’t have it all …

How Complex Network Research is Done• A network under study is represented and analyzed

mathematically• A randomly generated “ensemble” of networks is

created with which to compare to the network under study– Ensembles of several types of networks are

generated, such as random graphs, small world, regular ring, and scale-free

• The network ensembles are analyzed mathematically and compared to the network under study

• Both the network under study and the generated ensembles are subjected to the same process (e.g., attack), generally via simulation, and the results are compared

Tools for Complex Network Analysis

• Pajek, an open source tool that generates and visualizes networks– A useful book is

Exploratory Social Network Analysis with Pajek, by de Nooy et al.

• UCINET, a commercial tool from Analytic Technologies that quantitatively analyzes networks

• Python with the NetworkX package• MATLAB, a commercial tool from MathWorks,

which can generate, visualize, and quantitatively analyze networks after “a simple matter of programming” – Free network analysis packages are available for

MATLAB

Modeling and Simulation of Emergent Behavior

• Fundamentals of Modeling • Modeling and Simulation of Dynamical Systems (complex

interdependent systems that can exhibit emergent behavior)– Non-Spatial Modeling– Cellular Automata– Agent-Based Models– Network Models

• Other Simulation Paradigms for Dynamical Systems (not covered in the course)– Systems Dynamics

• Numerous commercial packages exist, such as Vensim• Can be described in terms of difference or differential

equations, so can be addressed by non-spatial modeling– Spatial models with partial differential equations– “Soft computing” (e.g., evolutionary computation, artificial

neural networks, or support vector machines)

Fundamentals of Modeling

• Scientific “laws” are really hypotheses, models of observed behavior

• There is no way to prove anything about nature, although it is easy to disprove something

• A model is a simplified representation of a system

• Rule-based modeling allows the system’s possible or future states to be extrapolated, unlike descriptive modeling, which just specifies the system’s actual or past states

• A good model is simple, correct, and robust

Modeling Procedure

• In general– Observe the system of interest– Reflect on the possible rules that might cause the

system’s characteristics– Derive predictions from those rules and compare them

with reality– Repeat to modify the rules until satisfied

• For complex systems– Define the key questions and choose the right scale of

modeling– Identify the structure (the parts and their connections) of

the system– Define the possible states for each part (state space)– Describe how the state of each part changes over time

through interactions with other parts• Models must be evaluated and validated, in terms of

simplicity, correctness, and robustness

Tools Used for Modeling and Simulation

• Python• NumPy and SciPy (MATLAB-like numerical

packages for Python)• matplotlib (MATLAB-like visualization library for

Python)• NetworkX (network graphing package)• Agent-based modeling toolkits, such as NetLogo,

StarLogo, or Swarm (see the Swarm Wiki for other tools)

• And, of course, MATLAB …

Non-Spatial Modeling• The simplest mathematical model

– Called “non-spatial” because the spatial location of the elements of the system is unimportant

• A dynamical system is a system whose state is uniquely specified by a finite set of variables and whose behavior is uniquely determined by predetermined rules – Examples are simple population growth, simple pendulum

swing, motion of celestial bodies • Two schools of modeling dynamical systems

– Discrete-time model (using difference/recurrence equations or iterative maps)

– Continuous-time model (using differential equations)• Non-autonomous, higher-order equations can always be

converted into autonomous, first order equations, by introducing additional variables

• Non-linear systems are generally simulated because they cannot be solved analytically

• Exercise: Logistic growth (logisticgrowthmodel.py)

Cellular Automata

• Adds spatio-temporal dynamics, especially for locally connected networks, to the modeling process

• Cells in a regular grid interact with their nearest neighbors according to simple interaction rules– Transition from one state to another depends on

simple rules based on interactions with neighbors

– The resulting system is a linear system

• Example: Simple diffusion, panic in a gym• Exercise: Decide whether or not to panic based

on the state of a simple majority of your neighbors (majority.py)

Agent-Based Models

• The only way to study the behavior of agent-based models is to simulate, since no global equation exists, only equations at the agent level

• Agent characteristics (what makes an agent an agent)– Local state– Spatial localization– Local interaction with other agents based on

predefined rules– No central supervisor– May learn autonomously– May produce non-trivial collective behavior

Agent-Based Models (cont’d)• To develop an agent-based model

– Identify the agents and get a theory of agent behavior– Identify the agent relationships and get a theory of agent

interaction– Get the requisite agent-related data– Validate the agent behavior models in addition to the model as

a whole (thus you must justify your model relative to the real-world, including emergent behavior)

– Run the model and analyze the output• Exercise: Garbage collection by ants (ants.py)

– Eventually, only one pile of garbage remains– No simulation of the environment is needed, just tracking of

the environment; however, if pheremones were emitted and decayed with time, the environment would need to be simulated as well, based on a model of pheremone decay

• An excellent two-part tutorial was presented by Macal and North at the Winter Simulation Conference in 2005 and 2006, respectively

Network Models• Simulation of the types of complex networks we

learned about in the first part of the course• Uses the NetworkX package from LANL

– Has methods to create random networks, small-world networks, and scale-free networks

• Exercise: A disease epidemic on a random network (networkepidemics.py)– Two parameters: Infection probability and recovery

probability– Red means infected; yellow means susceptible– Note that the disease is never completely

eradicated from a social network; this is called the fixation of a disease

– Can experiment with different network topologies

Further Reading

• Complexity and Chaos– Making Things Work, by Yaneer Bar-Yam (2005)– Ubiquity: The Science of History … or Why the World is Simpler th

an We Think (a.k.a. “Ubiquity: Why Catastrophes Happen”), by Mark Buchanan (2001, 2002)

– Chaos: Making a New Science, by James Gleick (1988)– The Dynamics of Complex Systems, by Yaneer Bar-Yam (1997)

• Complex networks– Linked, by Albert-László Barabási (2003)– Small Worlds: The Dynamics of Networks between Order and Rand

omness, by D. J. Watts (2003)

– Exploratory Social Network Analysis with Pajek, by de Nooy, et al. (2005)

• Modeling and simulation of dynamical systems– Cellular Automata, by Andrew Ilachinski (2001)– Python Scripting for Computational Science, 3rd ed., by Hans Petter

Langtangen (2006)– Simulation Modeling and Analysis, 4th ed., by Averill Law (2007)

Class Project