Multi-Agent Systems

by Albert Major

Introduction to Analyzing Agent-Based Models (ABMs)

• Course Context: The focus shifts from designing and implementing models to analyzing the data they generate. The goal is to output data that specifically answers a research question.
• Defining Metrics:
  • Metrics must be clearly linked to the research question or goal.
  • It is crucial to decide what to measure while designing the model. If you build the model first and think about measurement later, you may need to redesign elements.
  • Examples of metrics: Time required for 100% of the population to become infected; traffic throughput; population size over time.

Determining the Number of Simulation Runs

Why multiple runs are necessary

• Agent-based models contain stochastic (random) elements.
• Running a model once yields a single result that may be an outlier due to randomness.
• Running a model multiple times (e.g., 100 to 10,000 times) allows researchers to rule out randomness and identify systematic patterns.
• The goal is to generate confidence that the result is due to the parameters, not sampling error.

Approaches to selecting the number of runs

1. Rule of Thumb / Convention (Weakest Approach):
   • Selecting a number (e.g., 100 runs) simply because it is the norm in other papers.
   • While better than a single run or stopping when a "significant" result is found (which is a questionable research practice), it lacks statistical justification.
2. Power Analysis (A Priori Approach):
   • Uses statistical power to determine the required sample size (number of runs) to detect an effect.
   • Tool: G*Power is a common interface for this.
   • Inputs required: Statistical test type (e.g., ANOVA, regression), desired power (usually 0.8 or higher), alpha level (usually 0.05), and Effect Size.
   • Challenge: It is difficult to know the effect size before running the model.
   • Solution: Estimate effect size based on prior literature or assume a small-to-medium effect size to ensure the sample is large enough to detect meaningful changes.
3. Variance Stability / Coefficient of Variation (Iterative Approach):
   • Involves running the model with increasing sample sizes (e.g., 10, 50, 100, 500 runs).
   • Calculate the variance or standard error for the metric at each sample size.
   • Goal: Identify the threshold where the variance stabilizes (stops changing significantly) and stays below a specific tolerance level.
   • Challenge: You might perform more runs than necessary during the testing phase, or the variance could appear stable temporarily before changing again.

Case Study: Spread of Disease Model (NetLogo)

The lecture uses the "Spread of Disease" model to illustrate how model structure affects analysis and metrics.

1. Mobile Variant

• Mechanism: Infected agents move through the environment. Infection spreads when an infected agent touches a healthy agent.