Multi-Agent Systems

by Albert Major

Analysis of Agent-Based Models (ABM)

Determining the Number of Runs

• Coefficient of Variation (CV) Approach: The primary method discussed for deciding how many times to run a model. You should plot the CV against the number of runs. The appropriate number is the point where the CV crosses a specific threshold and stabilizes, indicating that the model's metric is no longer fluctuating wildly due to randomness.
• Alternative Approaches:
  • Power Analysis: Determining sample size based on the effect size you want to detect and the specific statistical test you intend to use.
  • Literature Consensus: Adopting the number of runs commonly used in similar published papers (though less rigorous than the CV or Power Analysis methods).

Experimental Design and Scenarios

• Goal of Design: The objective is not just to observe outputs but to uncover the underlying mechanisms driving agent behavior.
• Scenario Comparison: A systematic way to analyze model behavior is by defining distinct scenarios.
  • This involves varying specific parameters (e.g., Parameters A, B, and C) in a structured way (e.g., Setting A to 0, B to 10, C to 100 for Scenario 1).
  • Parameter choices should be theoretically motivated, not random.
• Case Study (AMFOR ABM - Wine Trading): The lecture used a Greek and Etruscan wine trading model to demonstrate mapping research questions to specific data metrics:
  • Question: What is the maximum wine production? -> Metric: Sum of total wine produced.
    • Question: How quickly does Greek wine replace Etruscan wine? -> Metric: Proportion of wine types over time (or the specific time point where Greek > Etruscan).
  • Question: Does viticulture density match history? -> Metric: Spatial presence/absence of winemakers on patches compared to archaeological records.

Model Duration (How long to run the model?)

• Stopping Conditions:
  • Defined State: The model stops when a specific condition is met (e.g., 100% of the population is infected).
  • Predetermined Time: Running the model for a fixed duration (e.g., 500 years or ticks).
• Equilibrium/Stabilization: If the model is dynamic (like a Predator-Prey model), you must run it long enough for patterns to stabilize.
  • Internal CV: You can apply the Coefficient of Variation logic to a metric within a single run over time. You calculate a rolling CV to identify when the oscillations (e.g., sheep population) settle into a stable pattern.

Data Extraction and Visualization

• Extraction: In tools like NetLogo, data can be exported directly from plots or via code (e.g., export-my-data) to CSV for analysis in Python or R.
• Visualization Best Practices:
  • Clarity: Plots must have clear units, labeled axes, and grid lines to aid interpretation.
  • Variability: When plotting results from multiple runs, you must visualize the variability, not just the mean. Use standard error bars, confidence intervals, or "shaded cones" of influence to show the range of outcomes.
  • Interpretability: A good figure should allow a reader to understand the result without reading the text.

Open Science and Replicability

• Replicability Strategies:
  • Public Repositories: Hosting code on GitHub, GitLab, or the Open Science Framework (OSF).
  • Documentation: Ensuring code is well-commented and accompanied by a report explaining the methods.
  • Pre-registration: Registering hypotheses, predictions, and analysis plans before conducting the research to increase transparency and reduce bias.
• Digital Sovereignty: Platforms like GitLab are highlighted for data sovereignty (keeping control of your own data).