Do Androids Dream of Electric Papers?

How do we know when an AI Scientist is actually doing science?

AI Scientists — systems that formulate hypotheses, design and run experiments, interpret results, and iterate with limited human intervention — are developing rapidly, but the field still lacks a shared way to evaluate and compare them. Different teams currently work with different tasks, datasets, environments and metrics, making cross-system comparison and cumulative progress difficult.

Our starting point: a credible benchmark can not evaluate only whether an AI Scientist reaches the right answer — it must evaluate how the result was produced. “Do Androids Dream of Electric Papers?” is a focused working workshop that breaks this central question into six that any credible benchmark has to answer:

1.

What should we test?

Tasks & capabilities

2.

Where should it do it?

Environments

3.

What should we observe?

Traces & provenance

4.

How should we score it?

Metrics

5.

How do we know the result is real?

Reproducibility & contamination

6.

How do systems compare?

Benchmark protocol

A shared task suite, reporting standards, and contribution formats and licensing for datasets and environments.