RFC 0002: Counterfactual-first metric design

  • Status: Accepted for alpha
  • Version: 0.1

Problem

Single recorded trajectories confound action effects with the initial scene and policy. A realistic future prediction may come from a scene prior while ignoring the candidate action.

Decision

The primary evaluation unit is an intervention group: one shared context paired with at least two uniquely named actions. Ground-truth futures are generated by restoring the same simulator state before each action branch.

For the LIBERO alpha, “same simulator state” includes MuJoCo mjSTATE_INTEGRATION plus Python-side runtime state that can affect controls: robosuite clocks/controllers and observable reset policy, global RNG state, and stateful gripper commands. The generator must prove repeated-rollout and branch-order invariance; equality of LIBERO's flattened time/qpos/qvel wrapper state alone is insufficient.

The alpha validates five complementary measurements:

  1. action dependence;
  2. counterfactual direction correctness;
  3. no-op stability;
  4. state trajectory error;
  5. candidate-selection regret.

Required sanity checks

Every new metric must document what it measures, what can game it, and how the following baselines should behave: exact oracle, copy-last, wrong-direction, and action-agnostic success prior. Learned metrics must additionally pin model weights and training data.

No composite score in v0.1

Action responsiveness, physical accuracy, and control utility are distinct. WAMProbe will publish a profile and paired confidence intervals before considering any aggregate score.