# Official visual simulations

The visual benchmark extension connects RL algorithms to recognizable navigation and robotics tasks. The data are episodes generated by official simulators, not an offline dataset of images or an externally pretrained agent. Environment packages and their dependencies are cached in the owner's designated data directory; they are installed into a private project environment.

| Environment | Challenge | Comparison |
|---|---|---|
| MiniGrid-FourRooms-v0 | Random agent/goal positions and openings between four rooms | PPO with sparse/potential rewards and flattened/geometry features; known-model BFS and random references |
| MiniGrid-DoorKey-6x6-v0 | Locate and pick up a key, unlock a door, then reach the goal | The same reward ablation and planning/random references |
| HalfCheetah-v5 | Continuous joint torques and simulated locomotion | PPO and SAC; random-action reference |
| Existing navigation studies | Slippery FrozenLake, Taxi pickup/delivery, owned stochastic maze | Four tabular methods, with new fixed-test Q-learning animations |

## Observation and reward profiles

MiniGrid normally returns a local view plus direction and mission text. Our **explicit fully observed profile** exposes the complete symbolic map, normalized position, orientation, carrying state, relative goal/key/door positions, door state and elapsed-time fraction. Constant mission text is omitted for these single-mission environments. The grid is encoded using the official object/color/state codes, not learned RGB representations. Map visibility helps isolate navigation and exploration; these results must not be presented as the default partially observed MiniGrid benchmark. All seven official actions remain available, including ineffective actions. No shortest-path features or planner actions enter learned policy observations.

The potential-shaped training profile adds `gamma * Phi(next) - Phi(current)` with gamma .99 and scale .05. Four Rooms uses full-map spatial distance to the goal. Door Key uses a staged spatial heuristic: key approach, key-to-door travel and unlocking, then goal approach. Spatial distance ignores orientation and is not an optimal action cost-to-go. The heuristic supplies model/map-derived reward information, disclosed as part of training. True goal termination has zero next potential; time-limit truncations retain the next potential for SB3's bootstrap. The discounted shaping sum telescopes, which is tested. The standard policy-invariance result assumes compatible discounting and boundary treatment; a time limit and finite-budget approximation still require care. Evaluation and checkpoint selection always disable shaping and use the official success reward.

The additional `geometry.json` preset compares a44-dimensional task-specific feature encoding with the flattened1102-dimensional map input on Four Rooms. It derives local3×3 walls, agent/goal room membership and the four doorway coordinates relative to the agent from exactly the same full map. Original pose/object features remain available. No shortest path, potential, expert action or planner output enters this encoding. The four-room topology assumption is explicit; other map sizes are rejected. Both encodings use the same256-unit PPO network, interaction budget, reward shaping and validation/test seeds. Input dimensions and consequently parameter counts differ. This controlled comparison tests a practical spatial representation choice, rather than claiming an architecture-independent algorithm ranking.

The `curriculum.json` preset keeps the geometry encoder and gradually widens training-start distances from the goal: caps3,6,10,16,24 until15%,30%,45%,60%,75% of the interaction budget. The final25% uses the exact official random-start distribution. Goal positions and doorway layouts still come from the official generator. The curriculum changes training initial states; validation and tests always use untouched official starts, dynamics, rewards and time limits. The wrapper updates potential accounting after resampling a training start. Tests verify both its initial-distance cap and its final equality to the official reset. This explores sparse-reward discovery using a declared training curriculum, not test-specific starts.

The BFS reference explicitly plans orientation, movement, pickup and door opening from the known deterministic model. It is a planning baseline with different model access, not an RL method trained from samples. The paired maze diagnostic subtracts each learned episode return from the BFS return on the same layout/reset. It measures a finite-horizon, known-model planning gap and is separate from the wireless discounted-MDP gap. Random references use independently seeded uniform actions. They share the 100 test reset seeds with fitted policies.

HalfCheetah uses the official state, action space, dynamics, reward and 1000-step time limit. The 17 inputs describe joint positions and velocities; six bounded continuous actions apply torques. RGB frames visualize the simulation and are not policy inputs. A high numerical return need not imply a natural gait.

## Matched interaction protocol

The eight fitted presets each use 1,048,576 interactions and fitting seeds 0, 1 and 2. PPO uses eight environments, rollout length256, minibatch256, ten optimization epochs, two256-unit tanh hidden layers, learning rate .0003, gamma .99 and GAE .95. Maze entropy coefficient is .01. SAC uses one environment, a million-transition replay capacity, 10k-step warmup, minibatch256, one gradient step per four interactions, two256-unit ReLU hidden layers, automatic entropy tuning and learning rate .0003. These are declared practical presets, not tuned state-of-the-art reproductions. Equal interactions do not equal equal gradient updates, data reuse, wall time or compute.

Every65,536 interactions and after final optimization, evaluate12 predetermined validation resets (200000–200011). Select the checkpoint with the best mean official return, retaining the first tie. The selected checkpoint is loaded fresh on CPU and tested on100 separate fixed resets (300000–300099). All fitting seeds are reported. MiniGrid reset seeds generate held-out layouts; HalfCheetah reset seeds generate initial-state perturbations. The same test reset set across fits supports paired environment conditions without pretending the three fitted policies are identical.

Return confidence intervals bootstrap episodes conditional on one fixed policy. Across-fit SD describes training variability across only three seeds. The maze table also reports goal success fraction and episode length. Validation sets are small and reused for selection; they cannot support a strong generalization guarantee.

## Reproduce and deploy

```bash
python -m pip install -e '.[simulations,notebooks,dev]'
for seed in 0 1 2; do
  decision-lab simulations --seed "$seed"
  decision-lab geometry --seed "$seed"
  decision-lab curriculum --seed "$seed"
done
python scripts/simulation_baselines.py
MUJOCO_GL=egl python scripts/audit_simulations.py
decision-lab report
python scripts/validate_simulations.py
```

`decision-lab geometry` runs the additional spatial-feature comparison. Use `--case DoorKey`, `--case potential`, or `--case HalfCheetah` to filter presets, and `--quick` for a separate smoke run. Never replace full reference evidence with smoke outputs. Full simulation runs write public evidence by case/seed; use a separate checkout or distinct profile names when changing protocols. The CLI refuses a custom output path that would collide with already published episode arrays. CPU training was used for this extension. Set BLAS/Torch thread counts to one when running multiple fits concurrently. Rendering MiniGrid/Gymnasium can use SDL's dummy driver. Headless MuJoCo needs an EGL-capable driver or an appropriate OSMesa setup; rendering requires more system support than training or non-rendered evaluation.

Selected actors are exported under `assets/actors/` as pickle-free float32 NPZ arrays with JSON architecture, preprocessing and action-bound metadata. PPO exports only its deterministic policy head; SAC exports only its mean action network and tanh/action rescaling. No critic, optimizer or replay buffer is distributed. `decision_lab.simulation_deployment.load_actor` loads an actor without deserializing an SB3 checkpoint. The original export audit checked action and complete episode equality against the selected models on the recording platform, including all fitted seeds. The publication validator additionally checks every archived prefix observation/action pair. Discrete inference and replay returns must match exactly; continuous actions use float32 precision tolerances. MuJoCo trajectories can amplify tiny CPU arithmetic differences, so fresh continuous-physics returns are reported as diagnostic replays rather than required to equal the archived scores. Initial states, episode lengths and termination flags still match exactly. The recorded benchmark scores and animations remain unchanged. Source archives include these compact actors and numerical evidence; wheels include the Python API and default configs.

GIFs use the first predetermined test episode from every fitting seed (0, 1 and 2). They are not selected by test score or appearance. Successful and failed episodes are retained. MuJoCo records every fourth simulation frame at the corresponding .2s duration; MiniGrid renders every step. GIF metadata binds the episode return, length, test seed and file digest to public episode arrays. The model-based BFS animations are labeled separately. The earlier LunarLander GIF keeps its original protocol. Captions show the episode seed and step count, so stationary failed policies remain visible.

To replay an exported policy and save your own deterministic episode:

```bash
python scripts/play_simulation.py \
  --actor assets/actors/MiniGrid-DoorKey-6x6-v0_PPO_potential_0.npz \
  --episodes 3 --gif results/deployment/doorkey.gif

MUJOCO_GL=egl python scripts/play_simulation.py \
  --actor assets/actors/HalfCheetah-v5_SAC_official_0.npz \
  --gif results/deployment/halfcheetah.gif
```

The [interactive dashboard](simulation_dashboard.html) embeds measured episode summaries and validation histories and loads the local GIFs. Open it directly in a browser from a checkout/source archive, or serve the repository with `python -m http.server`. It lets you switch tasks and training presets while keeping all three fitting seeds visible. `python scripts/create_simulation_dashboard.py` regenerates it. Optional real-browser verification uses Playwright1.63.0 with Chromium; install that tool separately and run `python scripts/check_simulation_dashboard.py`. The browser runtime is a development check, not a simulator or model dependency.

See [the visual gallery](simulation_gallery.md), [simulation success/return comparisons](simulation_comparison.csv), [aggregate results](results.md), [individual fits](individual_fits.csv), [simulation validation](simulation_validation.json), and [actor audit](simulation_actor_audit.json). Walkthroughs12 and13 explain the choices and replay portable actors.

## Primary references

- [Farama MiniGrid Four Rooms](https://minigrid.farama.org/environments/minigrid/FourRoomsEnv/)
- [Farama MiniGrid Door Key](https://minigrid.farama.org/environments/minigrid/DoorKeyEnv/)
- [Farama Gymnasium HalfCheetah](https://gymnasium.farama.org/environments/mujoco/half_cheetah/)
- [Ng, Harada and Russell, policy invariance under reward transformations](https://people.eecs.berkeley.edu/~russell/papers/icml99-shaping.pdf)
- [Stable-Baselines3 PPO](https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html) and [SAC](https://stable-baselines3.readthedocs.io/en/master/modules/sac.html)