[Draft for Conference] 2026 Submission

Real2Sim

3DGS conversion to environments for policy evaluation with per Gaussian physics priors.

3D Gaussian Splats Predictive Physics PolicyEvaluation
Luke Hollis, Tianxing Fan, and Heng Yang
Harvard University
Paper [Coming Soon] Code [Coming Soon] Demo

Convert 3DGS scenes captured by humans or robots or generated as synthetic scenes into fully articulated scene graphs with predictive physics properties for policy evaluation and training.

3D Gaussian scene captured from a real robot environment
01

Capture or Generate

Ingest real human and robot scans or synthetic Gaussian scenes as spatial evidence.

Kitchen scene with objects segmented for scene graph construction
02

Build Scene Graphs

Segment objects, infer articulation, and attach per-Gaussian semantics and physics priors.

Robot arm evaluating object interaction in a reconstructed scene
03

Evaluate Policies

Export simulation assets for Issac Lab, MuJoCo, LIBERO, others for robot policy rollouts, training, and sim-real comparisons.

Conversion
Three-step Predictive Real2Sim architecture from 3D scene capture to scene graph construction and Isaac Lab scene review

The system currently works by capturing or generating a 3D Gaussian scene, converting it into an scene graph with semantic and physics priors, then review segmented assets and placements in Isaac Lab before policy runs.

Predictive per Gaussian Physics Prior

We create annotated physics information on several 3d datasets and then use it to infer physics properties to objects segmented from each scene. We then create an object physics material from this information to represent the object in an OpenUSD file format that may be used across a wide range of simulators.

Real2Sim workflow showing physics-prior training, scene Gaussian fields, object reasoning, audit, and simulation handoff
Scene Physics Properties

Loading physics-prior cluster data...
Sim2Real Transfer

Next, we need to validate our policies trained in our 3d scanned environments by testing them back sim2real in our lab.

More coming soon.

Franka robot arm manipulating objects on a real tabletop
Policy Evaluation in Converted Scenes

We evaluated the DROID joint-position policy labels as well as other common VLAs and world models in the converted HCRG 3DGS Isaac environments. These are simulator-side rows only: Pearson, Spearman, and MMRV remain N.A. until we complete real-robot rollouts with the same policies in lab.

HCRG 3DGS PolaRiS policy coverage and metric gate plots
Reference
@misc{hollis2026real2sim,
      title  = {Real2Sim: 3DGS conversion to environments for policy evaluation with per Gaussian physics priors},
      author = {Hollis, Luke, Tianxing Fan, and Heng Yang},
      year   = {2026},
      note   = {[Draft Conference] 2026 submission}
    }