Geometry-native visual generation

GAE Geometry-Native Autoencoder

Learning a Geometry-Native Latent Space for 3D-Consistent World Generation.

  • 1 The Hong Kong University of Science and Technology
  • 2 ARC Lab, Tencent IEG
  • 3 The University of Hong Kong
  • 4 The University of Texas at Austin

* Equal contribution † Corresponding authors

We distill multi-level geometry-foundation features into one small, well-conditioned state, then perform flow matching directly in that state. Every generated latent decodes into appearance and 3D—without reconstructing geometry as an afterthought.

1B parameters Research project · 2026

A shared latent. Four connected outputs.
Generated worldArched lounge
01 RGBGenerated appearance
02 DepthScene structure
03 Camera posesGT Estimated
04 3D geometryProgressive point cloud
z
One geometry-native latentRGB · depth · camera · 3D
↗ Paper PDF arXiv ↗ Code ↗ Explore results ↓

See the world take shape

One latent. A world in motion.

Generated appearance on the left.
Progressive geometry on the right.

RGB video
Progressive geometry
0:00 / 0:06
18 scenes · one shared geometry-native latent

TL;DR

Put geometry inside the generated latent, not around an appearance-only latent as an external constraint. GAE compresses 3,072-channel geometry features to a structured 128-channel state that remains decodable to both RGB and geometry.

01 / Geometry latent

One latent family. Two favorable operating points.

A useful generation state must do more than reconstruct. GAE-64 favors compact generation and trajectory consistency; GAE-128 preserves the strongest reconstruction and cross-view correspondence. Together, the two operating points define a favorable Pareto frontier relative to the baselines.

24×fewer channels
3072 → 128
10⁶×better conditioning
κ ≈ 10⁸ → 10²
256.4mean generation FVD
best among latents

Each metric is normalized so that the best result lies on the outer ring. The axes jointly cover latent representation quality, DiT-generated RGB appearance, and the 3D consistency of generated views.

Latent representation quality and matched DiT generation

Best result on every axis is on the outer ring.

Latent representation quality DiT-generated RGB appearance quality DiT-generated views 3D consistency
Latent representation quality and matched DiT generation. GAE and GAE-64 jointly form the strongest overall frontier across reconstruction, appearance generation, camera motion, reprojection, and cross-view consistency. Hover over a method to isolate it, or click to keep it highlighted.

02 / Method

A geometry-native state from end to end.

The frozen foundation encoder supplies multi-view geometric structure. A learned codec makes that structure compact and diffusable; flow matching generates it; the output heads decode RGB, depth, pose, and point-cloud geometry from the same final state.

Stage 1: compact diffusable geometry latent, from frozen foundation features through geometry autoencoding to RGB and geometry heads Stage 2: conditional flow matching with reference latents, camera rays, and text, jointly decoding generated latents into RGB, depth, point clouds, and poses
1

Encode & distill

Freeze the geometry foundation model; learn only the compact multi-level codec.

2

Generate & control

Run x-prediction flow matching with text, reference evidence, and metric rays.

3

Decode jointly

Recover RGB, depth, pose, and point-cloud geometry from the same generated latent state.

GAE pipeline. Multi-level DA3 features are fused and compressed into a standardized compact latent. Text, clean reference tokens, and metric camera rays control flow matching; generated states decode jointly to RGB, depth, pose, and point-cloud geometry. Open vector PDF ↗

03 / Indoor scenes

Move the camera. Keep the world.

Browse 21 scenes spanning RealEstate10K and ScanNet++. From one observed frame and a metric camera path, GAE jointly generates an 81-view 672 x 378 RGB sequence and a geometry state. Select a scene, play the GAE-predicted RGB or generated depth, watch the progressive geometry reconstruction alongside the generated sequence.

Generated sequence

Hallway-to-bedroom traversal

Decoded geometry

Progressive point cloud

04 / Outdoor scenes

A wider world in motion.

Explore generated scenes and their 3D geometry, from city streets to open landscapes.

Generated sequence

City storefront

Decoded geometry

Progressive point cloud

05 / 3D consistency

Compare each video with the geometry it generates.

Each column shows a method's generated RGB sequence alongside its reconstruction score. All methods receive the same RealEstate10K scene, 9 views, interval 10, and one conditioning frame; the reference is DA3 run on the real target frames. Compare appearance quality alongside the measured geometric consistency.

Hallway into a window room

Video + geometry metrics paired by method · synchronized playback

06 / Latent comparison

Same camera path. Different latents.

Six latent representations are compared under the same DiT training and sampling protocol: GAE, RAEv2, the Stable Diffusion VAE, the Wan2.1 video VAE, and raw DA3 features from two layers. Evaluation uses 64 RealEstate10K and 64 ScanNet++ scenes, with 9 views, one conditioning frame, guidance scale 2.0, and 50 sampling steps. Videos are generated from noise conditioned on the reference image. DA3-based latents use a 252 px input with 14 px patches; the other latents use a 256 px input with an 8 px grid. Clips are resampled to 252 px for display and evaluation. Each PSNR value is averaged over the clip against the corresponding reference frames.

A-frame cabin on a forested ridge

07 / Text to image

One prompt. RGB, depth, and 3D.

Twenty text-generated examples span characters, robots, interiors, food, sculpture, and stylized portraits. Select a scene to compare its RGB image, depth map, and interactive point cloud, decoded from the same generated latent.

Generated RGB

Antique gilt reliquary

Generated RGB imageRGB
Same generated latent

Depth

Generated depth mapDEPTH
Same generated latent

Interactive 3D

Loading geometry
drag to orbit · scroll to zoom · right-drag to pan
Prompt

08 / Paper

Abstract

We present a compact geometry-native latent space as a shared foundation for perception and generation. Visual generators can produce photorealistic frames without preserving a consistent 3D scene. We argue that this is not only a modeling problem but also a representation problem: generators typically evolve appearance-centric latents, while perception models recover geometry in a semantically rich space that encodes cross-view structure. Rather than adding geometry as another output, we reparameterize a geometry foundation model’s features into a compact latent space for generation. We realize this shift with the geometry-native autoencoder (GAE), whose latent is jointly decodable to appearance, depth, cameras, and point maps. With this state, a standard conditional flow supports diverse generation tasks.

In controlled comparisons that hold the generator and training protocol fixed, replacing the latent with GAE improves both visual quality and independently measured 3D coherence: FVD falls by 12.7% and 23.1% on RealEstate10K and DL3DV, and camera-trajectory error is halved on RealEstate10K. Together, these results show that the latent space is central to geometry-consistent generation and can serve as a shared interface between perception and generation.

Read the full paper ↗