• 1University of Illinois at Urbana-Champaign
  • 2Zhejiang University
  • 3University of Maryland, College Park

  • * Equal Contribution

Abstract

Physical simulations produce excellent predictions of weather effects. Neural radiance fields produce SOTA scene models. We describe a novel NeRF-editing procedure that can fuse physical simulations with NeRF models of scenes, producing realistic movies of physical phenomena in those scenes. Our application -- Climate NeRF -- allows people to visualize what climate change outcomes will do to them. ClimateNeRF allows us to render realistic weather effects, including smog, snow, and flood. Results can be controlled with physically meaningful variables like water level. Qualitative and quantitative studies show that our simulated results are significantly more realistic than those from state-of-the-art 2D image editing and 3D NeRF stylization.
overview

Weather simulations

* You can select different weather conditions on different scenes and compare our method with baselines.
* 3D stylization denotes finetuning pre-trained NGP model using FastPhotoStyle.

Weather

Scene

Method

Controllable rendering

Our method simulates different densities of smog and distinct heights of flood and accumulated snow:

Simulations on drone views

Our method simulates flood on scenes captured by drones.

Rendering Procedure of ClimateNeRF

We first determine the position of physical entities (smog particle, snow balls, water surface) with physical simulation. We can then render the scene with desired effects by modeling the light transport between the physical entities and the scene. More specifically, we follow the volume rendering process and fuse the estimated color and density from 1) the original radiance field (by querying the trained instant-NGP model) and 2) the physical entities (by physically based rendering). Our rendering procedure thus maintain the realism while achieving complex, yet physically plausible visual effects.

Flood Simulation

(a) Original NeRF (b) Depth map (c) Water surface (d) Normal map with wave (e) Final ClimateNeRF
We first estimate the vanishing point direction based on the original image (a) and depth (b). With the vertical vanishing direction (yellow arrows painted (c)), we can insert a planar water surface. We use FFT based water surface simulation to produce a spatiotemporal surface normal map in (d). Our ClimateNeRF renders the scene with the simulated flood through ray tracing NeRF (e).

Snow Simulation

(a) Original NeRF (b) Surface normal (c) Metaball centers (red) (d) Snow with diffuse model (e) Snow with scattering
We first locate metaballs on object surfaces facing upward based on surface normal values (b). With metaballs (centers painted in red), we can estimate snow's density and color with a parzen window density estimator. (d) and (e) show the differences between fully diffuse model and scattering approximations, shadowed parts in (d) are lit in (e).

User Study

We perform a user study to validate our approach quantitatively. Users are asked to watch pairs of synthesized images or videos of the same scene and pick the one with higher realism. 37 users participated in the study, and in total, we collected 2664 pairs of comparisons.
Images Videos
Smog
Flood
Snow
The length of bars indicates the percentage of users voting for higher realism than the opponents. The green bar with the number shows our win rate against each baseline. The video quality of our method significantly outperforms all baselines.

References

  1. Victor Schmidt, Alexandra Sasha Luccioni, M ́elisande Teng, Tianyu Zhang, Alexia Reynaud, Sunand Raghupathi, Gautier Cosne, Adrien Juraver, Vahe Vardanyan, Alex Hernandez-Garcia, Yoshua Bengio. Climategan: Raising climate change awareness by generating images of floods. ICLR, 2022. [code]
  2. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ̈orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. [code]
  3. Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. NeurIPS, 2020. [code]

Acknowledgements

The website template was borrowed from Michaël Gharbi, RefNeRF , Nerfies and Semantic View Synthesis.