CARVIEW

MOTORHOMES

Select Language

HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Wed, 22 Mar 2023 13:19:23 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"641b005b-4cb5" expires: Sun, 28 Dec 2025 13:36:11 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: 22BA:36A0B4:7AB7C1:898882:69512FEF accept-ranges: bytes age: 0 date: Sun, 28 Dec 2025 13:26:11 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210058-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1766928372.634291,VS0,VE215 vary: Accept-Encoding x-fastly-request-id: b18a33758de66ae693c9702ece8d6fc8839fe5ed content-length: 4115 Real-time volumetric rendering of dynamic humans

Real-time volumetric rendering of dynamic humans

Ignacio Rocco, Iurii Makarov, Filippos Kokkinos, David Novotny, Benjamin Graham, Natalia Neverova, Andrea Vedaldi

Meta AI

Paper

Input videos + SMPL fits

Reconstructions

*Faces have been blurred for privacy

Abstract

We present a method for fast 3D reconstruction and real-time rendering of dynamic humans from monocular videos with accompanying parametric body fits. Our method can reconstruct a dynamic human in less than 3h using a single GPU, compared to recent state-of-the-art alternatives that take up to 72h.

These speedups are obtained by using a lightweight deformation model solely based on linear blend skinning, and an efficient factorized volumetric representation for modeling the shape and color of the person in canonical pose. Moreover, we propose a novel local ray marching rendering which, by exploiting standard GPU hardware and without any baking or conversion of the radiance field, allows visualizing the neural human on a mobile VR device at 40 frames per second with minimal loss of visual quality.

Our experimental evaluation shows superior or competitive results with state-of-the art methods while obtaining large training speedup, using a simple model, and achieving real-time rendering.

Method overview

On-device rendering

These videos were recorded from Virtual / Mixed Reality headset with color passthrough capabilities. We render reconstructed humans from ZJU-MoCap dataset with our custom shaders using TensoRF representation. Animation is done with linear blend skinning.

Sequence 377

Sequence 393

Sequence 394

WebGL demos

We also ported our code to WebGL to show real-time rendering capabilities and robustness of our rendering approach: it can run across most popular platforms.

BibTeX

@article{,
      author    = {Rocco, Ignacio and Makarov, Iurii and Kokkinos, Filippos and Novotny, David and Graham, Benjamin and Neverova, Natalia and Vedaldi, Andrea},
      title     = {Real-time volumetric rendering of dynamic humans},
      journal   = {arXiv},
      year      = {2023},
    }

Original Source | Taken Source