| CARVIEW |
Select Language
HTTP/2 301
server: GitHub.com
content-type: text/html
location: https://ttaa9.github.io/genren/
x-github-request-id: 139E:2DDCFF:7F3E6E:8EB6E4:69518897
accept-ranges: bytes
age: 0
date: Sun, 28 Dec 2025 19:44:23 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210023-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1766951063.286286,VS0,VE201
vary: Accept-Encoding
x-fastly-request-id: 19590f59c03f52a20a103acf4eed7d95a3174804
content-length: 162
HTTP/2 200
server: GitHub.com
content-type: text/html; charset=utf-8
last-modified: Sat, 24 Apr 2021 03:07:10 GMT
access-control-allow-origin: *
etag: W/"60838b5e-30ea"
expires: Sun, 28 Dec 2025 19:54:23 GMT
cache-control: max-age=600
content-encoding: gzip
x-proxy-cache: MISS
x-github-request-id: 703A:2B0FD4:7FA5F4:8F1F57:69518897
accept-ranges: bytes
age: 0
date: Sun, 28 Dec 2025 19:44:23 GMT
via: 1.1 varnish
x-served-by: cache-bom-vanm7210023-BOM
x-cache: MISS
x-cache-hits: 0
x-timer: S1766951064.500498,VS0,VE219
vary: Accept-Encoding
x-fastly-request-id: a182fc79394e846093ad0724b43daaadcc8255b1
content-length: 3879
Cycle-Consistent Generative Rendering for 2D-3D Modality Translation
Cycle-Consistent Generative Rendering for 2D-3D Modality Translation
|
|
|
|
|
|
|
|
|
|
|
|
| For humans, visual understanding is inherently generative: given a 3D shape, we can postulate how it would look in the world; given a 2D image, we can infer the 3D structure that likely gave rise to it. We can thus translate between the 2D visual and 3D structural modalities of a given object. In the context of computer vision, this corresponds to a learnable module that serves two purposes: (i) generate a realistic rendering of a 3D object (shape-to-image translation) and (ii) infer a realistic 3D shape from an image (image-to-shape translation). In this paper, we learn such a module while being conscious of the difficulties in obtaining large paired 2D-3D datasets. By leveraging generative domain translation methods, we are able to define a learning algorithm that requires only weak supervision, with unpaired data. The resulting model is not only able to perform 3D shape, pose, and texture inference from 2D images, but can also generate novel textured 3D shapes and renders, similar to a graphics pipeline. More specifically, our method (i) infers an explicit 3D mesh representation, (ii) utilizes example shapes to regularize inference, (iii) requires only an image mask (no keypoints or camera extrinsics), and (iv) has generative capabilities. While prior work explores subsets of these properties, their combination is novel. We demonstrate the utility of our learned representation, as well as its performance on image generation and unpaired 3D shape inference tasks. |
Talk
Method Overview
|
| Our method simultaneously learns a render generator and a 3D inferrer from unpaired data. The former probabilistically samples 3D texture and pose, conditioned on an input shape, followed by rendering (similar to a graphics pipeline); the latter infers a graphics code descriptor of a 3D object from a 2D image, including rigid pose, mesh shape, and texture (acting as a computer vision algorithm). Analogous to the domain translation literature, which inspired our cycle-consistent distribution-matching training formulation (depicted above), we term these mappings our shape-to-image and image-to-shape translations, respectively, which permit moving between the 2D and 3D modalities of an object. These functions allow generative image modelling and single-image 3D reconstruction, but also facilitate learning of a smooth, factorized, and manipulable latent object representation. |
 |
Tristan Aumentado-Armstrong, Alex Levinshtein, Stavros Tsogkas, Konstantinos Derpanis, and Allan Jepson Cycle-Consistent Generative Rendering for 2D-3D Modality Translation. Samsung AI Centre Toronto In 3DV, 2020 (Oral). [Paper] [ArXiv] [Bibtex] |
Acknowledgements |