HTTP/2 200 server: GitHub.com content-type: text/html; charset=utf-8 last-modified: Tue, 29 Aug 2023 04:28:39 GMT access-control-allow-origin: * strict-transport-security: max-age=31556952 etag: W/"64ed73f7-59f9" expires: Tue, 30 Dec 2025 04:00:16 GMT cache-control: max-age=600 content-encoding: gzip x-proxy-cache: MISS x-github-request-id: 10D8:21D6A4:99F6FA:ACD764:69534BF8 accept-ranges: bytes age: 0 date: Tue, 30 Dec 2025 03:50:16 GMT via: 1.1 varnish x-served-by: cache-bom-vanm7210026-BOM x-cache: MISS x-cache-hits: 0 x-timer: S1767066616.299253,VS0,VE205 vary: Accept-Encoding x-fastly-request-id: 12f9c9df476db4a6114984afd59af79d669e6b26 content-length: 3716

MagicAvatar:

Multimodal Avatar Generation and Animation

Jianfeng Zhang*,

Hanshu Yan*,

Zhongcong Xu*,

Jiashi Feng,

Jun Hao Liew*⁺

Bytedance Inc.   ^*Equal Contribution   ⁺Project Lead

arXiv Code Video

Introducing MagicAvatar, a multi-modal framework capable of converting various input modalities — text, video, and audio — into motion signals that subsequently generate/ animate an avatar.

Applications

---

▶ Text-guided Avatar Generation

Create avatar(s) with simple text prompts.

---

Text prompt

Generated motion

Generated video

Text prompt

Generated motion

Generated video

"An astronaut, kicking, in volcano"

"A rugged mountaineer, doing moonwalk, on the foot of the hill"

"A brave firefighter, punching a bag, in a burning building"

"A female dancer, crossing arms, in a grand ballroom"

"A female astronaut, practicing yoga poses, undersea"

"A yoga practitioner, practicing yoga poses, peaceful garden view"

"A group of k-pop stars, dancing, in volcano"

"A group of children, dancing, in a park"

---

"An astronaut, doing handstand, undersea"

"A basketball player, doing moonwalk, on a basketball court"

"A soldier, saluting, in a military base"

---

▶ Video-guided Avatar Generation

Given a source video, create avatar(s) that follows the given motion.

---

Source video

Motion

Generated video

Source video

Motion

Generated video

"A boy running, red jacket"

"A child, playing toy blocks"

"A man, dancing under water"

"A boy, freestyle singing, fire in the background"

"A girl, in yellow clothes, walking on the road, raining"

"A baby, in green clothes, dancing on the bed"

---

Source video & Motion

Generated video

Source video & Motion

Generated video

"Two people, Judo, on the fire"

"Four girls, dancing, nighttime, blue suits"

"Three girls, dancing, on the grassland"

"Girls dancing"

"Three girls, dancing, raining, flowers"

"Two persons, fencing, outdoor"

---

▶ Multimodal Avatar Animation

Animate an avatar of a specific subject.

---

Identity

Driving signal

Motion

Generated video

Driving signal

Motion

Generated video

"A girl is dancing"

"A girl is waving hands"

---

▶ Audio-guided Avatar Generation (coming soon)

Create an avatar based on audio input.

---

BibTeX

@inproceedings{zhang2023magicavatar,
    author = {Zhang, Jianfeng and Yan, Hanshu and Xu, Zhongcong and Feng, Jiashi and Liew, Jun Hao},
    title = {MagicAvatar: Multi-modal Avatar Generation and Animation},
    booktitle={arXiv:2308.14748},
    year = {2023}
}