Our Research

Research

Wenhao Chai

My current research focus on developing embodied intelligence inspired by cognitive science principles to interact with the physical world, building upon video understanding as a core perceptual mechanism. I propose a long-short term memory framework modeled after the human memory system, enabling pre-trained video LMMs to comprehend multi-hour video content without additional fine-tuning. To enhance efficiency, I introduce token merging, significantly reducing visual tokens with minimal performance degradation. I also demonstrate step-by-step agent system development in Minecraft, showcasing cognitive-inspired agent capabilities in virtual environments.

I have just begun shaping my research narrative around embodied intelligence and cognitive science. With "pretraining as we know it will end", I believe the future of artificial intelligence lies in aligning with human intelligence—and ultimately surpassing it.

Publication

The following showcases my complete past research experience and fields with timeline. The template is from here.

Large Multi-Modal Models (06/2023 - present)

How to efficiently build and evaluate large multi-modal models?

How to involve large multi-modal models in embodied agent system?

Show/Hide Work on LMMs

Video Understanding
- Long Video with Long-Short Term Memory [MovieChat]
- Long Video with Language Guidance Memory [MovieChat+]
- Video Detailed Captioning [AuroraCap]
Embodied Agent
- Build in Minecraft [STEVE-1]
- Multi-Agent [STEVE-1.5]
- + Knowledge Distillation [STEVE-2]

Generative Models (03/2023 - 03/2024)

How to generate high-quality images, videos and 3D worlds?

How to control and evaluate the generated content?

Show/Hide Work on Generative Models

Video
- Video Editing with Layered Representation [StableVideo]
Image
- Style Transfer in Fassion [Diffashion]
- Restoration with Diffusion Prior [DTPM]
- + Reinforcement Learning [VersaT2I]
3D World
- Single Object [Consist3D]
- City Layout in BEV [CityGen]
- City with Graphics Engine [CityCraft]

Human Pose and Motion (08/2022 - 08/2023)

How to estimate human pose and motion from images and videos?

How to generate realistic and controllable human motion?

Show/Hide Work on Human Pose and Motion

3D Human Pose Estimation
- + Domain Adaptation [PoseDA]
- + Domain Adaptation and Diffusion [PoSynDA]
- + Diffusion [ZeDO]
- + Domain Adaptation and Diffusion [ZeDO-i]
- Animal Pose Estimation [UniAP]
- Use 4D Radar Tensor [RT-Pose]
Motion Representation
- Mask Modeling [MPM]
- Contrastive Learning [UniHPE]
Tracking
- + mamba [MambaMOT]
- Ego-Centric in 3D Space [Ego3DT]
- + SAM [SAMURAI]

Our Research

Research

Wenhao Chai

The following showcases my complete past research experience and fields with timeline. The template is from here.

Featured

Videos

Videos

AuroraCap [Project Page]

SAMURAI [Project Page]

Ego3DT

STEVE [Project Page]

StableVideo [Hugging Face Demo]

MovieChat [Project Page]

UniAP

Organized

Workshops | Tutorials | Talks

5th International Workshop on Long-form Video Understanding: Towards Multimodal AI Assistant and Copilot

4th International Workshop on Long-form Video Understanding: Towards Multimodal AI Assistant and Copilot

First Workshop on Imageomics: Discovering Biological Knowledge from Images using AI

Featured

Posters

^† Attend in person.

Ego3DT: Tracking Every 3D Object in Ego-centric Videos

^† MovieChat: From Dense Token to Sparse Memory for Long Video Understanding

^† STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft

^† UniAP: Towards Universal Animal Perception in Vision via Few-shot Learning

^† StableVideo: Text-driven Consistency-aware Diffusion Video Editing

See and Think: Embodied Agent in Virtual Environment

^† Learning Diffusion Texture Priors for Image Restoration

Hierarchical Auto-Organizing System for Open-Ended Multi-Agent Navigation

^† Back to Optimization: Diffusion-based Zero-Shot 3D Human Pose Estimation

^† Global Adaptation meets Local Generalization: Unsupervised Domain Adaptation for 3D Human Pose Estimation