Our Research

Research

Wenhao Chai

My current research currently focus on developing visual intelligence to understand the physical world, building upon video understanding as a core perceptual mechanism. I propose a long-short term memory framework modeled after the human memory system, enabling pre-trained video Large Multi-modal Models (LMMs) to comprehend hour-long video content without additional fine-tuning. To enhance efficiency, I introduce token merging to LMMs, significantly reducing visual tokens with minimal performance degradation. I also demonstrate step-by-step agent system development in Minecraft, showcasing cognitive-inspired agent capabilities in virtual environments.

I have just begun shaping my research narrative around visual intelligence. With "pretraining as we know it will end", I believe the future of artificial intelligence lies in aligning with human intelligence—and ultimately surpassing it. I truly believe computer vision will continue to advance.

Publication

The following presents my comprehensive research experience and areas of focus, along with a timeline highlighting the periods when I was most actively engaged in each field. The template is from here.

Large Multi-Modal Models (06/2023 - present)

How to efficiently build and evaluate large multi-modal models?

How to involve large multi-modal models in embodied agent system?

Show/Hide Work on LMMs

Video Understanding
- Long Video with Long-Short Term Memory MovieChat
- Long Video with Language Guidance Memory MovieChat+
- Video Detailed Captioning AuroraCap
Embodied Agent
- Build in Minecraft STEVE-1
- Multi-Agent STEVE-1.5
- + Knowledge Distillation STEVE-2

Generative Models (03/2023 - 03/2024)

How to generate high-quality images, videos and 3D worlds?

How to control and evaluate the generated content?

Show/Hide Work on Generative Models

Video
- Video Editing with Layered Representation StableVideo
Image
- Style Transfer in Fassion Diffashion
- Restoration with Diffusion Prior DTPM
- + Reinforcement Learning VersaT2I
3D World
- Single Object Consist3D
- City Layout in BEV CityGen
- City with Graphics Engine CityCraft

Human Pose and Motion (08/2022 - 08/2023)

How to estimate human pose and motion from images and videos?

How to generate realistic and controllable human motion?

Show/Hide Work on Human Pose and Motion

3D Human Pose Estimation
- + Domain Adaptation PoseDA
- + Domain Adaptation and Diffusion PoSynDA
- + Diffusion ZeDO
- + Domain Adaptation and Diffusion ZeDO-i
- Animal Pose Estimation UniAP
- Use 4D Radar Tensor RT-Pose
Motion Representation
- Motion Generation PackDiT
- Mask Modeling MPM
- Contrastive Learning UniHPE
Tracking
- + mamba MambaMOT
- Ego-Centric in 3D Space Ego3DT
- + SAM SAMURAI

Our Research

Research

Wenhao Chai

The following presents my comprehensive research experience and areas of focus, along with a timeline highlighting the periods when I was most actively engaged in each field. The template is from here.

Featured

Videos

Videos

AuroraCap [Project Page]

SAMURAI [Project Page]

Ego3DT

STEVE [Project Page]

StableVideo [Hugging Face Demo]

MovieChat [Project Page]

UniAP

Organized

Workshops | Tutorials | Talks

5th International Workshop on Long-form Video Understanding

4th International Workshop on Long-form Video Understanding: Towards Multimodal AI Assistant and Copilot

First Workshop on Imageomics: Discovering Biological Knowledge from Images using AI