Yongjie Zhu

AI 2.0 (GenAI): Currently I build and deploy data algorithms for the multimodal data pipeline at Kling Team , covering multimodal understanding, multimodal data quality grading, and multimodal retrieval systems that support large-scale generative model training. My work focuses on building robust data-centric infrastructure for multimodal foundation models, including preference data construction, quality-aware data selection, and retrieval-aligned optimization to improve training efficiency and model performance.

AI 1.0 (Perception & Recognition): Previously, at Alibaba's AutoNavi Vision Technology Center, I was responsible for large-scale perception algorithms deployed nationwide, including detection, segmentation, recognition, and 3D reconstruction systems, along with full data-closed-loop optimization. My work bridged algorithm research and production deployment in real-world geographic and spatial intelligence systems.

I am looking for self-motivated interns for mulimodal understanding and generation research! If you are interested in internship at Kling or collaboration of research, please email me.

Email  /  Google Scholar  /  LinkedIn    

• [2025.12] Kling-Omni Technical Report is released. [Link].
• [2025.09] One paper accepted to NeurIPS 2025.
• [2025.05] One spotlight paper accepted to ICML 2025.
---- show more ----
[2023.10] One paper accepted to TPAMI.
[2023.02] One paper accepted to CVPR 2023.
[2022.11] Excellent graduation thesis of BUPT.
[2022.07] One oral paper accepted to ECCV 2022.
[2022.06] AdsCVLR is accepted by ACM MM 2022.
[2021.11] Received China National Scholarship.
[2021.05] Outstanding reviewer of CVPR 2021.
[2021.05] The extension of my bachelor thesis HyFRIS-Net got accepted by TPAMI!
[2021.03] SOLID-Net is accepted by CVPR 2021 as oral presentation (top 4%)!
[2021.03] DeRenderNet is accepted by ICCP 2021.

I obtained my Master degree from School of Artificial Intelligence, Beijing University of Posts and Telecommunications (BUPT), co-supervised by Dr. Chen Li, Prof. Si Li and Prof. Boxin Shi. My research interests lie in multimodal understanding, with an emphasis on integrating vision, audio, language, and physical cues for grounded reasoning. Recently, I have focused on physics-aware and generative approaches to help models understand the physical structure and dynamics of the world, enabling richer multimodal perception and reasoning.

We propose a reasoning-driven Universal Multimodal Embedding framework that leverages Embedder-Guided Reinforcement Learning to generate retrieval-aligned, evidential Traceability CoTs, significantly improving cross-modal semantic consistency, fine-grained matching, and benchmark performance under limited computational resources.

We introduce UltraVQA, a large-scale multi-dimensional video quality assessment dataset with rich human and GPT-augmented annotations, and propose Analytic Score Optimization (ASO), a theoretically grounded post-training objective that models quality assessment as a regularized decision process, achieving superior performance and lower MAE compared to existing closed- and open-source baselines.

We identify a negative correlation between visual and motion quality in video data and propose Timestep-aware Quality Decoupling (TQD), which leverages timestep-specific sampling to decouple imbalanced data and achieve performance surpassing conventional training with even better data.

We introduce the VidEmo framework and the Emo-CFG dataset, enabling affective cue–guided video emotion understanding and significantly outperforming existing VideoLLMs across multiple video perception tasks.

A modular attention-empowered Multimodal LLM that delving deep into the fine-grained cues for emotion understanding and cognition analysis.

This paper proposes a novel pipeline to estimate a non-parametric environment map with high dynamic range from a single human face image.

This paper proposes to complement the intrinsic estimation from volume rendering using NeRF and from inversing the photometric image formation model using convolutional neural networks (CNNs).

Given a single image and a 2D pixel location, our method can estimate the local lighting that is disentangled into ambient sky light, sun light and lighting-independent local contents.

We propose a multi-modal relevance modeling approach for sponsored search, and boost the performance via contrastive learning that naturally extends the transformer encoder with the complementary multi-modal inputs.

We proposed a self-supervised deep learning framework that can estimate the hybrid reflection model and detailed normal of the human face. The proposed hybrid reflectance and illumination representation ensures the photo-realistic face reconstruction.

Collecting high quality paired intrinsic and lighting data in a virtual city lets you train a model that estimates spatially-varying lighting from a single outdoor image.

Decomposing a single RGB image into its reflectance, shading (caused by direct lighting), and shadow (caused by occlusion) images.

Homepage of Yongjie Zhu [朱勇杰]
© Yongjie Zhu. All Rights Reserved.