ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment

Wanjiang Weng^1,2^*, Xiaofeng Tan^1,2^*, Hongsong Wang^1,2 Pan Zhou³

¹ Department of Computer Science and Engineering, Southeast University, Nanjing, China
² Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications
³ Singapore Management University

^* Equal contribution

Code arXiv

TL;DR: We introduce the first bilingual text-to-motion dataset and the corresponding model for bilingual text-to-motion generation, and a plug-and-play reward-guided alignment to further enhance generation quality.

Bilingual text-to-motion generation, which synthesizes 3D human motions from bilingual text inputs, holds immense potential for cross-linguistic applications in gaming, film, and robotics. However, this task faces critical challenges: the absence of bilingual motion-language datasets and the misalignment between text and motion distributions in diffusion models, leading to semantically inconsistent or low-quality motions. To address these challenges, we propose BiHumanML3D, a novel bilingual human motion dataset, which establishes a crucial benchmark for bilingual text-to-motion generation models. Furthermore, we propose a Bilingual Motion Diffusion model (BiMD), which leverages cross-lingual aligned representations to capture semantics, thereby achieving a unified bilingual model. Building upon this, we propose Reward-guided sampling Alignment (ReAlign) method, comprising a step-aware reward model to assess alignment quality during sampling and a reward-guided strategy that directs the diffusion process toward an optimally aligned distribution. This reward model integrates step-aware tokens and combines a text-aligned module for semantic consistency and a motion-aligned module for realism, refining noisy motions at each timestep to balance probability density and alignment. Experiments demonstrate that our approach significantly improves text-motion alignment and motion quality compared to existing state-of-the-art methods.

Visual comparison of bilingual & monolingual motion generation results.

Observations reveal that: (1) in bilingual motion generation, MDM and MLD exhibit limitations in processing bilingual inputs; (2) in monolingual motion generation, generated motions demonstrate persistent misalignment with input texts. The left figure shows that our BiMD successfully generates motion from both English and Chinese inputs. Furthermore, the right figure highlights that BiMD, integrated with our ReAlign, successfully mitigates the misalignment issue.

Reward-Guided Sampling in Diffusion-Based Motion Generation.

Illustration of the sampling process in diffusion-based motion generation frameworks. The blue region represents the sampling distribution \( p_t(\cdot) \) learned by the diffusion model, while the green region depicts the ideal sampling distribution \( p_t^I(\cdot) \) achieved by incorporating our proposed reward-guided sampling strategy with the sampling distribution \( p_t(\cdot) \).

More visual comparison of monolingual motion generation results.

Visual comparison of motion generation results on the HumanML3D dataset. Our proposed BiMD method with ReAlign integration improves alignment between text descriptions and generated motions and enhances overall motion quality compared to MLD. The red text denotes descriptions inconsistent with the generated motions.

Main Results

Comparison of text-to-motion generation performance on the HumanML3D dataset

Comparison of text-to-motion generation performance on the HumanML3D dataset. Percentages in subscripts indicate improvements over respective baselines. Our BiMD adopts a similar backbone of MLD's, and surpasses it on all metrics.

Comparison of text-to-motion generation performance on the KIT-ML dataset

Comparison of text-to-motion generation performance on the KIT-ML dataset. Since the models MLD and MLD++ for the KIT-ML dataset have not been released, we use the widely used MDM as the baseline.

Plug-and-Play Functionality of ReAlign

Performance enhancement of motion generation methods with plug-and-play step-aware reward guidance. Results are evaluated on the HumanML3D dataset, with improvements reported relative to baseline methods. Here, MLCM¹ and MLCM⁴ denote the 1-step and 4-step model in MotionLCM. MDiff is an abbreviation of MotionDiffuse.