Imagine a world where robots seamlessly respond to complex instructions like "walk backward, turn right, and then walk forward" or where AI can generate realistic human movements for animated movies based on simple text prompts. This is the exciting potential unlocked by MotionGlot, a groundbreaking AI model that generates motion across diverse physical forms, from robots to humans. Creating realistic and diverse movements for virtual characters or robots has always been a challenging task. Traditional methods often struggle to generalize across different body types or respond to nuanced instructions. MotionGlot addresses this by borrowing techniques from the world of Large Language Models (LLMs). Just as LLMs predict the next word in a sentence, MotionGlot predicts the next movement in a sequence, allowing it to generate a wide range of complex actions. The key innovation is the way MotionGlot represents movement. It converts motion trajectories into a sequence of discrete tokens, similar to how words are tokenized in language models. This allows the AI to learn patterns and relationships between movements, enabling it to generate novel sequences that are both realistic and follow instructions. To train MotionGlot, the researchers introduced a novel instruction-tuning template. This template allows the model to understand and respond to instructions across different embodiments. The researchers even demonstrated its ability to generate robot movements based on sentiments expressed in the instructions, such as a "joyful" walk versus a neutral one, associating different gaits with each sentiment. Furthermore, recognizing the scarcity of data for training robot movement, the team created QUAD-LOCO, a new dataset of expert-controlled quadruped locomotion paired with direction-based text annotations. For human motion, they enriched existing datasets with more contextually rich descriptions using GPT-4, creating the QUES-CAP dataset. This allows MotionGlot to answer questions with human-like movements, such as demonstrating "how someone would look if they were trying to get someone’s attention from across a noisy room." MotionGlot outperforms existing methods in several tasks, including text-to-motion generation and motion captioning. The results show not only improved accuracy but also increased diversity and realism in the generated movements. The ability to generate multimodal action distributions is particularly exciting, meaning the AI can suggest multiple plausible ways to achieve a specific goal. Imagine a robot navigating a complex environment – MotionGlot could offer several alternative routes, each with its own advantages and disadvantages. While the current research focuses on robots and human motion, the principles behind MotionGlot could be extended to other domains, such as animating animals or designing new forms of locomotion. This work represents a significant step forward in bridging the gap between language and movement, opening doors to a future where humans and machines interact more naturally and intuitively.