MIT, New York University unveil FLEX, which incorporates natural-language implications into motion planning

Researchers at MIT and New York University unveiled FLEX, a pipeline that incorporates unstated intent in natural-language instructions into a robot’s motion planning. It infers common traits among objects to avoid from a user’s instructions and extends the plan to objects not explicitly mentioned. The system was tested in simulations with a robotic arm.

With FLEX, an LLM infers common traits among objects and extends motion planning to objects the user did not specify, rather than requiring users to name each object to avoid individually. MIT said testing was limited to simulations, so how to prevent misunderstandings or incorrect generalizations in real-world settings such as homes will be a factor in evaluating its implementation.

FLEX’s distinguishing feature is that it connects natural language not just to direct commands for a robot, but also to editing the feature list used in motion planning. If a user says, “Don’t spill on the laptop,” for example, the plan can extend to avoiding a nearby phone as well as the specified device. Conventional “Add” adds individual objects each time an instruction is given, while “Generalize” infers common traits among objects. The design expands the role of natural language from listing instructions one by one to generalizing a small number of instructions and incorporating them into a plan. However, this generalization is not left to a language model alone. GPT-4o infers attributes such as “fragile” from specified objects, while computer vision checks whether the objects are still present in the environment and removes absent objects from the list. In other words, the system separates semantic inference by the language model from updates to the environmental state using vision. The roughly 5-second processing time is for the sequence of operations reported by the team; it does not indicate control latency or the overall responsiveness of a robot in real-world operation. In a previous report, [we covered a method from MIT for correcting a robot’s inaccurate object memories using images taken from different viewpoints](/en/news/mit-develops-method-to-correct-robots-false-object-memories-using-imag-bb2f60), a study that revisited object descriptions by comparing images from different perspectives. FLEX does not correct errors in object memory; instead, it infers unstated preferences relevant to planning from a user’s brief instruction. The two systems address different stages, but both examine how robots can mishandle human intent, in memory and motion planning, respectively. The industrial question is how to balance the possibility of reducing the need to enter individual rules by generalizing instructions against the effects that incorrect generalizations could have on routes or task outcomes. The research presentation showed three types of simulations, but did not show how well generalization holds across diverse objects and changing conditions in real-world environments. The next questions are how to detect and correct incorrect inferences of common traits by the language model, how errors in visual recognition affect the feature list, and how the system performs in terms of safety and processing time on a physical robot.

FLEX operates in three stages: “Add,” “Generalize” and “Mask.” After adding objects specified by the user to a feature list, an LLM (GPT-4o) infers common traits among them, and a standard computer vision system removes objects from the list if they are not in the environment. The sequence takes about 5 seconds. The team conducted three types of simulations: carrying coffee, moving a candle and carrying a bowling ball. It demonstrated actions that avoid liquid contacting electronics, approaching flammable objects and dropping a load onto fragile objects.