Researchers from MIT, Google, and Northeastern University developed InstructMesh, an AI-powered design system that enables users to generate, edit, repair, and fabricate 3D models using natural-language instructions.
The system addresses a common weakness in generative 3D AI: models can create objects that visually resemble the requested item but do not necessarily function properly when physically manufactured.
For example, an AI-generated cup might look realistic while containing structural flaws that prevent it from holding liquid.
InstructMesh combines Microsoft’s TRELLIS 3D-generation system with GPT-4’s language and reasoning capabilities.
Users can generate an initial design and then highlight individual areas that need modification.
The system interprets natural-language instructions and allows users to review changes before fabrication.
MIT researchers demonstrated the platform by creating items including a dragon-themed mug, a shell-shaped whistle, butterfly-wing glasses, an octopus-style beverage dispenser, a denim-style knee brace, and a robotic enclosure.
The research team also tested whether people without 3D modeling experience could identify and fix structural problems.
Nearly 80% of models recreated by TRELLIS from popular Thingiverse designs contained some type of structural flaw.
Novice users were able to identify and repair those problems approximately 90% of the time using InstructMesh, according to evaluations by an expert.
Researchers envision future versions incorporating physics simulations to predict whether objects will withstand particular uses or which materials would be most appropriate.
The technology could also eventually be incorporated into augmented reality systems.
KEY QUOTES:
“We wanted to bring together the talents of 3D generators and the reasoning skills of LLMs in an interactive space to make objects that people actually want.”
“The users got what they prompted for and easily tweaked designs where needed. What they saw is what they got, and the items worked as advertised, so to speak.”
Faraz Faruqi, Lead Author and Former MIT CSAIL Researcher
