New tool lets users repair AI-generated 3D models, then fabricate them just the way they want

A new tool called "InstructMesh" simplifies the process of designing and printing practical 3D models, such as household items, accessories, and robots. The software, developed by researchers at MIT, Google, and Northeastern University, addresses the common issue of AI-generated 3D models that may not work as intended due to unrealistic designs.

InstructMesh allows users to modify specific parts of the blueprint before printing, making it accessible for both experts and novices. By combining Microsoft's TRELLIS system and the large language model GPT-4, InstructMesh merges visual and textual knowledge to create 3D models from text and image prompts.

Researchers demonstrated its capabilities by creating unique 3D designs, such as a mug shaped like a dragon, a blue whistle resembling a shell, and glasses with butterfly wings. They also fabricated a knee brace resembling denim and a "bristle bot" resembling a colorful shrimp.

When tested with popular models from Thingiverse, nearly 80% of the generated designs had structural flaws. However, InstructMesh helped novices identify and fix these issues around 90% of the time. The tool's intuitive interface allows users to easily express their ideas while offering precision adjustments.

Looking ahead, InstructMesh could be integrated into augmented reality platforms, enabling the creation of items tailored to specific surroundings, like a phone case that matches a wallet. The software may also incorporate physics simulations to predict how designs will function in real-world scenarios and incorporate the more recent TRELLIS.2 for finer refinements.