A better way to turn 2D designs into 3D models for rapid prototyping

Engineers frequently employ vision-language models to generate new designs, such as for airplane or automobile parts. To scrutinize how these components perform under real-world conditions, they utilize computer-aided design (CAD) software to create 3D models of the designs, which can then undergo virtual crash or durability tests.

A team of researchers from MIT and other institutions has developed a system enabling a vision-language model to automatically convert 2D designs into CAD programs that are more precise and functional than previous methods, while requiring significantly less computational power. This innovation could streamline the rapid prototyping process, reduce costs, and assist engineers in recognizing advantageous design options they might otherwise miss.

The system generates new data based on the model's performance as it attempts to convert 2D images into CAD programs. By analyzing its failures and incorporating them into a dataset with successful solutions, the framework instructs the model on how to rectify specific errors and address challenging problems it would otherwise struggle with on its own.

Lead author Giorgio Giannone, a research affiliate at MIT's Design Computation and Digital Engineering (DeCoDE) Lab, explains that the goal is to equip engineers with the ability to present the framework with an underperforming CAD model, set a compute budget, and allow the system to autonomously transform the model's mistakes into enhanced training data.

"The motivation behind this work is to provide numerous image-to-CAD-code models with a means to improve themselves, learning from their own errors instead of relying on additional human-generated data," says Faez Ahmed, an associate professor of mechanical engineering at MIT and co-senior author of the study. The research was presented at the International Conference on Machine Learning.

The researchers are focused on developing vision-language models (VLMs) for CAD generation, which take a 2D image and descriptive text as input and produce Python code executable in a CAD software program to create a 3D model of a physical object. They examined the challenges of deploying existing VLMs for this task and identified the primary obstacle as the scarcity of diverse, high-quality CAD datasets required for effective training.

To address this gap, they created a data augmentation system called GIFT (Geometric Inference Feedback Tuning), which generates data tailored to improve the performance of a specific VLM for CAD generation.