Researchers present LegoGPT — a neural network that generates assembly schemes for models from Lego pieces
Researchers from Carnegie Mellon University introduced the LegoGPT neural network, which generates assembly instructions for models made from Lego bricks. The project's code is open source, and a free demo is available on Hugging Face.
The neural network takes a text description of a model made from Lego bricks as input. Based on this data, the system generates three files:
output.png— an image of the assembled model.
output.txt— step-by-step assembly instructions in text format.
Assembly instructions in .txt format
output.ldr— instructions in LDraw format, used in Lego CAD programs.
Researchers note that the neural network generates stable models where bricks are attached to each other. The repository also includes code for a model texturing system. The project authors emphasize that LegoGPT is a fine-tuned version of Llama-3.2-1B-Instruct.
What LegoGPT was trained on
Data from the ShapeNetCore dataset was used for training. Based on each 3D model, researchers created figures from Lego bricks in various styles while preserving the overall shape. The models were then checked for integrity using Gurobi to ensure there were no floating blocks. For each model, frames from 24 angles were created, and detailed descriptions were generated using OpenAI's GPT-4o.
In total, the dataset contains 47,000 figures made from Lego bricks, based on 28,000 3D models. All models fit within a 20 × 20 × 20 unit brick area. Researchers also note that the neural network was trained on objects from 21 categories, including furniture, cars, musical instruments, ships, and interior items. Therefore, the neural network cannot generate items of other types.
How to get access
The LegoGPT code is published in an open GitHub repository. A free demo is available on the Hugging Face portal. It lacks the figure texturing function and LDraw format export. Researchers also shared the dataset used to train the neural network. The repository includes instructions for running LegoGPT locally and fine-tuning it.
