PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs

Wentao Hu1,2*     Jia Zheng3*     Zixin Zhang4*     Xiaojun Yuan4     Jian Yin1,2     Zihan Zhou3
1Sun Yat-sen University 2Guangdong Key Laboratory of Big Data Analysis and Processing 3Manycore Tech Inc. 4University of Electronic Science and Technology of China
*These authors contributed equally to this work

PlankAssembly reconstructs 3D CAD model from three orthographic views.


In this paper, we develop a new method to automatically convert 2D line drawings from three orthographic views into 3D CAD models. Existing methods for this problem reconstruct 3D models by back-projecting the 2D observations into 3D space while maintaining explicit correspondence between the input and output. Such methods are sensitive to errors and noises in the input, thus often fail in practice where the input drawings created by human designers are imperfect. To overcome this difficulty, we leverage the attention mechanism in a Transformer-based sequence generation model to learn flexible mappings between the input and output. Further, we design shape programs which are suitable for generating the objects of interest to boost the reconstruction accuracy and facilitate CAD modeling applications. Experiments on a new benchmark dataset show that our method significantly outperforms existing ones when the inputs are noisy or incomplete.

PlankAssembly Shape Program

A cabinet is typically assembled by a list of plank models, where each plank is represented as an axis-aligned cuboid. A cuboid has six degrees of freedom, which correspond to the starting and ending coordinates along the three axes:

Cuboid(xmin, ymin, zmin, xmax, ymax, zmax).
Each coordinate can either take a numerical value or be a pointer to the corresponding coordinate of another cuboid (to which it attaches to).

PlankAssembly Dataset

The PlankAssembly Dataset consists of 26,707 shape programs derived from parametric CAD models.

User Editing


  author    = {Hu, Wentao and Zheng, Jia and Zhang, Zixin and Yuan, Xiaojun and Yin, Jian and Zhou, Zihan},
  title     = {PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs},
  booktitle = {ICCV},
  year      = {2023}


This work was done during Wentao Hu's internship at Manycore Tech Inc. This work was supported in part by the Key R&D Program of Zhejiang Province (2022C01025). Jian Yin is supported by the National Natural Science Foundation of China (U1911203, U2001211, U22B2060), Guangdong Basic and Applied Basic Research Foundation (2019B1515130001), Key-Area Research and Development Program of Guangdong Province (2020B0101100001).