Lately, virtual reality (VR) is being investigated for the field of construction. VR enables investigation from a stage where there is no actual object. The most common approach in VR is interaction with a controller. Therefore, the sense of body movements is often far from reality. Reflecting actual body movements is required for a more near-realistic experience in VR. In addition, VR linked to 3D CAD instantly enables the examination of design change. This paper proposes a method and framework for reflecting real body movements by tracking the body and real-time linkage of 3D CAD and VR. Azure Kinect and Leap Motion can track body movements and provide skeletal data. The VR body is linked to a real body by using a skeleton. The VR body enables to recognize how one's body moves in the VR space and its position in relation to other objects in the VR space. The ability to observe the distance between the body and VR space will expand the use of VR for this type of safety and buildability check. In addition, the tracked skeleton can be applied to measure the energy consumption of the body and extend the scope of analysis in VR. A real-time linkage between 3D CAD and VR was achieved by Rhinoceros and RhinoInside. In this study, steel-frame construction is used as an example of construction at heights and in confined spaces. The background behind the creation of this system is the liberalization of architectural shapes through 3D CAD. As the use of free shapes and dimensions in design and construction increases, the number of projects that cannot be handled by existing construction methods also increases. Therefore, a method for studying the implementation of each building should be developed to verify its buildability.