Research towards automation of heavy construction machines for efficient and safe construction processes that are robust to various environments and disturbances has been conducted for many years. In this paper, we show two contributions towards this objective. Firstly, we explore the use of reinforcement learning to automate construction machines. Secondly, we evaluate the effectiveness of three methods to reduce learning time of reinforcement learning: designing reward function, pre-training with BC, and Changing frame-skip rate. Reinforcement learning approach is expected to gain robustness against disturbances through learning. We run experiments on two different realistic tasks. The first task is to reduce sway of a load suspended from a mobile boom crane, behaving as a single pendulum. The second task is to load an excavator bucket with soil with a hydraulic excavator. We demonstrate the effectiveness of algorithms using the reinforcement learning approach on the commercial simulator, Vortex Studio developed by CM Labs Simulations.