In machine dynamics, the tool point frequency response functions (FRFs) are employed to predict the stable machining conditions. In this paper, a combined analytical–
experimental substructuring procedure is proposed to determine the tool point FRFs usable for different holder–tool configurations. Contact interface of holder–spindle and tool–holder is modeled using translational and rotational springs and dampers spread in the length of contact surface. These joint parameters are defined using finite element method. This enables the analyst to introduce the contact stiffness and damping in more detail with taking into consideration the variations of normal pressure in the tool–holder and holder–spindle joints. The dynamic analysis of the holder is done using Timoshenko beam theory by Tchebyshev method. The tool dynamics is modeled based on Euler–Bernoulli beam theory using the method of equivalent diameter. For the purpose of shifting the tool stability lobes to a higher level, tool damping parameter is modified by internal frictional damper and the effect is analyzed by analytical methods and experimental study. After joint parameters are defined continuously by finite element method, a new method for continuous dynamic coupling is presented. The method employs the measured spindle-machine FRFs and analytical models of the tool and holder to predict the tool tip FRFs. In this new method, continuous coupling in two separate domains of response model and modal model is presented. Such structural modeling avoids us to do complex modal tests for a different set of combinations of the holder and tool with specific milling machine. An experimental case study is provided to demonstrate the applicability of the proposed method in dynamic modeling of machine tool.