This paper includes the findings of an experimental study on instabilities of the chip formation process during end milling of Ti6Al4V alloy and the influence of these instabilities on chatter formation. It has been identified that the chip formation process has a discrete nature, associated with the periodic shearing process during machining. The chip formed during machining of titanium alloy Ti6Al4V is found to be mainly with primary serrated teeth appearing in the main body of the chip. Secondary serrated teeth resulting from the coagulation of a certain number of primary serrated teeth also happen to appear at the free or constrained edge of the chip, especially when the system enters into chatter. In order to identify the interaction of these chip instabilities with the prominent natural vibration of the machine tools system components, the different mode frequencies of the vibrating components of the system have been identified using experimental and finite element modal analyses, and vibration responses during actual cutting have also been recorded using an online vibration monitoring system. The vibration signals in frequency domain (fast Fourier transform) have been analyzed to identify the chatter frequencies and the peak amplitude values. Chatter was found to occur at two dominant mode frequencies of the spindle. These mode frequencies at which chatter occurred have been compared with the chip serration frequencies in a wide cutting speed range for different conditions of cutting. It has been concluded from these findings that chatter occurs during end milling due to the resonance of the machine tools system component when the frequency of primary serrated teeth formation is approximately equal to the “prominent natural frequency” modes of the system components, which are the two mode frequencies of the VMC machine spindle in this particular case.