Extensive research has been performed to investigate effective techniques, including advanced sensors and new monitoring methods, to develop reliable condition monitoring systems for industrial applications. One promising approach to develop effective monitoring methods is the application of time-frequency analysis techniques to extract the crucial characteristics of the sensor signals. This paper investigates the effectiveness of a new time-frequency analysis method based on Empirical Model Decomposition and Hilbert transform for analyzing the nonstationary cutting force signal of the machining process. The advantage of EMD is its ability to adaptively decompose an arbitrary complicated time series into a set of components, called intrinsic mode functions (IMFs), which has particular physical meaning. By decomposing the time series into IMFs, it is flexible to perform the Hilbert transform to calculate the instantaneous frequencies and to generate effective time-frequency distributions called Hilbert spectra. Two effective approaches have been proposed in this paper for the effective detection of tool breakage. One approach is to identify the tool breakage in the Hilbert spectrum, and the other is to detect the tool breakage by means of the energies of the characteristic IMFs associated with characteristic frequencies of the milling process. The effectiveness of the proposed methods has been demonstrated by considerable experimental results. Experimental results show that (1) the relative significance of the energies associated with the characteristic frequencies of milling process in the Hilbert spectra indicates effectively the occurrence of tool breakage; (2) the IMFs are able to adaptively separate the characteristic frequencies. When tool breakage occurs the energies of the associated characteristic IMFs change in opposite directions, which is different from the effect of changes of the cutting conditions e.g. the depth of cut and spindle speed. Consequently, the proposed approach is not only able to effectively capture the significant information reflecting the tool condition, but also reduces the sensitivity to the effect of various uncertainties, and thus has good potential for industrial applications.