The specific cutting energy in machining is known to increase nonlinearly with decrease in uncut chip thickness. It has been reported in the literature that this phenomenon is dependent on several factors such as material strengthening, ploughing due to finite edge radius, and material separation effects. This paper examines the material strengthening effect where the material strength increases nonlinearly as the uncut chip thickness is reduced to a few microns. This increase in strength has been attributed in the past to various factors such as strain rate, strain gradient, and temperature effects. Given that the increase in material strength can occur due to many factors, it is important to understand the contributions of each factor to the increase in specific cutting energy and the conditions under which they are dominant. This paper analyzes two material strengthening factors, (i) the contribution of the decrease in the secondary deformation zone cutting temperature and (ii) strain gradient strengthening, and their relative contributions to the increase in specific cutting energy as the uncut chip thickness is reduced. Finite element (FE)-based orthogonal cutting simulations are performed with Aluminum 5083-H116, a work material with a small strain rate hardening exponent, thus minimizing strain rate effects. Suitable cutting conditions are identified under which the temperature and strain gradient effects are dominant. Orthogonal cutting experiments are used to validate the model in terms of cutting forces. The simulation results are then analyzed to identify the contributions of the material strengthening factors to the size effect in specific cutting energy.