Unlike ferrous materials, where the cementite (Fe3C) phase acts as an abrasive that contributes to flank wear on the cutting tool, most titanium (Ti) alloys possesses no significant hard phase. Thus, the origin of flank wear is unclear in machining Ti alloys. To address this question, a Ti-6Al-4V bar was turned under various conditions with uncoated carbide and polycrystalline diamond (PCD) inserts, most commonly used tool materials for machining Ti alloys. These inserts were retrieved sporadically while tuning to examine the wear patterns using a confocal microscope. To correlate the patterns with the microstructure of the original bar, the microstructure was carefully characterized using Orientation Image Microscopy™ (OIM) with electron-backscattered diffraction (EBSD). From the wear patterns, two distinct types of damage were identified: (a) microscopic and macroscopic fractures on the cutting edges and (b) scoring marks on flank faces. This paper demonstrates that both types of damage were caused primarily by the heterogeneity in hardness in the α-crystals, where the plane perpendicular to the c-axis in an α-crystal is substantially harder than any other direction in the α-crystal as well as the isotropic β-crystal. In addition to such heterogeneities, adhesion layer, ubiquitous to machining Ti alloys, detaches small fragments of the tool, which resulted in microscopic and macroscopic fractures observed on flank wear.