A new joining process for magnesium alloys, rotation friction drilling riveting (RFDR), is proposed in this paper. In RFDR operation, a semitubular rivet with a grip rod and a rivet cap rotating at high speed is brought to contact with the riveted sheets, generating frictional heat between the rivet and riveted sheets, which softens the sheet materials and enables the rivet to be drilled into the sheets under reduced force. While being inserted, the rivet pierces through the top sheet and flares into the bottom sheets due to the action of the cavity die, thereby forming a mechanical interlock between the rivet and riveted sheets. Our studies showed that RFDR of the magnesium alloy sheets could be carried out at room temperature and provided the joints with superior shear strength and fatigue property when compared with self-piercing riveting (SPR). The effects of the operating parameters of the RFDR process on the quality of the joints were also investigated in the study. The results showed that while the rivet rotating speed little affected the shear strength of RFDR joints, the rivet shank length and the downward pressure had a significant influence on the mechanical properties of the RFDR joints. Therefore, it is very important to choose the right rivet shank length and downward pressure for producing RFDR joints with high quality.