In recent years, higher requirements on vehicle performance and emission have been posing great challenges to lightweighting of vehicle bodies. Mixed use of lightweight materials, e.g., aluminum alloys and magnesium alloys, is one of the essential methods for weight reduction. However, the joining of dissimilar materials brings about new challenges. Self-piercing riveting (SPR) is a feasible process to mechanically join dissimilar materials, however, when magnesium alloy sheet is put on the bottom layer, cracks occur inevitably due to the low ductility of the magnesium alloy. Friction self-piercing riveting (F-SPR) process is a newly proposed technology, which combines the SPR with friction stir spot welding (FSSW) and has been validated being capable of eliminating cracks and improving joint performance. However, in the F-SPR process, the generation of the transient friction heat and its effect on interaction between the rivet and the two sheets are still unclear. In this paper, a three-dimensional thermomechanical-coupled finite-element (FE) model of F-SPR process was developed using an ls-dyna code. Temperature-dependent material parameters were utilized to calculate the material yield and flow in the joint formation. Preset crack failure method was used to model the material failure of the top sheet. The calculated joint geometry exhibited a good agreement with the experimental measurement. Based on the validated model, the transient formation of F-SPR mechanical joint, stress distribution, and temperature evolution were further investigated.