In this paper, two novel hole-pattern seals were assessed for applications at the balance piston in a 14 MW supercritical CO2 turbine, focusing on the improvement of the seal leakage and rotordynamic performances. These two novel hole-pattern seals were derived from the conventional straight-through hole-pattern seal (HPS) with the same sealing clearance, diameter, axial length, hole diameter and depth, including a stepped hole-pattern damper seal (SHPS) and a grooved hole-pattern damper seal (GHPS). To enhance the seal net damping capability at high inlet preswirl condition, a straight swirl brake also was designed and employed at seal entrance for each type seal. A comprehensive assessment and comparison was conducted on the conventional HPS and the present two novel hole-pattern seals (SHPS and GHPS) with a static concentric rotor. The leakage flow rates, rotordynamic force coefficients, cavity pressure, and swirl velocity developments were analyzed for three hole-pattern seal designs with/without swirl brakes at two inlet preswirl ratios (0.1, 0.5), using a transient computational fluid dynamics (CFD)-based perturbation method based on the multiple-frequency elliptical-orbit rotor whirling model and the mesh deformation technique. To take into account of real gas effect with high accuracy, a table look-up procedure based on the National Institute of Standards and Technology (NIST) database was implemented, using an in-house code, for the fluid properties of CO2 in both supercritical and subcritical conditions. Results show that the present two novel hole-pattern seals have better sealing capability, especially for the GHPS seal which leaks less by a factor of 44%. In general, the GHPS seal possesses the lowest positive effective stiffness, highest effective damping, and the lowest crossover frequency of 60–70 Hz, especially at high inlet preswirl case. From a viewpoint of the rotor stability and unbalance sensitivity analysis, the GHPS seal without entrance swirl brake is a better seal design scheme for the balance piston seal in sCO2 turbine.