Marlin’s autotune is derived, if you go back far enough, from some LabView PID demo code, if I recall correctly. (By way of some Arduino PID demo code.) That technique (relay method) only works particularly well for first-order processes, so I don’t think it’ll be very effective for a stepper driven drivetrain… Could be wrong though.
Have you checked what the reflected inertia ratio is for the moving mass versus the rotor mass? That’s often an issue with servo tuning. There’s no guarantee PID feedback is suitable for all drivetrain configurations.