Do I need to back off on the extrusion speed or is there a better way to even out the starts?
You could try reducing the prime amount after a retraction. Or enabling coasting. Or both. I had to do both.