The pin needs to be held vertical by the frame on the mower. The caster is built into the fork. Exactly like a shopping cart wheel. The caster makes the wheel self-centering. More caster means stronger self centering.
If the wheel is turning to one side, the frame holding the wheel to the mower may be bent and the pin tilted to one side. The frames that are made of small angle iron like this one can get bent pretty easily by backing into stuff.