The fuel rail in the injector pump is spring loaded.When you turn the switch to start voltage is supplied to the injector solenoid which retracts the solenoid plunger. The unit starts and you release the key. A device called the controller then supplies voltage to the solenoid to continue holding it open so the unit will continue running.
You can remove the solenoid from the injector pump, start the unit and it will continue running, but you cannot shut the engine off. Turn the key to off does nothing. You will have to choke the air off to stop the unit or take a short 3/8 inch drive extension and gently and slowly insert it into the hole you removed the solenoid from until you push the spring loaded rail in and stop the fuel supply and stop the engine. Your problem Is either the injector pump relay or controller not switching the voltage from start to run to keep the solenoid retracted.