I'm definitely scared about my children being hurt. That's why I brought the question here. My dad had me disking (supervised from another tractor) on a WD45 at 6 years old. I think this was a little crazy. So my perspective is skewed, and because of that I don't know where a good middle ground is at.
That's why I thought about a smaller riding mower with the deck removed would be a good next step. And from this thread I'm thinking about adding a roll-bar, a bumper, remote shutdown, seatbelt with kill, and maybe a throttle or HST modification to block full speed. Those were my first take-aways from everyone's comments.
As for supervision this is going to be used for them following me driving around on my garden tractor doing work. But at some point, I'd probably feel they are capable of driving back to the house if they were getting tired, or needed to go to the toilet...ect. I can't predict how it would play out, it's my first time being a parent and I definitely don't have it figured out.