Get them every year in the lawn. They look nasty but never had a bad encounter with one. They are impressive hauling a payload of cicadas to their burrow. When you first see one you think it is a giant yellow jacket and run like ****. They will buzz you but never seem to attack anything but cicadas.
I'm inclined to leave them alone as well, but that stinger really looks like it could do a number on you.
A couple years ago I got nervous about a colony of black wasps living under my deck, so I wiped them out one morning while they were all cold and lethargic. Then I read that they are very passive, and that their primary prey is black widow spiders. I still feel bad about killing them all.
We have small wasps that live in the ground in large colonies. Name for them around here is "Ginny wasps". Step on their home and you get 8-12 stings within a half second. Very painful. Had to rush a dog to the vet when she made that mistake.
They do make quite a mess in a manicured lawn. The elderly neighbor lady just hates them. For several years she would call me about this time of year to dispatch as many as I could. I kept telling her they are harmless. But they scared her and she didn't like the numerous dirt piles.