When I find the nasty white faced hornets I spray them, just about the only ones I spray with insecticide (they nest up under eves here, and if there is something irritating them down below they just drop straight down and sting. Real nasty).
Paper wasps are pretty laid back and I just ignore them unless they are in the way and have to be removed.
Yellow jackets tend to be fairly calm until the nest reaches about 100-200 insects, then they get aggressive in their territory, attacking anything moving nearby. I have missed nests when they are small then gotten nailed when the nest "went critical" and I was just walking by, like the previous 10 times.... but now the nest was big.
One way I use to remove the yellow jackets (or paper wasps) is a shop vac with several of the rigid tubes attached to the hose so I'm not too close to the nest. The yellow and white faced buggers go after movement, if you have enough nerve not to move AT ALL as they buzz around, they can be hoovered up. [if you think about it, they are zooming around fast, obviously not using scent for tracking, but light/movement, I haven't noted them using IR either. They sometimes sting anything nearby in a fit of fury, but usually try to nail the object that is moving. So, if you don't move, you might get a few stings but not a swarm. But if you flinch and run, well....] I stick the end of the nozzle in the nest hole and jiggle it, they get riled up and fly out the "entrance" into the shop vac. Every now and then I pull it out so the vac can inhale some that have returned to the nest and are still trying to find the entrance. Takes longer than insecticide, but you don't have the poison sprayed around the house.
Most of them don't seem to survive the trip into the vac, but some do, so I don't dump the vac for a while and even then use a trash bag to cover the bucket... there is enough dust that they take a while to clean their wings before taking off, enough time to get the trash bag on. Even so, I wait for a cool morning so they are even slower, just in case!