I've had this same problem this week! I have a raspberry patch in my backyard that I look forward to picking from every year--last year it netted me 5 large bowls' worth--but now I get attacked by these aggressive blackish bees if I disturb a certain area. I've not been able to see a nest yet--the patch is quite dense and, admittedly, a bit of out control--but I know exactly which branches to rustle to get them flying up and around. They dive-bomb my head, and I've felt them fly into my face and hand. I normally wear a full-body rain jacket as protection from thorns and mosquitoes, and the window screen mesh helmet I whipped up works great, but I still feel vulnerable in the hands, especially since I like to keep the picking hand ungloved. Based on the glimpse I got of the two that landed on the towel wrapped around the hand holding the flyswatter, I'd say they're too black to be Africanized bees, and look more like the carpenter bee.
I know I don't want to use poison. I'm thinking of sprinkling some silica gel on the area I think the nest is and just drying them out. Maybe I can rig something up that just keeps shaking the branches and doesn't let them resettle and makes them give up their home? Or maybe I'll just just go into battle for a little time each day. I've killed some, but it's nowhere near enough to make me feel safe about picking the patch completely clean.