A friend has property similar to what you describe. She bought a JD 3032. I used it a couple of times and my complaint with the model was that it lacked cruise control. I mention this because it was a great tractor for what she used it for half the time (i.e. clearing out the barn), as it has easy to use hydrostatic drive and a nice loader. But, for brushhogging (the other half of its intended use), on rough ground, it bounced all over the place. This caused my foot to bounce with the tractor and took a lot more effort than it should have to cut down the pastures. I think this is why there's still gear tractors offered. Or at least hydrostatic ones with cruise control - set it and then you can pay attention to just steering and adjusting the 3pt.
My point in writing is, there's a lot of value in John Deere's "e" series tractors that don't have removable loaders. Perhaps this style might work for you? It wouldn't for me because it makes the tractor less functional on my small property.
But, if you do have to improve the arena area yourself, a backhoe may well be a devise worth investing in. This means perhaps getting yourself into a more powerful tractor.
I'd suggest that if you don't have a need for a mid-mount mower, take a hard look at the L-series Kubotas. They're more cost effective than a
B3030, with stronger capacities (shorter on luxuries, though). And, although they may look a little intimidating at first because they're bigger, my guess is after driving a few different tractors around, you'll come to appreciate their comfort and ease of use. Over rocky ground, the larger tires make for a better ride compared to a smaller tractor.
A lot of us aren't tied to one brand - some tend to stick with a certain color because they have a good relationship with their dealer. If you put in the time now to research and test drive several different brands and models, you'll save yourself a lot of money in the long run. Specifically, you won't be trading in next year. This effort will allow you also consider brands that don't appear popular at first, but perhaps have a lot better value for your application. I mentioned the "e" series from JD because they skimp on certain luxuries that make the tractor a lot more price competitive. But, New Holland, Kioti and Cub Cadet make perfectly fine tractors as well. If you're "lucky," perhaps dealers in your area sell multiple brands, so you can try out different manufacturer's approaches back-to-back.
So, I definitely recommend test driving a bunch. Then go back home while things are fresh and look at the specs - lift capacity at the loader and the 3-point hitch. Also, wheelbase and length/height overall. Then go back and test drive those couple of models that fit you and your husband's demands to see if you're comfortable on one specific size (both HP and physical) tractor. Enjoy!!