My advise for you is to look at the entire rainbow.
For this reason: inexperience!
You can research tractors for days, be dead set on one tractor maybe two, get it home and find out that the controls aren't to your liking or that you wished you bought a different tractor.
Tractor dealers won't let you take the machinery off site, they may have a demo for you to play around on, maybe they will let you start one and drive in circles in the parking lot lol. But that just scratches the surface of the machine.
Research will tell you the HP range you need to be in, it will also tell you what to look out for on a tractor when looking at them in person but until you really own one, you have no idea what you like about tractors and non of us can tell you what that is also.
So what I would tell you is to buy bigger than you think you need, there's a reason lots of guys on here say buy the next size up or double the size you think you will need, you can never foresee the tasks that will come up and eventually you will wish you had a bigger tractor.
Gardening is the only time you will prob wish you had a smaller tractor.
Which brings me to that question, how are you planning to engage the ground? How big of a garden? Most guys get tillers which are expensive but you can buy a old 1 or 2 bottom 12-14in plough and 6-8ft 3pt disk. If it's a smaller garden I would say tiller.
My first recommendation to you is plan on adding a 3rd function for the FEL and 2-3 rear remotes even if you don't plan to use them right away, it's something that will be mighty handy later for attachments and you won't regret it.
Most compacts are 4x4 anyway, I'm sure you can get a 2x4 but most are standard with 4x4.
I would agree with the 30-40hp range just based off your wants for mowing. But bigger is not bad.
What you need from a tractor is quick attach everything, quick attach bucket, quick attach FEL ext. some "economy" compacts come with loaders that are almost impossible to take off and buckets that bolt, makes finding attachments really difficult after.
Most use SSQA and JD has a proprietary quick attach, still very easy to use.
You want to order the tractor in the mind of making it "universal", there's 1000s of makers of attachments and in order to use them all you need more rear remotes, a 3rd function for the FEL and a quick attach on the FEL.
Another suggestion is if you see a neighbor on his tractor or maybe even a farmer in a field, pull into there driveway, wave them over and state your name, where your from and ask them if they have a min, and tell them that your in the market for a tractor, tell them your not experienced with tractors but ask them what they like and don't like about there's. You can gain first hand experience that way and you won't be so blind going into a dealer.
Tractor ratings are sometimes exaggerated, like lift capacity at the pin ext, so you may find that your FEL that's rated at 1100lbs can't lift or curl a 1000lb pallet of rocks because the weight isn't close enough to the pins. Which is why buying bigger is better.
My story started much like yours, I didn't have a tractor, never have but I researched the crap out of them, I calculated the exact tractor I needed based of FEL lift capacity, in the 30-40hp range I was gonna sign on a Mahindra 1533 HST but was on the fence about the 1538 HST. I looked at NH, Case, Mahindra, JD and Kubota and found the Mahindra to be the best value for what I needed. Less than a week before signing my dad found a used JD cabbed 4320 in one of those tractor trader magazines, we drove the hour and half one way to check this tractor out, the guy was a farmer that had a used machinery dealership kinda business on the side. So the Mahindra was 33hp and could lift 1500lbs, the JD was 48hp and can lift 1800lbs, way more tractor than I even thought I needed, I fell for that tractor and shook hands with the guy. 2 weeks later I brought it home (financing delay).
My use for a tractor is handling round bales for cattle, 6 months later I'm really really glad I didn't buy a smaller tractor, I find myself thinking that my tractor is borderline as small as I want and could see myself using a 5 series, the bales I'm buying now are over 1000lbs, prob closer to 1200lbs but with pallet forks my loader has a hard time curling them from the weight being so far away from the pins.
So I guess good luck, research for awhile, pay attention to what guys like and don't (some tractors require you to be a gymnast to operate them correctly)(duel brakes, deff lock and directional HST at the same time)) and get first hand experience anyway you can.