John,
I have not owned a backhoe with any of my tractors, but I have owned both 3000 and 4000 series machines and can quickly contrast the differences. I currently have a 3720 which is about the perfect machine for me. It has a lot of power and operates a loader quite well. I also had a 4310 (similar to the 3320) and a 4520. The 4520 was a hoss. It is much larger, which is both good and bad. Pulling heavy duty attachments it did great. It had ample power and with the increased weight could operate larger attachments well. It also had a very stable feel with loaded tires, and rode great over bumpy terrain or with a full loader bucket. The 3720 is its equal in power, the 4310 would not compare with either. With a bucket full, or on rough ground, a mid frame and does not ride quite as well over bumps. My 3720's loader performance with a 61" bucket is similar to the 4520 with a 72". It is much more manuverable, though. I can fit my 3720 into spaces that the 4520 had no hope of going. I can almost go where my former 595 could as far as space, but with much more capability. In more open spaces the 4000 series would be a better choice, but if you have tight areas or soft ground the mid frame is the choice. There are many areas on my property where the 3720 would work circles around the larger machine due to its manuverability. It is heavy enough to do heavy work, and with 35 PTO HP can easily power most any PTO, or hydraulic attachment. The weight could be an issue with a backhoe, but I think either machine would have limitations here in certain cases. As for your trailer, don't do it with a large frame. My neighbor with the 4320 hauls his on a 7K trailer and it REALLY does not appear safe. I had a 7000# trailer initially with my 4520 and it was a load!!!!! You can figure a 4120 spec'd to your needs would weigh 7000# alone and both your trailer and tow vehicle would have a job. I currently have a Gator-Made 14K trailer and it would haul the 4520 easily and barely notices my 3720. Do not scrimp on the trailer. On the other hand, if you like the 7K trailer it would haul a mid frame fine.
John M