Given the chance, cattle tend to be "ambush" grazers. They leave the trees or brush, graze, and retreat to the trees or brush to rest and chew their cud. Forested shade keeps the fly pressure down on animals.
Goats, like deer and llamas are browsers and prefer woody leaves and stems. All are great for clearing undergrowth. Goats are born acrobats, and think nothing of scaling trees or fences, so goat proof fencing is not a small thing, at least in my experience.
Like others here, if it were my land, I would fence as close to the boundary as possible. If that means taking out big trees, I would chat with neighbors about it. I don't have strong feelings either way, just that if you do run wire by trees you use pressure treated wood spacers to keep the wire, and nails, out of the trees. You will thank yourself later.
All the best,
Peter