There's not a single person reading and/or posting on this thread that would tolerate their neighbor deciding how their property is managed. Not one.
I have a neighbor similar to this one. He built a new fence joining us. He decided to move it 8ft over on me, 1/4 mile long. I firmly told him the fence had to be moved and gave him 3 months to do it. Two days before that deadline it still hadn't been moved. I firmly talked to him again and said in two days I'm going to crank up my tractor with grapple and tear out the fence. I'll pile all the wire and steel posts in my scrap iron pile. I'll saw the hedge posts into firewood. He said "I'd rather you didn't do that". I said "then move your fence". That afternoon he started removing it by hand. At the end of the deadline day the fence was relocated where it legally belonged.
This situation needs to be handled similarly. No need for the Sheriff or Lawyers. This needs to be a man to man settlement. How it's handled will have a direct bearing on all future interaction with this neighbor.