Crap, I also regret opening and reading through this thread.
I just turned 40. My house is done and easy to live in, but I have a lot of desires to add sheds, outbuildings, a sauna, etc. around the property. The idea of having my own mill to make these structures (and other fun woodworking ideas) out of black cherry, walnut, and oak logs is SO TEMPTING.
Three things that are holding me back:
1) Cost. A good mill, outfitted the way I want it, is $6-8000. Not a trivial expense, and would take a LONG time to pay back from lumber savings alone
2) Covered storage. I can't stand to see equipment sit outside a rot, and it seems a sawmill should DEFINITELY be kept under a roof.
3) Time; I don't have much spare of it, with my full time career and two younger kids. What if I buy a mill and never have time to use it? That would just be sad.