I'll turn it off if it's going to be idling for more than a few minutes or I'm working with another person and need to communicate. It's a quiet tractor but you're still raising your voice to be heard. I'll turn it off if I smell exhaust fumes as I don't like diesel exhaust. Sometimes I'll turn it off just to have quiet.
I think that the idea of letting diesels idle because they're hard to start comes from the old days of gas pony engines instead of electric starters and big truck engines that were hard to fire up. Modern CUT diesels start easily.