Just plain dirt can erode around the pipe. This is a case where clean fill is not as desireable. "dirty' fill, with stones, rocks, and plant matter would hold better. You would still need a good invasive ground cover to hold the outer layer of dirt.
But as was mentioned, for a permenant job, haul in the appropriate aggregate for your area. Gravel, crush-run, sand, rip-rap, etc.
There are many variables to care for, depends on the flow during a storm, which also depends on the dowhill grade, etc. etc. etc.? Is the area prone to flash floods during T-storms, that kind of thing?
When using local dirt from the site, I have seen folks then dump and tamp-in a lot of rocks on the upstream side. In highly erosive sitautions I have seen folks end up pouring a concrete apron around the upstream mouth of the culvert.
Storms like Erin aren't going to be your normal occurences (let's hope), and to try to build to that level may get quite costly. In a storm like Erin, you'll get overflow (water flowing above and over the road surface), but, otherwise, you want to try to build so that doesn't happen. If it continues to overflow, even in lesser storms, the culvert diameter may be too small.
If you concede that water is going to overflow the surface even during routine rainstorms, then you are probably looking at concrete around the culvert, as well as on the road surface.