+1 on your neighbor's suggestion, & btw your soil reminds me a bit of my own.(?) (~50/50 sand/gravel) If fairly well drained, your existing base shouldn't need built up much. When a base is more porous than surrounding soil it can hold water. Coarse stuff will have plenty of voids and may need a lot of settling or compacting to prevent that. (often an issue here in our winter climate.)
'Crusher run' will 'lock down' like few others, and will 'track' around minimally once settled, but it will get dusty if you work it a lot. (in our mushy spring thaw conditions it will resettle/refinish quickly) Don't expect you BB to be the end-all for smoothing. You're bound to get some rippling when using it. If it ever snows down your way use the FEL vs a 'pavement plow' to clear gravel lanes, or let it melt off. You don't want 'curbs' piling up on the sides (with gravel a blade scrapes loose) that won't let rain water drain off to the sides.
Firmly tamping adjacent to asphalt will reduce crumbling & break-off if someone drives a bit wide on the paved portions. Plan to use a fair bit of Roundup to keep weeds at bay, knowing how they'll pop up everywhere, and esp ~12" off the pavement ('chemical trim') to keep an eye on/maintain what your mower will have to cross.
