Try getting under the edge of it. Compacted gravel has allot of strength as long as it's in place, but when you get under it, you can break it up real easy. Concrete works this way too, but it's not as easy.
For pipe or conduit, I use a pick and start at the edge. I dig under the gravel and into the dirt, and it just falls apart. Then I just keep working my way across. I can cross my gravel driveway in an hour this way.
Good luck,
Eddie