From high rpms to idle should be nearly instantly if under no load. The 504 uses the C-153 engine, and governor is pretty much the same one used on the C-123's, so I'm a little familiar with them. Open this link to show an exploded view of that governor.
CASE IH | Schematic, Manuals, Specifications and Diagrams for governor | MyCNHi US Store
The 2 most common problems that would cause what you are experiencing are #1, sticking fly weights (#27 in the exploded view) #2 the connection rod from the gov. rockshaft to carb. (#1 in the exploded view) Normally if spring # 10 breaks, it will allow the engine to runaway @ high rpms. If thrust bearing#31 has come apart, they most generally will only run at about 1/3 maximum rpms.
Before going much further into detail I have a few questions. #1, Is this a new to you tractor..?? Was it like this when you bought it..?? #2, If not a new to you tractor, did this seem to happen all at once, or come on gradual..?? #3 If this seemed to happen all at once, I have to ask did you work on it, and what did you do to it..??
I have to ask, because if the carburetor was removed to be cleaned, or for whatever reason when re-installed if the connection rod to the carb was installed from the engine block side, it will bind on the engine block and cause these symptoms. I know, because I did this very thing when I rebuilt the C-113 engine in my little BN Farmall (also pretty much the same Gov.) many moons ago.
If you look at the exploded view, you will notice the rod is turned to hook to the carb. through the hole from the carb body side towards the block. Small cotter pin goes through the rod to keep it in place on the engine block side. An easy enough mistake if you don't have a photographic mind, and apparently I don't. That was 40+ years ago, but a lesson I learned, and do remember to this day. Hopefully that is the problem, if not post back, and we'll go from there.