Another possibility could be that you have a puncture, slice/rupture in the line going to the pump. Check if you have a check-valve at the pressure tank, on the line going to the pump. It should be brass, about 6 inches long and maybe 2 inches in diameter. If you do, then the tank would hold the pressure, but the line leaking would slow the fill time. And before anyone says that the check valve is at the pump, it is common practice to add another at the tank end of the line.
If you decide to pull the pump, there should be a shock arrester every 20 feet or so. These keep the pipe from rubbing on the well casing which is the major cause of pipe failure. These should either look like stars that fit on the pipe tightly with arms radiating out to grip the casing, or they will be a sleeve that clamps on the pipe and when you bring the ends together, it increases the diameter, again contacting the casing. If these are mising, then it sounds like the well guy tried to save himself a few bucks at your expense.
Another thing to check, is for a kink in the pipe. If it was kinked during installation, it creates a weak spot which has a good tendency to fail. Also, if there is a coupling in the line going down the well, it may have loosened up, allowing leakage. A coupler is always a no-no on the downward section. If there is one, and you don't want to change the line, make sure it is tight. Slightly heat the end of the pipe before you tighten the clamps.
Also, make sure you have a support rope or wire, and that the pump supports on this, and not the pipe. Stretching of the pipe is the 3rd leading cause of failure. This is all assuming that you are using poly pipe. If you have metal pipe, you might have leakage where the sections couple together.
Darn, I knew that being a plumber in a past life would pay off. The bill is in the mail. /w3tcompact/icons/laugh.gif