I am a land surveyor in Illinois and Iowa, so I am not really sure how things work in Pa. Some state require tags to be put on monuments with the surveyors license number on it, or caps on top of the corner marker, which sounds like what you are describing. In a perfect world, all surveyors would come up with the same location for your corner. It isn't a perfect world, so sometimes we disagree. Sometimes there is a conflict in the deeds that gives a different locations, sometimes a surveyor see the evidence of the boundary in a different manner than someone else.
You should not have pulled the marker. Like someone else said, it could be some kind of offset, or it may not have anything to do with the boundary. If the surveyors license number was on it, you should have contacted him and asked him "whats up". I welcome people who call me or ask me in the field what we are doing. I like to educate the public and let them know what we are doing. I would say in the future, don't pull the markers, most states have a law against it. If it is just a wood stake that makes the corner visable on a temporary basis, there is nothing wrong with pulling them once you get tired of mowing around it, but the metal stake in the ground should be left alone.