My job frequently requires that I go to downtown DC and until a couple of years ago I worked in downtown DC and commuted via the metro.
Here are some personal observations.
1) The DC metro underground system has grown more crowded, dirty, less reliable and less "safe" over the years. It used to be a pleasant experience. It is no longer.
2) The National Mall, where all of the monuments, and the Smithsonian are located is generally safe and clean
3) #2 above applies to the Mall and 1-2 blocks on either side - during daylight hours. Beyond that area, DC is like any other large city with it's share of panhandlers, homeless, crime, etc. After dark, I would not feel safe ANYWHERE in DC.
4) #2 above does not apply when there is an "event" going on on the Mall - either a celebration like 4th of July or a protest march. The area very quickly becomes crowded, dirty, grid-locked and extremely unpleasant.
If you are careful, going to DC to visit the monuments and Smithsonian can be a great experience!