It's harder to track, but you may actually make more because you accept cards. Basically, your customer base is considerably larger because people can use plastic. Many small businesses and consumers only look at individual transactions rather than the aggregate. This makes them inadvertently dissuade customers by penalizing them for their choices.
According to Gas Buddy, about 4% of people pay with cash. So, for every 100 customers, you would make a little more on 4 of them, but would potentially lose up to 96 customers if you did not accept plastic. Sure 1/2 of those are debit card transactions, but without the credit card, processing would not even exist. (it is a loser for the bank).