If this is your oldest account, closing it will drop your credit score, for a little while. You'll have to weigh the short term savings against the long term impact of keeping that account open.
FICO and other credit scoring algorithms put a heavy emphasis on the age of your oldest account. Assuming that your student American Express card was your very first credit card, that five year head start gives you a healthy bump to your FICO score. Canceling that card now will drastically shorten both the average and the total age of your credit history.
You're probably getting more value than you think from paying that annual fee. Let's pretend your student credit card costs you a $100 annual fee. If closing it drops your score by as many as 50 points, you might see a slight increase in your monthly car insurance or your renter's insurance premiums. They might not add up to $100, but they could be close. Regardless of what consumer advocates and some lawmakers think of the practice, some employers even use your credit score to decide whether you'll qualify for a job interview. Saving $100 now could cost you a chance at the next step of your career.
American Express has built a reputation for negotiating lots of things on behalf of customers, but waiving annual fees isn't one of them. However, you should call an American Express customer service agent anyway to find out how you can feel better about paying an annual fee on a credit card you rarely use. Depending on who you speak with, you might get your older card enrolled in a better rewards program, or you may qualify for an even better status on your newer card. Ask nicely, and you might be surprised.
If an older, unused credit card costs you money and you have no plans to get a new job or buy a house in the next few years, you'll survive a temporary drop in credit score. But, just a few years out of school, it may be in your best interest to make peace with the price of that annual fee.