That depends on how your bank reports your account to the credit bureaus.
Many credit-scoring algorithms assign points based on the length of your overall account history, the age of your oldest open account, and the number of lenders reporting positive status. Because most banks treat unsecured and secured credit cards as entirely separate accounts, the benefits of switching could be offset by the impact of reclaiming your card's security deposit.
For example, Capital One has built a reputation for graduating their secured credit card customers to partially secured status, then removing deposit requirements entirely after months of on-time payments. If your account number stays the same, your credit score will keep climbing as your account gets older.
It's more likely that you'll start a secured credit card account with an issuer that charges under $50 per year for the privilege, like First Progress. Once you've built a strong enough credit report to attract pre-qualification offers from companies like Chase or Citi, you face a decision. Dropping the secured credit card will save you money in the short term, but it reduces your overall amount of available credit, your total number of active lenders, and the age of your longest open account. (You'll still get points for the date you first opened your secured account, however.) All of those factors will depress your credit score for a short time, during which you might find it harder to qualify for the best mortgage rates or the lowest insurance premiums.
If you're planning to buy a home, lease a car or change jobs within the next six months, you may find it less risky to lump the annual fee and keep your secured credit card for another year. In this scenario, you're growing your overall credit while increasing the number of lenders in good status. As long as you've got just a tiny balance on your original card, the potential positive impact on your credit score could help you save more money over the long run than you'll pay in fees and finance charges.
One final note that could put your question in a different light. Most credit-scoring models on their own don't account for the name or the type of lender when calculating your credit score. Therefore, a secured card with a $500 limit looks just as good as an unsecured card with the same spending power. However, many mortgage and car lenders analyze your raw credit report data when underwriting secured loans. Seeing a more selective lender on your list of accounts, like American Express, can sometimes trigger a better deal than making a decision based on your credit score alone.