"You can turn the corner and be in a bad area quickly!" -- if you find yourself near Johns Hopkins at night, run do not walk back where you came from. I would stay at the Inner Harbor as well. Lots in walking distance and everything you need. Little Italy is between Inner Harbor and Fells Point if you like Italian.
Also, I always liked Annapolis too - Pussers Landing on the dock.