Yeah, I could see that.  If the investment advisor was charging ongoing fees (as many of them do), I'd stick them with the bill for the penalties.  (Not that they'd pay it, but it can't hurt to ask!)