Is it a bank, or a credit union?  I let my clients pay their taxes themselves, but for direct deposit of refunds there are occasional problems with credit unions. 

What does the IRS notice say?  "We're going to try again," or "make other arrangements" ?