Yes...you should be concerned (at least theoretically).

The AICPA has a template for what the letter should say, and so far it has worked for any client on whose behalf I've had to send it out to a lender.  

It's a full page that basically says "read the g*damn return, do your own due diligence, and I have NO responsibility to you if they default on the loan (which, is what the lenders are hoping for - someone to go after if the borrower doesn't make the payments).

edit - link to example (last page):

https://www.aicpa.org/content/dam/aicpa/interestareas/frc/downloadabledocuments/third-party-verifica...