Add to above This is from pub 936  "Mortgage ending early.
If you spread your deduction for points over the life of the mortgage, you can deduct any remaining balance in the year the mortgage ends. However, if you refinance the mortgage with the same lender, you can't deduct any remaining balance of spread points. Instead, deduct the remaining balance over the term of the new loan.
A mortgage may end early due to a prepayment, refinancing, foreclosure, or similar event"