Look into Form 4952. Did you file that in 2019 and 2020?
Did you deduct the property taxes in prior years?
"t was not reported as rental on Sch E" leads me to ask "Was it rented?"
