What does "being there" have to do with it?  The guy who owns a duplex and rents out the half where he doesn't live, is certainly "there" a lot of the time. 

In a 1998 Tax Court case (Harris), IRS argued that the $25,000 limit on passive activity losses applied to a self-storage building.  The judge agreed.  If you aren't going to let me take a loss, I'm not going to pay you SE tax if I show a profit.