The text regarding the neglect of fellowship is counsel and warning... Not a command for worship. Not fellowshipping for a period of time does not mean a change in loyalty. And fellowshipping does not necessarily mean going to a building. You can still meet with your unbelieving neighbor in order to share food, the gospel, and friendship. A form of fellowship that I would suggest meets the criteria.
Like so many things today we interpret it through the eyes of the modern-day church and then apply it to what it says in the bible. It is more helpful to understand what the bible says and then apply it to the modern-day church. Sad to say if what the bible says conflicts with denominational doctrine, the bible is usually given the flick.
Meeting in buildings is totally irrelevant to the New Testament Church because they had none and their main venue was the home. However, people today have this weird idea that to meet together you have to go to a dedicated building set aside for one purpose. Religious activity. And that is what it is. Religious activity. Fellowship and relationship which is what the New Testament Church was all about, is minimal today.
The NTC met every day mainly to share a meal together as a simple practical matter, making sure everyone had a main meal that day which was not always the case with a lot of believers because of their poverty. That is what breaking of bread means. To have given them a sip of wine and a piece of bread would have been a total insult to the needy believers.
Around such a meal which was eaten in a home, the other three pillars of the church took place. Prayer, fellowship, and the apostle's doctrine.
If you can't meet in a church building, nothing to worry about. You can meet in NT fashion. Where two or three are gathered together in His name.
And I can guarantee what you will get out of it will be much more than a religious meeting in a building for that purpose.