You misunderstand. This physical world and body is a mere image, a manifestation of the spiritual reality on high. That which is born of the flesh is flesh, and that which is born of the Spirit is spirit. "Then the dust will return to the earth as it was, And the spirit will return to God who gave it."
Did you flunk chemistry? "Spirit" is physical, for "wind" is PHYSICAL (AGAIN, there is no distinction between both words in the Greek and Hebrew), and even though it is made up of substance we cannot see with the eye, it is indeed substance, molecules of O2, CO2, etc (chemistry 101). As such, a "spirit" is a PHYSICAL being, for it is made of WIND. And even though it is not FLESH AND BONES as we are, it is made up of substance just as we are made of substance, molecules and atoms that are all created by God, but a spirit exists in a DIFFERENT form than we do that God makes possible by his POWER. This is similar to how God turned an angel into the form of FIRE, that then appeared to Moses in that form. This relates to a scripture that declares, "God makes his angels into flames of fire, and his angels into winds (spirits)". God can turn you into a rock if he so pleased, and make it so that you can still see, think, hear, smell.... he is the creator, he can do whatever he pleases and make anything possible, by his POWER. Wind is created, just as the organic makeup of your body is also created. So something that is "spiritual" (adjective of the noun spirit), is speaking of something that pertains to the Spirit, just as something that is "electric" pertains to something that uses electricity. If I said my computer is electric, it does not literally mean that my computer is all electricity, it means that it uses electricity, while still being an actual object. So something that is referred to as spiritual, does not mean it is necessarily invisible, and DEFINITELY NOT FIGURATIVE, it just means that it pertains to the Spirit. Heaven is a spiritual place, because ANGELS WHO ARE SPIRITS LIVE THERE, BUT IT IS ALSO A PHYSICAL PLACE. Because not always do angels remain in the form of spirit, but transform back into a form of flesh and bones (and yes, holy angels can switch between the form of spirit and the form of flesh and bones), hence how the angels that appeared to Abraham, and ate and drank with him, because they had PHYSICAL bodies to do so! And even the manna that was given to the Israelites, was called "angel's food". This was PHYSICAL food, that they were able to eat. Heaven is a PHYSICAL PLACE! Yet it is far out of our reach and we do not know its exact location beyond where the stars are, and wherever its location is, is what is referred to as the "third heaven". The body of Jesus Christ was able to be touched and seen, "It is I myself! Touch me and see", and this same physical body, ascended into a physical place, the third heaven.
When the spirit is said to return to God, it means that your BREATH returns to God. For the breath IS the spirit. Read about it here The Soul and Spirit of the Natural Body, and the Spiritual Body of the First Resurrection | Wisdom of God .