I'm not the one failing to understand that Galatians verse and subject brother. Paul said in that very same Galatians 4 chapter, "Now we brethren, as Isaac was, are the children of promise."(Gal.4:28). Why would Paul link Isaac with us as the children of promise, and of the freewoman?
It's simple. It's because God first gave His Salvation Promise through Abraham, and it was by Faith, and Abraham believed, and that Promise is about The Saviour and The New Covenant, even BEFORE the law was given 430 years later, like Paul said back in Galatians 3. The inheritance to Abraham was NEVER about the law, period. It was always involving Faith on The Saviour, even back to Abraham's day. And how did that Promise of The Saviour become known to the world? Through a believing remnant of the seed of Israel by Faith, which is Paul's subject of Hebrews 11 and Galatians 3 and Romans 9-11 even.
Gal 3:14-18
14 That the blessing of Abraham might come on the Gentiles through Jesus Christ; that we might receive the promise of the Spirit through faith.
15 Brethren, I speak after the manner of men; Though it be but a man's covenant, yet if it be confirmed, no man disannulleth, or addeth thereto.
16 Now to Abraham and his seed were the promises made. He saith not, And to seeds, as of many; but as of one, And to thy seed, which is Christ.
17 And this I say, that the covenant, that was confirmed before of God in Christ, the law, which was four hundred and thirty years after, cannot disannul, that it should make the promise of none effect.
18 For if the inheritance be of the law, it is no more of promise: but God gave it to Abraham by promise.
(KJV)
Oh, but you just skipped Galatians 3 about that Promise though, didn't you? Without understanding Galatians 3 first, understanding why Paul said Gal.4:28 about Isaac and us will be left hanging, which is exactly what you'v done in favor of your own tradition.
And you just don't get it about Abraham's household either. Hagar and Ishmael were cast out... of Abraham's household, which as an allegory is the opposite of your idea. The unbelieving scribes and Pharisee Jews were always trying to prove to Christ they were of Abraham's seed, when a lot of them were not (like the foreigner Kenite scribes). Why would they claim Abraham as their father? It's because of the Promise, not because of the law. Don't you see they were trying to supplant the Promise by Faith with the law instead, which is why they claimed Abraham?
Even John the Baptist knew that point about them...
Matt 3:7-9
7 But when he saw many of the Pharisees and Sadducees come to his baptism, he said unto them, O generation of vipers, who hath warned you to flee from the wrath to come?
8 Bring forth therefore fruits meet for repentance:
9 And think not to say within yourselves, We have Abraham to our father: for I say unto you, that God is able of these stones to raise up children unto Abraham.
(KJV)
The law was not given through Abraham, The Promise by Faith was, so you cannot just try to change that and infer Abraham is about inheritance through the law.
Nor by that can you try to change Abraham's inheritance in The Promise by Faith continuing through his son Isaac, then to Jacob (Israel), then to Joseph and his two sons. God's Promised Salvation by Faith is what God's birthright to Israel is about. It began with Abraham. That has nothing to do with bondage to the law. It's THAT part of the seed of Israel which continued in that Promise which YOU are missing in your theology, and it's no doubt because of men's traditions you've been taught, or picked up from who knows where.