The bible doesn’t say everyone was at Babel just that there was one language and these were the people who went east. These people were punished because of autonomy verse 4 confirms it
4 Then they said, “Come, let us build ourselves a city, with a tower that reaches to the heavens, so that we may make a name for ourselves; otherwise we will be scattered over the face of the whole earth.
You see after the flood God gave the command to go and multiply the earth but these people chose their own will for their lives over Gods will for their lives.
Autonomy is the fall of man kind as God knows what’s best for us and wants us to live out His will for us in our lives but we go against God and choose our own will. Satan knows this and temps us with it the first example of this was Eve in the garden.
3 Now the serpent was more crafty than any of the wild animals the Lord God had made. He said to the woman, “Did God really say, ‘You must not eat from any tree in the garden’?”
2 The woman said to the serpent, “We may eat fruit from the trees in the garden, 3 but God did say, ‘You must not eat fruit from the tree that is in the middle of the garden, and you must not touch it, or you will die.’”
4 “You will not certainly die,” the serpent said to the woman. 5 “For God knows that when you eat from it your eyes will be opened, and you will be like God, knowing good and evil.”
You see Eve didn’t want to be subject to God which was the will of God for her she wanted to be like God.