I quite agree. So, was Cain a giant?
The Bible is silent as to whether Cain was a giant [like the offspring of the fallen angels (Gen. 6)].
You probably are aware of a person, documented in history to be very large, named 'Sargon'. Based upon my studies, Sargon's linkage to Cain is somewhat dubious.
My personal opinion is that Cain, and his offspring, were not giants. After Cain's conception [Gen. 3:6], God issued the [second] prophecy of the Bible. It was regarding Cain's offspring [Gen. 3:15]. The Bible further identifies this offspring [Kenites] to have infiltrated [and corrupted] the ancient Israelites. Logic dictates to me, that, in order to perpetuate such infiltration/corruption and fulfill the prophecy of Gen. 3:15, the Kenites would have to be the same size as the Israelites. If the Kenites were giants, it would be highly unlikely that they would have successfully infiltrated/corrupted the Israelites.
Whether God knew beforehand that Satan would attempt to mess up His plan for this second age through an effort to corrupt the bloodline to Christ is unknown. I suspect that God probably anticipated Satan's strategy. After all, Acts 3:20-21 indicates that the announcement that the Son of God would be sent in this second age was made in the first age. We see in Rev. 12:4 that Satan was scheming from the first age to 'devour' as soon as God's plan for Christ actually commenced. Accordingly, once God discovered Satan's actual plan [Gen. 3:6], God had the contingency plan all worked out [Gen. 3:15].
Naturally, my stated opinion leads to the question, 'Why then were giants the result of the mating of the fallen angels with flesh woman?'
Again, this is my opinion only -- as I cannot document all of it. I believe the distinction between Satan's offspring and the fallen angel's offspring, though both Satan and the fallen angels are in non-flesh bodies, lies in first age judgment and purpose for this second age. As you probably know, Satan was sentenced to perish in the first age [Eze. 28]. Instead of terminating Satan then, God is using Satan in this second age as part of God's plan to test all of us. Satan is not terminated until the white throne judgment of God. Satan's effort to corrupt God's second age plan is expected -- is part of God's plan.
Whereas, the fallen angels were not part of God's plan for this second age. The purpose of this second age is to give all those that followed Satan in the first age to get a single opportunity, born innocent into the flesh, to be saved. Those angels that refused to be born into the flesh most likely were among Satan's first age followers. Instead of being born innocent thru woman, they mated with flesh woman -- such acts were outside of God's plan. Hence, it is my opinion that God decided then [upon their mating] to have this hybrid offspring to stick out from the others. One way to do that would to allow all or some of this hybrid offspring to be giants.