It comes from a forgery - the protoevangelium of James, which lied about its own authorship (claiming it was written by James when it was written 100 years later), contained no fewer than three direct contradictions to Scripture, and was condemned by Pope Innocent in 405 AD.
THAT, my friends, is the source of the "ever-virgin Mary" idea. THAT is the foundation upon which Catholics have built this doctrine.
That's historical fact. No debate, no dispute. It's incredible, really.
They take the stuff from that document, enlarge the doctrine, and then go back and read Matthew 1:25 where it says that Joseph and Mary didn't have sex until Jesus was born and they say that means that they would never have sex because of Mary's perpetual virginity. And they take the verses that talk about Jesus' siblings and instead of interpreting those normally like regular humans would do, they apply this doctrine and reinterpret those simple passages and invent all kinds of stories to explain them. Wild stuff.