Do you remember that a angel spoke to Joseph?My opinion...and this is just my opinion...I think Jesus Christ in His family life was miserable. I believe Joseph wouldn't have shown much affection to Him for obvious reasons. He would have done his obvious duty but still, Jesus was not his and he knew it. And I believe Jesus other brothers knew it due to Josephs lack of affection. I believe Jesus endured much suffering in His early years long before He entered His public ministry.
Stranger
Mathew 1:20
Do you remember what Joseph did to protect his son?
Joseph was, after all, His earthly father and he had experienced such supernatural happenings that Joseph must have known that Jesus was a special boy.
Mathew 2:13-23
I think Joseph was probably a very good father and this is why God chose this particular family for Jesus. He seemed very well adjusted as an adult.
Luke 2:52
Luke 2:46-47