Your baby brother dies while you are still a child. You've never really lived life, just on autopilot.
Do you think your agnosticism is, or could possibly be, connected with this event in your life?
A child is traumatized and asks, consciously or subconsciously, “Since God exists, why did he allow my innocent baby brother to die?”
You asked me not long ago what I would feel if I were in your shoes. If I were in your shoes when you were a child, I would have been sad, confused, angry, afraid, even guiltridden. If I couldn’t resolve those feelings, I would probably grow up feeling helpless, hopeless, explosively angry and at times dead inside.
My question would have changed, at over time, from “Since God exists ...” to “If God exists …” and, if I couldn’t arrive at a satisfying (peace inducing) answer to the modified question, it would eventually either destroy me or cause me to modify one more time - “Since God doesn’t exist, my innocent little brother is lost forever.” (I don’t think you’ve reached the last modification of thought yet.)
None of this may be what you actually feel or felt. I’m projecting myself on your life.
I’m not a psychiatrist or psychologist. I may be way off base, but I think you probably have to go back to this very traumatic and painful time in your life - when you had a childlike belief that God existed, assuming that you did - and find a satisfying answer to the questions, “Since God exists, why did he allow my innocent little brother to die? and “Is he lost forever or will I see him and reunited with him some day?”