Here is a true story. I was discussing the wonderful life I had as a child with a friend (who happens to be a person of color). I grew up primarily in the 50s. I could ride my bike to a friend's house, go to the store and get a soda, play touch football in the nearby park, flirt with girls, etc. My comment was "life was great back then". His reply was "that's only because you were white". I never forgot that lesson.
That's an example of a terrible friend right there, I think you should remind him/her to repent of their unnecessary racism against you, which it clearly was his/her intent to make you feel bad for being white. Which made you feel guilty as you said you never forgot that lesson.
What lesson? Why should you feel guilty of growing up having a good childhood? Why! Why do you settle for friendships like that?
I would have explained that life was good back then because there were very little people of colour around to spoil it, which is true isn't it, that now streets are over run with undesirables making our own children's lives a misery to be brought up in. But I can understand why people wouldn't have stated this as I would have, for the fear of offending the feelings and being labeled a racist. The truth is always truth no matter who it upsets.