I agree with Eldad 100%. You need to forgive your brother. This is not for him really. It will only help him if you ask for forgiveness. However, we need to forgive for our own healing and growth. If you forgive him, you will be free, if not, the spiritual consequences are worse.

That being said, just because you forgive doesn't mean you have to trust him again. I would tell him how you felt about that comment that he made and how deeply it affected you. After that, if he wants your trust back, he'll have to earn it. If he doesn't care, then yeah, disown him (in some small sense), but don't hate him.

Hikaru