I feel for you, brother.
Ask the Holy Spirit to put his peace and love and wisdom over the situation.
Over yourself and the others.
Don't hold grudges, try to forgive, although its ok to feel angry.
Do not let fear get a foothold.
It might help to face the pastor not so much as to change his mind as to say to him that you would never do anything to hurt a child and that it was unfair to do what he did based on rumours. It doesn't matter if nothing changes, it might help for your own sense of self-respect etc.
its a hard one though.
how did your father start these rumours?

Bertil.