I disagree with you on this issue wholeheartedly.

A person needs to do what they are called to do. The only person whose opinion matters is God's. If God has called you to minister to children, then you need to minister to children no matter what some people might think if they discovered you were a pedophile. This isn't about you or the parents or even the kids, it's about doing what God created you to do.