You may disagree, but it is scripturally wrong. God WANTS every Christian to be as an open book and to be a testament for him. Sometimes this means not doing something that is perfectly acceptable so it will not cause anyone to stumble.

1 Corinthians 10:
31So whether you eat or drink or whatever you do, do it all for the glory of God.
32Do not cause anyone to stumble, whether Jews, Greeks or the church of God—
33even as I try to please everybody in every way. For I am not seeking my own good but the good of many, so that they may be saved.

I am not saying he cannot or should not do anything with children. He, actually himself has said twice that children at church are not an issue for him, so I think it should be dropped.

But to say that other people's opinions of the actions of any Christian is irrelevant, is in complete disregard to God's Scripture. This is true of you, me, Joe down the street or Pastor Earl. Everything we say and do reflect on God.