The problem with your point of view is that it ignores the context in which that statement was made and erroneously assumes that committing adultery in your heart is the same thing as committing adultery with your body.

When Jesus spoke about committing adultery in your heart, he was delivering the famous sermon on the mount. In case you are not familiar with this sermon, it was a sermon that he taught to his disciples. This sermon is important because for the first time, he isn't teaching the basics of Christianity to the masses, but is teaching his disciples the more advanced concepts of living a Christian life.

In this section of the sermon on the mount, Jesus is challenging his disciples to take their relationship with God to the next level. Whereas a new believer might have a problem with sins that are actions (such as physically committing adultery or murder), as we grow in our faith and grow closer to God, the sins we struggle with are usually no longer actions, but thoughts. What God is saying in these passages is, it's time to take it to the next level. I know you don't murder and I know you don't commit adultery, but now that you have mastered your body, it's time to work on your mind. That's what he's saying when he begins his talk on adultery of the mind in Matt 5:27.

If you want further proof that adultery with your mind and adultery with your body are not the same thing, take a look at what he says about murder right before that. Jesus chose to illustrate how sins can be committed with your mind by using two examples, murder and adultery. Of murder he said:

Matt 5:21-22 "You have heard that it was said to the people long ago, 'Do not murder, and anyone who murders will be subject to judgment.' But I tell you that anyone who is angry with his brother will be subject to judgment. Again, anyone who says to his brother, 'Raca,' is answerable to the Sanhedrin. But anyone who says, 'You fool!' will be in danger of the fire of hell.

In that verse, which is actually what sets us up for the discussion on adultery, God clearly tells us that when you commit murder in your mind, you are not answerable to the laws of man, only to the laws of God.

Adultery is the same way. If you commit adultery with your body, you are answerable to your spouse and could find yourself divorced with God's approval. But if you commit adultery only with your heart, God would not approve of that divorce, but you are putting yourself in danger of the fires of hell.

So, to answer the original question, no. Looking at pornography is not legitimate grounds for divorce biblically speaking. Of course, if you want to talk about a secular humanist point of view, which sadly, many churches have adopted, then it is certainly grounds for divorce. But speaking only with the bible as a source, it is not grounds for divorce because there has been no real unfaithfulness