You need to establish clear boundaries from the get go. The problem is that you didn't do that and now have to establish them after he is accustomed to having things his way.

You need to be firm. Next time he does something like that, you tell him that you don't like that and it's time for him to go home. He can return when he's ready to show you some respect and you're gonna give him a full day and night to think about it. Then, stick to your guns. He needs to go home and can't come visit at least till tommorrow. If he does it again, then he obviously didn't learn his lesson, maybe he needs two days.

There is no reason to get physical. If it gets to the point where he will not respect you unless you get physical with him, then it's time to tell him he's not welcome at your house ever again.