While waiting for visitation and custody rights, is this okay to do with my ex?

Of course it's ok for him to go to your house and see the baby and stay with the child , Your making to much of it and making it complicated for you and your ex.