"To give you a direct statement right there is me playing a game I'm not interested in playing," the actor told cohost Joy Behar.