“He’s not speaking to me now. If you ain’t with it, you ain’t down. And I ain’t with it. I’m not budging on that. I’m not with it,” Pusha T said in a recent interview