"I think it's important to hear a whole conversation before casting your own judgment or criticism on something," Bilson exclusively tells EW.