"It was the hardest period for us because it felt like the world was against us. Here's the thing — we were against each other if I'm completely honest."