‘I felt very privileged but I was like, “Oh I just want to sit back here and not get in the way,”’ actor recalled