"My daughter actually thought I'd rented the place for just us," the actor said. "Because within 10 minutes, everybody was gone."