Fauci argued that the unvaccinated are affecting others because they're “allowing the propagation and the spread of the outbreak,” and pushed back against critics