Siegel said he was one of the first to warn of the Fed's "inflationary policies" in 2020 and 2021, but "the pendulum has swung too far in the other direction."