The housing market has been overly sensitive to what the Fed has done with interest rates, both when they've been high or low.