Tom Schwartz seemed to be one of the only folks sticking by Tom Sandoval's side in the wake of the affair but now, even he's taking a step away ... saying he hasn't seen or spoken to his pal in a while.