Shashi Tharoor said he would have refused to pull out of the Congress chief elections had he been asked to do so.