Olympic bronze medallist Bajrang Punia on Thursday indicated he might withdraw from the Asian Games if all the panchayats that backed him during the protest at Jantar Mantar ask him to do so.