Those on the lowest incomes are being hit hardest by tax penalties, a non-profit says.