Self-checkout at some retailers has resulted in long lines, inventory loss and frustrated shoppers, prompting some companies like Target to test new lanes.