Retail theft is a major problem in California and the rest of the nation, merchants and law enforcement officials said. The state is trying something new.