Gardeners' World presenter Monty Don has hit out at the 'gardening mafia' who watch his show only to prove themselves right.