A source said that Anne and Edward had been 'privately concerned' about Andrew and 'wanted to check on him', but Charles was unhappy with Andrew being in the public eye