The 29-year-old suffered a calf injury while on international duty after recovering from a thigh problem and is now out for up to a month.
It has left Villa with just two fit centre-backs in Ciaran Clark and Jores Okore - who has not played this season after recovering from a cruciate knee ligament injury - for Monday's visit of Southampton.
Lambert said: "That's the one I'm not happy about at all, Philippe
He is the one which has annoyed me the most
He went away with his national team and if you look at his record he has only played six minutes, against West Ham, for a month
He goes away with Switzerland and ends up pulling his calf in a training exercise.
"It's mismanagement from the FA in Switzerland
They should have been in dialogue and seen his record in the first place and that he didn't play for a month.
"It's OK for them to go away and think 'we have only lost a centre half' but we have lost a centre half for a month again, who has been a vital part of the team."
Nathan Baker has a knee injury while Ron Vlaar also suffered a calf problem on duty with the Netherlands.
Both are out for Monday's game and Lambert insisted he should not have been put this position.
"We still have options but in Philippe's case it should have been avoided
He was totally fine when he came on in that little cameo role against West Ham," he said.
"Jores is just coming back from an injury but he's getting better
We have lost two experienced players who have been playing well."
Source : PA
Source: PA