Reo-Coker, who will skipper Villa during Stiliyan Petrov's two-month absence through injury, is out of contract next summer but Houllier revealed chief executive Paul Faulkner has already held talks with his agent.
It is a remarkable turnaround in fortunes for the former England Under-21 captain, who had few opportunities under previous manager Martin O'Neill after a training ground bust-up.
He will lead the side for the first time since Petrov suffered his knee ligament injury in the Carling Cup fourth-round clash with Burnley at Villa Park.
Houllier said: "I don't want to talk about the past. I don't want to bother with that. Just from what I see, Nigel's attitude and his work-rate has been consistent.
"In terms of performance, he knows exactly his role in the team. He works for the team and he has improved a lot, even over the last few games.
"Exemplarity is also a very important factor to me and that's why he's the captain. I hope he will stay with us."
Asked if there will be contract negotiations in the near future, Houllier added: "It's too early to say but Paul Faulkner has started to have talks with his agent. He knows I want to keep him."
Houllier admits he is wary in general about giving lucrative contracts to players because of the negative effect it can have on their performances.
But he does not believe Reo-Coker falls into that category.
He said: "You have to be careful when you give good contracts to players. You don't want a player, once he has a contract, to get back into a sort of comfort zone, put a bit of weight on sometimes and not work as much.
"You have to make sure the contract is not the end in itself. It must the start of a fighting experience to win things for the club.
"If you give a contract to a player it means you trust him for the future. I just want fighters.
"I want players who are very hungry for success and not players who just think 'feet up and relax'. That's all.
"I think Nigel has probably gone through good times and bad times until now.
"But he knows the years to come are vital for him and it's better to be in an environment he knows, he likes and in which he will enjoy his football and therefore improve."
Houllier has plenty of respect for Burnley as he stressed the importance of the Carling Cup.
He said: "We have done our homework. We know about them. They are a good football team who play good football.
"It's a good, important game for them and for us. I hope they have a go but they must expect that we have a go at them as well. It will be an interesting tie.
"The players are aware a win will bring them to the quarter-finals.
"Every cup is important because it is important to bring silverware to the club."
In addition to Petrov, striker Gabriel Agbonlahor (groin) and defender Luke Young (hamstring) are sidelined.
Source: Team Talk