Particularly his ex-wife Brooke Mueller.
A source close to Mueller tells E! News that Charlie and Brooke haven't come to any custody agreements because she doesn't want their sons Max and Bob around his porn star girlfriend Bree Olson and Natalie Kenly...
"Brooke absolutely does not want Charlie's girlfriends present when the children are at his house and wants someone there to monitor the kids' visits at all times," the insider tells us, adding that she's refused all of Charlie's offers thus far.
But negogiations are still on-going between their respective lawyers and any reports that the contentious couple is heading to court is "premature," according to our source.
"The lawyers are continuing discussions today," the source says, "We'll see what happens."
The twin boys, who are almost 2 years old, were taken from Charlie's Sober Lodge last week after Brooke filed a restraining order against him.
Neither Sheen nor Mueller has returned calls seeking comment on the custody battle.