There was even one point when Khloé banned Scott from visiting the home that she once shared with Lamar Odom!
In this deleted scene from last Sunday's episode, Scott is out to lunch with Kourtney Kardashian, where the two are discussing the big plans she has to build Scott an Andre Agassi-worthy tennis court.
But when the topic of conversation switches over to Khloé moving in with Rob Kardashian, Scott's tone immediately changes.
"It's so weird that Khloé's leaving her house," Scott says. "It's even weirder that there was a time I wasn't even allowed in that house."
Watch the clip above to find out why Khloé banned him from entering her home!
Tune-in to the brand new Keeping Up With the Kardashians this Sunday at 9/8c on E!