Turns out he doesn't disagree with her rather cutting assessment.
The SNL and Portlandia star sat down with Howard Stern this week and wasted no time in 'fessing up to his relationship failings. After, of course, a bit of prodding from the self-proclaimed king of all media.
"She must f--king hate you," the Sirius XM shock jock told Armisen, when the topic of his ex-wife came up.
"I imagine so," the comedian replied. Not that the conversation had any hope of ending there.
"I've read some quotes from her about you," Stern went on. "She said you're a great actor, but too bad you didn't act like a human being...you must have been a terrible husband."
"I think I was a terrible husband," he agreed. "I think I'm a terrible boyfriend.
"I want it all—fast," he went on, explaining his proclivity to rush into commitments. "I want to be married...the amount of girls I've lived with right away...and then somewhere around a year, two years, I get freaked out.
"I feel bad for everyone I've gone out with," he said.
As it happens, the 46-year-old, who split from Moss in 2010 after just eight months of marriage, has been married twice before: Aside from his vow swap with the Mad Men star, he also walked down the aisle with British singer Sally Timms.
Of his first marriage, which ended in 2004 after six years, Armisen told Stern that was "more for her green card."
The old romantic.