"Oh, because you did? I'm sorry, Nancy, but you weren't exactly the greatest moment of my sex life." He's not trying to be harsh, but he's trying to make a point.

The two of them had slept together a lot in the past for various reasons: it was convenient, they understood what the other wanted and they didn't need to waste time. Nancy was stable and consistent. But that doesn't necessarily mean that it's his favourite. Sometimes the best moments are when you don't expect them. Nancy was reliable but Lydia is exciting.