Damon's talking about the scene at the end of Rose (2x08), Where he confessed he loved her and then compelled her to forget it. If I recall correctly he said, "It's because I 愛 あなた that I can't be selfish with you." So in 4x01 "I am that selfish. the first time we met's not all that あなた remember." is a remark to that memory. Hope it helped.
Damon speaks about his first 愛 confession. But he at the same time indicates that he is not selfish, especially when it comes to Elena ("Because I am that selfish, but あなた knew that already, the first night we met is not the only thing あなた remember."). Check out the 記事 "The Truth About Love: Why Elena's Logic in 4x01 Doesn't Make Sense", where it is explained why Damon's choice to save Elena is not selfish, because he is willing to くま, クマ her hating him as long as she is 安全, 安全です and alive.
Damon is referring to the scene in 2X08 ("Rose"), in which Damon tells Elena that he loves her and then compels her to forget. It is totally and 100% heart-breaking. The first episode of season 4 depicts Elena remembering that specific memory.