Surely he must have thought of it, but he says that when he realised what he felt, a year later, it was too late by a long time. It's a line I've puzzled over and come up with various possible interpretations of what he could have meant, but I'd be interested to know if you have a new one! He can't just mean that he was married, I think, because (a) he doesn't seem to see the marriage as in competition to Jack - he doesn't think of Jack as an extra-marital affair and (b) he's still married when they meet again, but it's no barrier to them getting together. What do you think he means?