A man who spends most of his life away from his family looking after a random town and spending an excessive amount of time with an attractive young woman* is a good a decent husband and father just because he doesn’t shag her.
A lot of people completely misunderstand infidelity. They think that just because someone spends a lot of time with someone else that he is neglecting his family. Of course not. The only thing that is important about infidelity is whether on not it is consumated. And Ed Bloom here, bar a quick snog, merely spends years doing up the house and spending time with the younger lady in question when his kid is growing up and his wife is lonely at home. Nothing wrong with that.
*Considering that this character is played by Helena Bonham-Carter, you may choose to select this statement as the lie instead.