What comes out when he's angry with John is nearly always the stuff that John did right...but Sam doesn't see it that way.

YES! I think that's exactly right, and what so many in fandom miss, that Sam's reactions don't have to mean that John was nasty or abusive or deliberately neglectful, or anything but a man doing the best he could in impossible circumstances--and managing to be a good father along the way. He was broken by Mary's death, and in turn, he broke his sons. But that damage on his part wasn't deliberate or intentional, it was simply because he was human.

It's doubly sad, though, that John, by being a good father, sowed the seeds of Sam's anger and eventual estrangement.