It's not in your head, and it's not a character flaw. As men age, two things quietly change.
First, circulation. Blood flow is the engine behind everything and like every other part of your body, that system gets less efficient with age, stress, and years of sitting at a desk.
Second, and this is the part nobody warns you about: the pressure loop. One disappointing moment creates anticipation before the next one. Anticipation creates tension. Tension makes everything harder. Within a few months, the anxiety is doing more damage than the underlying change ever did.
Pills attack the first problem for a few hours. They do nothing about the second one — if anything, the planning, timing, and waiting make the pressure loop worse.