That really depends on how risk averse you are, what your payment is, and how stable your job is. For example, my payment is tiny because I got a great rate, bought below my means, and have owned it for several years (so inflation is doing its thing). At this point, I spend more on food than I do on my house.

To mitigate risk, I keep a sizable emergency fund (sustain lifestyle for 6 months), which is currently invested in very safe bonds and money market funds returning a higher rate than my mortgage rate. Why would I pay down my mortgage when I can get more essentially risk free in bonds?

I really like Dave's question: if you had a paid off house, would you get a mortgage on it? My answer is, hell yeah if I could get the same rate I have now! It's free money!

If my rate was >5%, I'd pay it down aggressively. But it's way below that, so I'm holding it until I have enough to retire.