The other day, I stood staring at my rows of books and CDs and thought, “One day, someone is going to come into this room, put all this stuff and everything else I owned into boxes and take it to a charity shop.” Welcome to my world of joy.

Although I hadn't had this particular thought before, it's typical of my almost constant consciousness of the inevitability and unpredictability of death. Does this make me a pessimist? Not one bit. I'm writing this on a bright summer's morning, and I'm looking forward to the delights of company, croissants and coffee that await – provided that is I don't keel over and die first.

The truth is, I don't really get this idea that we have to choose between optimism and pessimism. It seems to me there are two questions we need to ask, and realism is the answer to both.

The first is, what is our assessment of the likely trajectory of the future? In The Rational Optimist, Matt Ridley argues that bon pensant pessimism is unjustified. We have more reason to think life is going to continue to improve for most people than we do that we are all going to hell in an environmentally-unsound supermarket trolley. Yes, there's climate change, but plagues, diseases, malnutrition and murderous wars haven't stopped the ascent of humanity, so why should a few extra inches on the mercury put an end to progress? Ridley convincingly argues that although such breezy optimism is easily dismissed as shallow, harbingers of doom have been wrong more often and more grossly than heralds of sunshine.

Ridley overstates his case somewhat, but he doesn't fall neatly on one side of a divide between optimists and pessimists anyway. Roger Scruton, for instance, has recently written a book in praise of pessimism, but actually he's largely in agreement with the optimistic Ridley. (Scruton outlined part of his argument in the May/June 2010 issue of New Humanist.) What both vigorously oppose is the deluded, utopian optimism that makes people think that rational, and especially central, planning we can make the world a much better place. Scruton trusts tradition more, while Ridley puts his faith in trade and innovation, but both are united in advocating a realism that cautions against the hubris of trying to engineer a perfect world.

Given that much of the future is uncertain, there is some scope to lean towards the more optimistic or pessimistic ranges of the possibilities. But for anyone who values rationality, this leeway is rather limited. The main thing is to face the future honestly. When it comes to the future of our planet, dull realism beats blind hope and fatalistic despair every time. Yes, there is always a chance that catastrophe is impending, but not even the worst case scenarios of the International Panel on Climate Change, for instance, spell The End. At the same time, it is just as daft to think that everything's going to be rosy no matter what we do, so burn fossil fuels, baby, burn.

The truth, as usual, is actually quite boring: it is likely that we will see humanity carrying on in its own sweet way, punctuated by crises and disasters of varying scale. But I don't think I'll be getting a huge advance for my proposed book Trundling On: Why Humanity is Going to Have Big Problems But Still Get By (Probably). The writers who get hailed as the great prophets are always those that promise something more extreme.

The real problem is, of course, that even if Ridley is right, for each of us individually, the future is not so bright. In fact, it's pitch black. We are hardly out of adolescence before we begin to notice the ravages of age which will at first slow and then drag us down, until something finishes us off for good. Average life expectancy may be over 80, but that means half of us won't make it that far. A cardiac arrest, a clumsy fall or a stupid bugger in a car – of which there are millions – is all that stands between us and checking out early.

So, yes, there is plenty to look forward to, but we're just not going to get to most of it. And the more of it we are around to see, the fewer of the ones we love are left to enjoy it with us. At some point, probably more than once, your heart will be broken or you're going to break someone else's. Not just broken but beaten, stamped on and dragged round the streets behind galloping horses.

This isn't pessimism, it's realism. It will happen, for sure. Which leads to the second question: given a realistic assessment of what is to come, how should you feel about it? Take the prospect of the detritus of my life being carted away on the tails of my coffin. How should I feel about that? Not great, I wouldn't have thought. Since brooding over it isn't going to stop it happening and will only spoil the interim, there are good reasons for not letting it ruin that coffee and croissant I'm looking forward to. But at the same time, it would be odd to pretend that there's nothing bad about the prospect at all. The age-old argument that we won't be around to feel any loss, so we shouldn't worry about it seems to me to miss the whole point. I'm not looking forward to not being around because I'd rather be looking forward to being around.

So the most optimistic I can be is this. In the long term, we're all screwed. Fact. For any one of us, for all we know, we're screwed in the short term. Fact. All the more reason, then, to not so much look forward for the good things to come, but to look around at the good things that are already here. We can't live entirely in the present, so we have to allow ourselves some kind of future horizon. If we anticipate life in the next few months at least, it'll come most of the time, and will fail to do so just the once.

My optimism is rooted in the belief that this is enough. The day ahead is not just one that I can get through, it's one that will have been worth living. That's not something I can ever seriously doubt when I'm sat writing, watching a good film, or having that coffee and croissant across the table from the person I share my life with. If that is less than perfect, it is surely more than good enough.

The Rational Optimist: How Prosperity Evolves by Matt Ridley is published by Fourth Estate. The Uses of Pessimism and the Danger of False Hope by Roger Scruton is published by Atlantic.