When I was 18, a sort of ultra-leftist, and in search of intellectual heroes, I thought Perry Anderson was probably the cleverest man in the world. It was not a completely stupid judgment. I no longer believe in making such lists; but if I did, Anderson would surely still rank very high indeed. Spectrum, his latest book after a relatively long silence, consists of 13 essays on contemporary or recent thinkers, plus two appended pieces on The London Review of Books and on Anderson's father. Most have previously been published in the LRB itself or in Anderson's own New Left Review. Their remarkable range, theoretical sophistication, sharpness of judgment, and sheer intellectual power indicate that Anderson's verve and skill are quite undimmed by time. Anderson writes, if anything, better than ever. But in most other ways, the contrast between his early work and Spectrum suggests how almost everything has changed between then and now: the political context, the intellectual atmosphere, the purpose and spirit behind the writing.

It is widely accepted that if the Marxist left is not entirely dead, it now exists only in a few university departments on one hand, and as a crude anti-war, anti-imperialism, anti-globalisation tendency on the other. The former is politically irrelevant, the latter intellectually moribund. In the second, activist register, it might well be that the absurd George Galloway must be seen as Britain's most important Marxist. If so, it's a thoroughly depressing conclusion; which would undoubtedly have horrified old Karl himself. He might well have viewed US conquest of Iraq and Afghanistan as he did British rule in India – as historically progressive and thus to be welcomed, for all its brutalities and blunders.

In Considerations on Western Marxism (1976) Anderson had already explored an intriguing disjuncture between the lack of mass popular appeal for socialist ideas, especially of a Marxisant kind, in western Europe after 1945 and the unprecedented proliferation in the very same period of leftist and Marxist theory. Subsequently the phenomenon has become more dramatic still; perhaps especially in the USA. Amidst the near-total absence of a significant socialist left on a popular level – an absence manifest for many decades in North America, newly evident in Europe after the rapid collapse of parties owing any kind of allegiance to a Marxist tradition – there has been an ever-proliferating growth in the production of proclaimedly radical academic theories. What Anderson seemingly did not, perhaps could not, have anticipated in 1976 was the extent to which his own later career would come to fit the pattern he had identified and deplored.

The Perry Anderson of the 1960s was effective leader of a group of brilliant young socialist thinkers, grouped around the New Left Review, who saw themselves as conducting a comprehensive demolition job on every aspect of British intellectual life. Their project of creative destruction had as its spearhead a series of bracingly ferocious essays by Anderson himself and his close comrade Tom Nairn. They saw British society as essentially brain-dead – for deep-rooted historical reasons. Britain's 'bourgeois revolution' had remained always incomplete, so that control over the British state stayed in the hands of a largely preindustrial, patrician elite. Britain lacked equally a properly modernising industrial bourgeoisie, a suitably class-conscious proletariat, and an intelligentsia with the capacity for bold or even serious social analysis. The country, including not least its Labour movement, was mired in political deference and hierarchy, constitutional sclerosis, economic inefficiency, cultural insularity and archaism. With truly heroic – or in critics' eyes, monstrously arrogant – ambition, Anderson and his colleagues set out to cure these ills, partly by their own writing, partly by translating and publicising a great body of continental Marxist theory.

For some years, that project was pursued with astonishing energy and consistency. Alongside the destruction of Britain's ramshackle old intellectual monuments, Anderson began to build the new structure which might replace it. The foundations for this, it seemed, were being laid in his two remarkable 1974 volumes of comparative history, Passages From Antiquity to Feudalism and Lineages of the Absolutist State. Their bold Marxist reinterpretation of European development from the Romans to the 17th century would – it seemed obvious – be followed by sequels bringing the story up to the present, to show the deep historical logic behind today's necessary socialist transformation.

Those sequels have never appeared, and this was intimately, though not simply, related to loss of faith in the coming transformation. Anderson and Co were far too sophisticated, and usually too hard-boiled, to be mere intellectual cheerleaders for any would-be socialist revolutionaries, in Britain or beyond. They were certainly never apologists for the 'actually existing socialism' of the Soviet Union. But the successive failures of the late-'60s student revolution from which many NLR cohorts had been drawn, of the 'Bennite' British Labour left, of Eurocommunism and of the Soviet model itself, knocked away many of the underpinnings of their wider project. The tide of history (to invoke the kind of cliché Anderson's own writing resolutely avoids) had turned against them.

Perry Anderson's own response was uncompromisingly tough-minded and realistic, seeking to draw up a clear balance-sheet of where the left now stood and how it might move in future – the bottom line being, it seemed, 'nowhere much for a while yet'. But where did this leave his own work? The answer was that the grand system-building project was abandoned, or at least put in long-term suspension. Anderson returned to the tasks of critique, in a long series of essays, many on individual thinkers from across the political map, some on broader intellectual trends. Many were sharply negative, but he also offered broadly sympathetic (albeit never uncritical) appraisals of theorists whose work he valued. Having suspended his own grand system-building ambitions, he presented particularly acute evaluations of other great synthesisers and holistic social theorists: people like Michael Mann, Roberto Unger, and in Spectrum, Jürgen Habermas, Robert Brenner, and Göran Therborn.

The Anderson of the '60s was undertaking a heroic demolition job. In the '70s he was constructing a vast new edifice to put in the place of that which he'd demolished. Today there is very little trace of that larger architecture. Instead we have fragments – brilliantly polished and superbly shaped, but no longer parts of a greater whole.

All that sounds like a depressing story, a narrative of intellectual retreat and failure. But it is only half the picture. If Anderson's writing has, near inevitably, lost its former revolutionary élan, it has gained in depth, richness and nuance. The sheer scope is startling and enviable – though it rarely seems to extend to women writers, or to non-European ones.

As Cambridge historian Stefan Collini has said: "It hardly seems fair that one man could move with such ease through the history of so many periods and regions or through so many different kinds of writing." There is also more humility and humanity in Anderson's style than there once seemed to be. The essays here on his father, and on the great historian EP Thompson (with whom Anderson clashed bitterly in the past, but to whom in Spectrum he pays gracious tribute), have a personal warmth which his early work largely lacked. Above all, the quality of analysis remains breathtaking, exhibited in Anderson's capacity to grasp and unravel both strengths and weaknesses in immensely complex ideas. If Anderson's move, from being a confident herald of the thought of the future to a brilliant but sombre dissector of the thought of the past, may seem like a kind of descent, then what remains is still not negligible. It's a very great deal.