The 'war on terrorism' is proving as frustrating and elusive as a campaign against shadows. Al-Qaeda has melted into the Afghan mountains and the slums of Karachi. For every arrest of a suspected terrorist organiser, a dozen new cells, groups and plots are uncovered. As soon as US military forces direct their fire at extremism in one part of the world, a new atrocity claims lives elsewhere. The bomb in Bali, the suicide bombs in Israel, the attack on an oil tanker in Yemen and the bloody hostage siege in Moscow are evidence enough that Islamist extremism is neither beaten nor sleeping.

But this hydra-headed extremism is beginning to undermine the entire American-led war declared on September 11. For it is clear that al-Qaeda is not a single, all-powerful organisation with cells around the world dedicated to the overthrow of the West – some kind of Islamic Smersh. Islamist hatred of the West is not something invented or controlled by Osama bin Laden; his disappearance and possible death has neither modified the virulence nor halted the flow of recruits to Pakistani, Palestinian or Chechen Islamist groups striking at Western targets.

Militant, xenophobic extremism across the Muslim world is largely home-grown. It is the result not of September 11 but of a profound crisis within Islam. A religion with more than 1.2 billion adherents is struggling with modernity, grappling with political, spiritual and social contradictions. Terrorism, the paroxysms of fanatics and the resort to indiscriminate violence are not part of a war between East and West, the notorious 'clash of civilisations' identified by Samuel Huntingdon. They are the result of a clash within a civilisation: the spiritual counter-revolution within Islam that may be as cataclysmic as the Reformation in Christianity.

The evidence of turmoil and upheaval are everywhere. In most societies from Morocco to Indonesia there is a keen sense of injustice, anger at corruption, a lack of accountability, a suffocating social conformism and a huge generation gap between those glimpsing forbidden knowledge through the internet and those struggling to preserve the old ways and the old certainties. There is a sense that the rulers have failed the people, and new debate about the issue that has vexed Islam since the beginning, the choosing and the legitimacy of rulers.

Most of the Muslim world is now ruled by secular nationalists, who learnt their socialism in the West. This is the guiding philosophy of the generation that threw off colonial rulers and influence – Nasser in Egypt, the FLN in Algeria, the Baath party in Syria and Iraq, Gaddafi in Libya, Sukarno and Suharto in Indonesia. Their regimes, however, have become institutionalised, intolerant of opposition and unwilling to experiment with democracy. Many have moved from being anti-colonialist to pro-Western, bolstering their credibility by aligning themselves with the world's dominant powers in the hope of aid, weapons and political support. As a result, the flourishing corruption, lack of human rights and political stasis are now linked, in popular imagination, with secularism, economic exploitation and pro-Western elites.

Riding this popular anger, militants have seized on religion as the one vehicle that has not been suffocated by the dominant Western global culture. They contrast the golden age of Islam with today's poverty, ignorance and low growth in the Muslim world. They preach a simple, seductive message that a return to basic faith and fundamental principles will lead to a return of honesty, pride and achievement. Like the early Protestants, they excoriate the laxity, corruption and venality of the rulers. And they insist that not only must the Muslim world be cleansed of this godlessness, but it must also rid itself of the pernicious influence responsible for this misery: Western secularism.

It is a crude, revolutionary message – and one that conjures up atavistic memories of the clash between Islam and Christendom, from the early Muslim conquests, through the Crusades to the European colonial wars. The problem is that it is a cul-de-sac – as most educated Muslims know. The modern world cannot be wished away; a Wahhabi-type puritanism offers no guidance for the practice of statecraft in a globalised economy. But moderates have no alternative spiritual guidances with which to confront these Savanarolas. Islam has not had any reformation or renaissance: the last attempt to synthesise religion with modernity was by Muhammad Abduh, an Egyptian philosopher at the end of the nineteenth century. Few remember him now – and certainly not the ignorant and often obscurantist sheikhs and imams who now control religious practice and social mores.

Those committed to technological progress, therefore, are attempting to make their societies more just, more efficient and more productive in the teeth of religious opposition.

The Kemalists, Ataturk's successors in Turkey, still regard conservative Islam as the enemy (though the strong showing of the Islamic party in the recent general election shows that even secular Turkey cannot escape the wave of Islamist influence). They are not helped by Western reaction which is increasingly demonising not only the fundamentalists but the religion they preach. Every time that Billy Graham's son calls Islam an evil religion or Silvio Berlusconi speaks of the West's superior civilisation, the moderates are weakened and the militants gleefully insist that the enemy is really the West. The war on terror is thus easily portrayed as a war on Islam. America's attempt to remove Saddam Hussein is denounced as a new attempt to get control of Islamic land. The clash of civilisations becomes a self-fulfilling prophecy. But the greater clash – for the heart and renewal of Islam – is still to come.