What happened to jazz? Jazz, as saxophone colossus Sonny Rollins said, is "without a doubt intellectual music. You have to cultivate a liking for jazz." Jazz is intellectual music for intellectuals. The problem is, at the moment, it doesn't seem to be delivering. So what is the intellectual to do?

Israeli saxophonist Gilad Atzmon believes we have reason to be worried. In an article in the Guardian, Atzmon began by sketching out the credentials of mid-twentieth century American jazz: " When Martin Luther King started his campaign the jazz community stood right behind him… [jazz] exposed the fundamental injustice within the capitalistic system."Wow, I can see why the intellectual liked it then, it was a kind of swinging socialism. So where did it all go wrong? For Atzmon, what happened was that after the pure golden age – roughly from the zenith of bebop in the late 1940s, to Coltrane's heart-felt and excruciating explorations of the outer reaches of the musical and emotional register in the late sixties – jazz was killed off by a combination of the appropriation of jazz by the cold war era American propaganda machine, the institutionalisation of jazz in the academy, and the fetishisation of technique over social and political content.

Through this process, for Atzmon, jazz loses its subversive edge by being stripped of its racial being, its embededness in Afro-American culture and experience: "jazz became a form of meaningless white noise." But that's not the end of jazz's problems, not by a long shot. A worse fate was waiting at the dawn of the 21st century. Atzmon warns "we are now at the apex of [jazz]'s commercial phase". Where once ignored by record companies and the mass markets, jazz finds itself newly commercial and respectable through the emergence of the easy listening jazzstrills – Jamie Callum, Katie Melua, Norah Jones. With no discernable politics, berets or flattened sevenths, this new jazz-lite cadre represents for Atzmon the ultimate betrayal of jazz's anti-corporate ethic.

There is much to agree with in Atzmon's thesis; but there is something that grates about his story, something which, it seems to me, underpins the whole nostalgic intellectual hand-wringing about jazz and its lost power.

It has to do with definitions and history. Jazz began as the music of the streets, of the poor and disempowered, of the outsider. So of course it was political. How could it not be? As our sage Sonny Rollins also said: "Just by being black in America I'm political." But that doesn't mean that jazz was always about the best way to organise community collectives, or how to ensure steel-workers obtained the right to health benefits (if it was ever about that). It was, primarily, good time party music. Which of course was itself deeply political – slaves and their descendants were disallowed social pleasure along with all the other benefits of full citizenship, and jazz laid claim to public space in pursuit of black pleasure – deeply political, though not perhaps the kind of politics that serves the agenda of the cosmopolitan intellectual.

The argument is that one way to think of jazz is that it is a synonym for 'black popular music'. This is not to suggest that only black people can make, or dig, jazz, but it does suggest a deep and abiding connection both with Africa (jazz's rhythmic thread) and with the continuing social reality of America (and then the world's) disempowered, outsider populations. And if jazz is thought of as black popular music then a counter-history to Atzmon's runs something like this:

Before bebop the biggest genre in jazz was swing. Swing originated in the juke joints and rent parties of Kansas City, Chicago and Harlem. Band leaders like Jay McShann presided over what were in effect popular orchestras, whose job was to make people dance and keep them dancing.

So okay, jazz has a history before Atzmon's point zero of bebop. He can't, after all, say everything. But it is the suggestion that bebop somehow spoke for black America, with black America's agreement, which rings false. It would be wrong to characterise bebop, uncritically, as an expression of any particular racial or social group, above and beyond the mesmerising, super-cool artists at its core. Bebop was a stunning, unique and minority statement, by artists frustrated by the lack of credibility given to their chosen medium, and the sterile emphasis on standards on the nightclub circuit. They sped jazz up and switched it around. Horn players played rhythmically, bass and drums melodically, everyone skipping across scales and up and down their instruments at a hair-raising pace. As Amiri Baraka pointed out in Blues People, Dizzy Gillespie, Charlie 'Bird' Parker and Miles Davis self-consciously employed the style and mannerisms of the European avant garde as a way to insist that their art was art to have it recognised and respected. Again deeply political against the backdrop of American racism. But the unfortunate corollary of this was the estrangement of jazz from the audience for black popular music. In winning its respectability as art, bebop prepared the way for jazz to be captured by the elites.

There followed the 'golden age', as far as the cosmo-political jazz aficionado was concerned from 1940s Bird to late '60s Coltrane, in fact all the way up to Miles Davis's Bitches Brew, recorded in 1969.

Even as jazz trainspotters were picking over the bones of Bird's solos, long after he himself was dead, jazz was moving outwards, getting freer, and downwards getting funkier, sometimes separately and sometimes together. Coltrane, always one to bend and twist the tunes even when playing relatively sweetly, began a process of creative deconstruction which would have made Derrida blanch (or dance?) entering the great period of Alabama, Africa, and A Love Supreme. Meanwhile hard bop, and funk – the churchy, rootsy sound of Horace Silver and Lou Donaldson, especially on the Blue Note and Prestige labels, re-anchored jazz to the roots of popular black dance music, and its twin shrines, the church and the juke joint. And then all hell broke loose with the unleashing of soul. Inspired by jazz and often jazz players themselves (Maceo Parker, Brown's ever faithful saxophonist has all the chops and all the skill) soul was a parallel stream which refreshed and converged with jazz throughout the next decades.

As rock started to, well, rock, jazz incorporated elements from it too. For the European intellectual hung up on Coltrane's 'voice crying in the wilderness' sound, the perfect emblem of black suffering and transcendence (which of course it was), what happened next looked and sounded like a sell-out – the ingress of the electric bass guitar, the use of four square rock rhythms and breakbeats.

Fundamentally black popular music hasn't offered the (let's say it, white) intellectual what they crave since the sixties. Jazz, from the perspective which is hostile to anything that smacks of 'commercialism', looks dead by the dawn of the 1970s. Yet this is the era of the great 'electric' albums by Miles Davis and Donald Byrd, Roy Ayers and Gary Bartz – where jazz as art met jazz as dance music, and won a substantial (and substantially multi-racial) audience. Many of the great records of this period were, it is true, released on 'major' commercial record labels – RCA, Columbia etc. Herbie Hancock's Headhunters, with Bennie Maupin's beautiful muscular sax over grumbling synth basslines and funk breakbeats, became the biggest selling 'jazz' album of all-time, and signalled ahead to hip hop and electronic music of all kinds. But I don't think the intellectuals liked where the sign-posts were pointing.

Rather than being a story of original purity, with a pre-lapsarian golden age, and a latter day descent into over-intellectualism and then commercialism, jazz was from the start and remains caught in the interstices between art and commerce, elite and noisy crowd. Jazz is not politically or socially profound in a vacuum, but against the backdrop of the times. For most of the 20th century this backdrop was woven with the threads of American racism. Jazz certainly was a vehicle and a way to voice protest, resistance and assertive forms of black identity, but of course it wasn't the exclusive property of any one individual or group. These days, things are perhaps not quite as extreme in terms of racial politics, but there are other problems on the horizon. What has jazz to say about them? Where are the jazz musicians prepared to embrace this 'double consciousness' of jazz, to revel in its indeterminacy, yet commit to its particular rigour?

Step forward Soweto Kinch (right), award winning alto sax player, and one of UK jazz's new young hopes. Still in his early twenties, Kinch took up jazz after graduating in modern history from Oxford, and records for the North London based Dune label.

I spoke to Soweto to share my puzzlement about jazz, politics and intellectuals. What he said suggested to me that he might be just the right kind of intellectual to spearhead jazz into the next century. Why? Because he seemed to recognise in his playing and how he talks about it, the essential doubleness of the genre – its need to commit equally to (in his words), "the intellectual and the guttural the esoteric and the soulful". Our conversation went from Sibelius to Public Enemy, via Marx and A Tribe Called Quest. Reflecting on it, Kinch said this was what he was after in his music too, to combine the sweet and the lowdown, to multi-skill, to be at home in the language of the mind and the body.

Underpinning his canny analysis of jazz as a self-evidently minority, intellectual endeavour, is a strong sense of the calling of jazz, the commitment; even, in a phrase I've not heard from a musician for a while, the duty of a jazz artist. What is this duty? To offer an alternative to consumerism and empty materialism (symbolised in our conversation by bling bling hip hop). So jazz is political? Yes, it can mobilise. But how? By suggesting another way of life. Jazz, for Kinch, can be distinguished ultimately from other kinds of popular music, black or otherwise, because it is in the end about collective improvisation. It carries a politics of unity and cohesion predicated on discipline (practice) and flexibility (improvisation). It is a politics of listening, responding, engaging, trading; of being tuned in with others. "In this day and age, choosing jazz is making a stand," says Kinch.

When it comes to the racial politics of jazz, Kinch grabs the opportunity to distance himself (and his labelmates at Dune) from any kind of simple black nationalism. "We want to open up what has been presumed to be exclusively black - black style, black vernacular - whatever. We are against that kind of reductionism: jazz is about all encompassing human values which are for everyone." His point is that it has often been black people who have had the courage to stand up and express these values, and the urgency to do so, but the collective values themselves belong to no specific group.

In the end the arguments made by Gilad Atzmon and Soweto Kinch share a lot of ground. They both emphasise jazz's anti-materialism and the collective transformative potentials of jazz as a collaborative art form. But in connecting the story of jazz to the larger story of black popular music and expression, by refusing the high/low culture split, and the mind/body dualism, Kinch may be better equipped to get the ear and respect of the young and drifting.