There are in fact two questions, invariably confused with each other: first, how was the category (the box) of genocide originally constructed; second, how are individual cases selected to fit the box. The term 'genocide' was invented in 1944 by Rafael Lemkin, a Polish Jewish academic lawyer living the USA. In 1948, under the massive shadow of the Holocaust, the UN Convention on the Prevention and Punishment of the Crime of Genocide adopted his terminology to produce the official definition of genocide that has shadowed us ever since. At the core of the definition are "acts committed with intent to destroy in whole or in part a national, ethnic racial or religious group." The problems of this definition soon became apparent – especially the question of proving intent and the deliberate exclusion of groups defined in political terms. In 1981 the anthropologist Leo Kuper vividly showed the political compromises behind the definition and the fatal power still left with the sovereign state. Rather than preventing genocide, he pointed out, "it reads like a manual for contemporary practice".

Three new books on genocide exemplify major tendencies in the current literature. Manus I. Midlarskey's more ambitious comparative and historical study, The Killling Trap, applies a welcome rigour to the Lemkin-UN definition. He identifies two elements as crucial. First is the matter of state intent and policy: was it the policy of the perpetrating state to commit mass murder with exterminatory intent? Second, were non-combatants of a particular ethno-religious identity, the objects of extermination (for example, in order to eradicate them all from a given territory). The causal theory is inherent in the definition: a necessary condition for genocide to occur is "vulnerability of the target group" and real or purported connection with threats to state security. Applying the two master criteria results in three incontrovertible cases of 20th century genocide: the Armenians in 1915-1916, the Holocaust of 1941-1945 and the Tutsi of Rwanda in 1994.

But Midlarksy introduces further complications: there are also 'politicides': mass murders of designated socio-economic or political enemies of the state that leave the majority of the population intact after purging offenders and also do not destroy cultural infrastructure. One example was the massacre of 150,000-200,000 Hutu in Burundi in 1972. Midlarsky pays special attention to three cases excluded because of the absence of exterminatory intent. One, mass violence by Serbs against Bosnia Muslims; two, the Nanjing massacre of 1937; and three, the 1904 killings of the Herero in German-governed South West Africa. Thus, by theoretical rather than sentimental logic ("cynical realpolitik constrained by loss"), the Srebinica massacre of July 1995 was 'just' an episode of ethnic cleansing with a clear genocidal incident.

Midlarsky's historical and comparative superstructure, though, is so broad that his theoretical variables become more and more abstract. He rejects goal attainment and rational choice theories, as well as those concentrating on ideology and utopianism. Instead, "Realpolitik is understood as policies that preserve and strengthen the state, while loss is the experience of either (1) transfer of territory, population, authority, or some combination thereof to another political entity, or (2) military defeat or significant casualties in political violence (eg war) that either are about to be or have already been incurred." His flowchart of the transformation of massacre into genocide, shows: risk acceptance, domain of losses, state insecurity, brute force, realpolitik, loss compensation-altruistic punishment, risk minimisation. Heavy stuff.

A quite different genre is the exposure of one case. Darfur: The Ambiguous Genocide, Gerard Prunier's characteristically detailed and laconic account of the post-2004 Darfur crisis in the Western Sudan is a fine example: the first contested genocide of the new century. Most of us know of Sudan only through the violence between the North and the South, part of an ongoing religious conflict between the Arab Muslim North and the African Christian South. But just when this became a media story of negotiations, so another conflict emerged in which the Sudanese state came to use vicious political violence against the 'African' residents of the Western Sudan.

The attacks on these groups are described by Prunier as "counter insurgency on the cheap," leading to "quasi-genocide". He concentrates on the relationship between the internal history and the external public and international media story: "…the problem of defining whether it was a genocide or not was largely a foreign problem since the vocabulary used had much larger implications for the international community than it had for the Darfuri themselves." The deepening conflict (eventually leading to some 300,000 Darfuris being killed) created further sophistry in the international community, now reduced, in Prunier's words, "into a state of irritable and fruitless complaining". At first there was a 'humanitarian crisis' – in other words, just another insoluble problem. What is conventionally known as 'world opinion' now cared about Darfur, even if the actual mechanics of what was happening remained obscure. The United Nations, the EU, the African Union all lined up to express ritual indignation, either nearly using or nearly avoiding the word 'genocide'.

There were four positions in this semantic game. 1. Explosion of tribal conflict. 2. Counter insurgency campaign gone wrong. 3. Deliberate ethnic cleansing (the government of Sudan trying to displace or eliminate 'African' tribes in order to replace them with 'Arab' ones). 4. Something like a genocide.

Prunier's own position is more ambiguous than it needs to be – why does he say that according to the UN 1948 definition, it's obvious that Darfur is a genocide, but according to the definition he applied to Rwanda, it is not? Critically, his point is that whether the 'big G word' is used or not, makes little difference to the people suffering on the spot. Nobody denied that an enormous quantity of human beings had been killed, but was it genocide? "Although it made little difference to the interested parties who continued to die without recourse to international legal concepts, the word became a question of the utmost relevance in the media. What was it that we were witnessing?"

Ordinary readers will be attracted to James Waller's book Becoming Evil but they will search hard for any clear answer to his question: "Why do ordinary people commit extraordinary evil?" His project is to replace theories of bad people by theories of bad situations – but he keeps stepping backwards by making lists of factors "from many disciplines". In one minute he's talking about human nature, the next minute about how our responses to authority are shaped. He offers, rather patronisingly, a guide to what his fellow social psychologists have been doing since Milgram's famous obedience studies fifty years ago. Unfortunately, none of them comes anywhere near the shattering importance of the original 'obedience to authority' studies.

The term 'dumbed down' hardly does justice to Waller's style, as in: "Like many other people, social scientists have had a hard time wrapping their minds around exactly what evil is and is not." His original question becomes: "What factors lead some of us to perpetrate extraordinary evil while others of us stand by indifferently or, occasionally, resist extraordinary evil?" Waller's guide, though, gives an interesting cluster of relevant subjects: tendencies of human nature: ethnocentrism, xenophobia, and desire for social dominance; forces that mould the identities of the perpetrators: cultural belief systems, moral disengagement, and rational self-interest; cultures of cruelty: professional socialisation, merger of role and person. And finally, the social death of the victims: us-them thinking, dehumanisation of the victims, and blaming the victims.

Just about every book on genocide either begins or ends with a set of apologies. It is as if the writer senses that nothing he or she writes can ever live up to the moral demands of the horror. Midlarsky modestly hopes that detachment, abstraction and analytic rigour do not prove to be offensive to the survivors. He offers a special apologia for theories that try to understand the perspective of perpetrators. These must look so callous to the victims, survivors and families – but to explain is not to condone; victims did not die in vain; only the understanding of causes offers possible prevention. These heartfelt apologies are no doubt sincere. But they cannot hide the tragedy of knowing that all these academic labours – the institutes, fees, models, royalties, conferences, journals – will not come to much in even foreseeing let alone preventing the next horror. This is surely the point – and not Waller's misplaced self flagellation about needing "to understand the conditions under which many of us could be transformed into killing machines," nor, even less, his thanks to his children "for being a rainbow of realistic hope at the many times when goodness seemed so far away from me as I immersed myself in an impenetrable monotony of cruelties over the past few years."

The trouble is far, far more profound than these apologies admit. It lies in the very nature of the legal construction of atrocities. Adorno saw just this. "What the Nazis did to the Jews was 'unspeakable'," he wrote round 1950; language has no word for it, even 'mass murder' would have sounded too banal. "And yet," Adorno continued, "a term needed to be found if the victims… were to be spared the curse of having no thoughts turned unto them. So in English the concept of genocide was coined. But by being codified, as set down in the International Declaration of Human Rights, the unspeakable was made, for the sake of protest, commensurable. By its elevation to a concept, its possibility is virtually recognised: an institution to be forbidden, rejected, discussed. One day negotiations may take place in the forum of the United Nations on whether some new atrocity comes under the heading of genocide, whether nations have a right to intervene that they do not want to exercise in any case and whether, in view of the unforeseen difficulty of applying this in practice, the whole concept of genocide should be removed from the statutes. Soon afterwards there are inside-page headlines in journalese: 'East-Turkestan genocide programme nears completion'."

Half a century later, this is at least one Frankfurt school irony which has turned into a prediction. Even the Turkish reference is uncannily right. The British and most European governments would not want the Turkish human rights record – the Armenian genocide that didn't happen and current torture in jails – to be an 'obstacle' to Turkish membership of the EU. And it won't be. And neither will the preservation nor the deconstruction of official definitions of genocide do anything to stop the next unspeakable horror.