The British government regularly makes documents that were once stamped 'confidential' or 'secret' available for inspection by ordinary citizens at the National Archives in Kew, close to Kew Gardens in Surrey. Amongst March's batch of documents were some to do with left-wing singer-songwriter Ewan MacColl. Nowadays, MacColl is primarily known for his songs of populist appeal rather than his politics. 'Sweet Thames, Flow Softly' and 'Dirty Old Town', for example, that have developed lives of their own, the sort of songs that ditch their authors' names and are labelled 'folk songs' as if they came from the ancient past. MacColl's most famous song is in another league altogether: made famous by American soul singer Roberta Flack and one of the most covered songs of our age, 'The First Time Ever I Saw Your Face' touchingly recalls MacColl's first meeting with Peggy Seeger, half sister to American folk legend Pete, and MacColl's third wife. But Kew's dark materials reveal that love poetry was not on the mind of the security services and police when they had MacColl under observation.

Ewan MacColl was part working-class hero, part-cipher. During the early 1950s he was a mainspring of Britain's second folk revival. The music that MacColl and his contemporary Bert Lloyd championed kept its working-class dirt under its nails. This politicised musical movement drew inspiration from the traditions of English political radicalism and nurtured the rumour that their brand of folk had a hotline to the Kremlin. The fact that Special Branch, the constabulary and the secret squirrels of the Post Office therefore kept tabs on MacColl and co made a certain kind of sense – many folkies did want to challenge if not the state, then the status quo.

But official interest in MacColl went back a long way before the folk scare. "He did know that there were records on a lot of things that he did," Peggy Seeger told me recently. "When they refused him his visa to go to the States in 1961 or 1962, they presented him with a dossier that was about six inches high. Reports, reports, reports. This was when the American Embassy was in a little Georgian house just off Grosvenor Square. You actually talked to the consul; he was the one who decided if you got a visa. It was very informal. We actually compared our dossiers. Mine was about two inches high and I hadn't even done anything. We just assumed that people knew what we were doing."

The teenage MacColl may have been on the spooks' radar as early as 1929. He took part in a hunger march that year and was beaten up by the police who attacked the marchers at night while they were sleeping in a railway roundhouse. "It was a cataclysmic experience for him," says Seeger. "They would have had his records from then probably and they probably kept tabs on him because of his father, who was a well known troublemaker."

MacColl didn't make life easy for the security forces. He laid many false trails. One was that he was born in Auchterarder, Scotland (actually the birthplace of his mother). He seems never to have made any attempt to set the record straight. MacColl was actually born James Henry Miller, in Salford, Lancashire (he changed his name around 1949). The earliest among the newly released documents is his July 1935 passport application. The 20-year-old 'Jimmie Miller' gives his place of birth as 4 Andrew Street, Salford and occupation as 'playwright'.

This was not total invention. He achieved some success as a political dramatist long before the folk revival, and collaborated with his young, equally radical actress wife, Joan Littlewood, in the formation of an experimental theatre company in Manchester and then the Theatre Union, the 'theatre of the people'. Greeted with some popular success touring in the north of England between 1936 and 1939, Littlewood and Miller were arrested one night while performing the incendiary play Last Edition, which dealt with events leading up to the Munich pact. They were both charged with disturbing the peace and bound over for two years (during which they were banned from performing). This must have convinced the spooks that the radical couple posed a threat to national security. But there was more.

One of MacColl's key collaborators in the politicised folk revival was legendary American ethnomusicologist Alan Lomax, who had surveillance problems of his own. He had fled America for England in 1950 to escape the attentions of the House Un-American Activities Committee. Lomax, with his impressive recording and researching credentials found work with the BBC and stayed in London until 1958. It was Lomax who introduced MacColl to Bert Lloyd and collaborated with them both in setting up the Ballads and Blues club in London, the legendary incubator of the folk revival. Nonetheless Lomax, with a distinguished track record working for the Library of Congress, was given a rough ride, as revealed in his BBC file which, amid memos about fees and expenses and exchanges with the Ministry of Labour about employing a foreigner, contains one confidential exchange from 1956 which refers to his "potted Marxism" and "sexual permissiveness".

MacColl, like Lomax, worked for the BBC and had a BBC file. But materials in the BBC Written Archives Centre at Caversham, Berkshire are not generally available for inspection. They, unlike the Kew materials, have not been converted to microfiche or film and cannot be downloaded over the Internet. What is apparent from the Kew documents is that the BBC – for whom MacColl worked periodically as an actor, narrator and producer – knew about the political allegiances of 'Jimmie Miller'.

By 1939 Miller and Littlewood were under regular observation. One police report from April 1939 says, "at weekends, and more particularly when Miller's parents are away from home, a number of young men who have the appearance of Communist Jews are known to visit Oak Cottage [in Hyde, Cheshire]. It is thought that they come from Manchester."

The opened archives mainly consist of reports and documents from various local, county and metropolitan police forces, the Special Branch and military records. Articles from the Daily Worker and Challenge, intercepted letters and suchlike further illuminate periods of his life. But reading through the documents is not the easiest of tasks. The way materials are presented makes matters confusing. For example, there are entries from files that suggest omissions, more likely incompetence than conspiracy though you can never be sure.

One extract dated September 1947 has a cross-dated reference to April 1932 during the time Miller was living at Coburg Street in Broughton. It alludes to a 'Chief Constable Salford' report about the British Workers Sports Federation – "This 'Federation' is a name adopted by a branch of the Communist Party and its sole aim is the furtherance of the Communist Party Policy, and in common with all other party factions it seeks to bring about in every possible wasy [sic] an atmosphere of 'class Antagonism'."

MacColl died in October 1989, having delivered the manuscript for his autobiography, written in the late 1980s. Journeyman has plenty of omissions and skip-forwards. Reading through Kew's MacColl papers alongside his own account requires reading between the lines in both cases. Evident to anyone with detailed knowledge of the period, its politics and the personalities, is that there has been a judicious sieving of what can and cannot enter the public domain. Too much points to more.

One page of the dossier has Peggy Seeger's photograph on it beside one of Joan Littlewood. While the authorities are normally so painstaking about crossing referencing there is no reference to Seeger's file, which suggests that someone has mistaken Seeger for Littlewood. It's the kind of trivial error that could, in the wrong circumstances, have had serious consequences.

Perhaps the most intriguing thing about the new material concerns Miller's wartime service. He omits all mention of it in Journeyman. Kew shows he enlisted in July 1940 as a private in the King's Regiment and by December 1940 had been declared a deserter. "When they told him," Seeger recalls, "that he was never going to be sent abroad to fight, which according to him was the reason he deserted. He was told that he'd been in the fight too soon." Miller's card had been marked as a 'premature anti-fascist' in the parlance of the times, yet he never properly faced charges for desertion, though the police and other authorities knew his addresses. "He said it was an exchange, an agreement, and he wasn't going to say what he was told," says Seeger. "He thought that was why he was never picked up. He was very, very close-mouthed about this. He left a real mystery behind about that."