“I knew some of them straight away,” he told me last month. “I had feared Bucca all the way down on the plane. But when I got there, it was much better than I thought. In every way.”

The jihadist, who uses the nom de guerre Abu Ahmed, entered Camp Bucca as a young man a decade ago, and is now a senior official within Islamic State (Isis) – having risen through its ranks with many of the men who served time alongside him in prison. Like him, the other detainees had been snatched by US soldiers from Iraq’s towns and cities and flown to a place that had already become infamous: a foreboding desert fortress that would shape the legacy of the US presence in Iraq.

The other prisoners did not take long to warm to him, Abu Ahmed recalled. They had also been terrified of Bucca, but quickly realised that far from their worst fears, the US-run prison provided an extraordinary opportunity. “We could never have all got together like this in Baghdad, or anywhere else,” he told me. “It would have been impossibly dangerous. Here, we were not only safe, but we were only a few hundred metres away from the entire al-Qaida leadership.”

It was at Camp Bucca that Abu Ahmed first met Abu Bakr al-Baghdadi, the emir of Isis who is now frequently described as the world’s most dangerous terrorist leader. From the beginning, Abu Ahmed said, others in the camp seemed to defer to him. “Even then, he was Abu Bakr. But none of us knew he would ever end up as leader.”

Abu Ahmed was an essential member of the earliest incarnation of the group. He had been galvanised into militancy as a young man by an American occupation that he and many like him believed was trying to impose a power shift in Iraq, favouring the country’s larger Shia population at the expense of the dominant Sunnis. His early role in what would become Isis led naturally to the senior position he now occupies within a revitalised insurgency that has spilled across the border into Syria. Most of his colleagues regard the crumbling order in the region as a fulfilment of their ambitions in Iraq – which had remained unfinished business, until the war in Syria gave them a new arena.

He agreed to speak publicly after more than two years of discussions, over the course of which he revealed his own past as one of Iraq’s most formidable and connected militants – and shared his deepening worry about Isis and its vision for the region. With Iraq and Syria ablaze, and the Middle East apparently condemned to another generation of upheaval and bloodshed at the hands of his fellow ideologues, Abu Ahmed is having second thoughts. The brutality of Isis is increasingly at odds with his own views, which have mellowed with age as he has come to believe that the teachings of the Qur’an can be interpreted and not read literally.

His misgivings about what the Islamic State has become led him to speak to the Guardian in a series of expansive conversations, which offer unique insight into its enigmatic leader and the nascent days of the terror group – stretching from 2004, when he met Abu Bakr al-Baghdadi in Camp Bucca, to 2011, when the Iraqi insurgency crossed the border into Syria.

At the beginning, back in Bucca, the prisoner who would become the most wanted man in the world had already set himself apart from the other inmates, who saw him as aloof and opaque. But, Abu Ahmed recalled, the jailers had a very different impression of Baghdadi – they saw him as a conciliatory and calming influence in an environment short on certainty, and turned to him to help resolve conflicts among the inmates. “That was part of his act,” Abu Ahmed told me. “I got a feeling from him that he was hiding something inside, a darkness that he did not want to show other people. He was the opposite of other princes who were far easier to deal with. He was remote, far from us all.”

* * *

Baghdadi was born Ibrahim ibn Awwad al-Badri al-Samarrai in 1971, in the Iraqi city of Samarra. He was detained by US forces in Falluja, west of Baghdad, in February 2004, months after he had helped found a militant group, Jeish Ahl al-Sunnah al-Jamaah, which had taken root in the restive Sunni communities around his home city.

“He was caught at his friend’s house,” said Dr Hisham al-Hashimi, an analyst who advises the Iraqi government on Isis. “His friend’s name was Nasif Jasim Nasif. Then he was moved to Bucca. The Americans never knew who they had.” Most of Baghdadi’s fellow prisoners – some 24,000 men, divided into 24 camps – seem to have been equally unaware. The prison was run along strictly hierarchical lines, down to a Teletubbies-like uniform colour scheme which allowed jailers and captives alike to recognise each detainee’s place in the pecking order. “The colour of the clothes we wore reflected our status,” said Abu Ahmed. “If I remember things correctly, red was for people who had done things wrong while in prison, white was a prison chief, green was for a long sentence and yellow and orange were normal.”

When Baghdadi, aged 33, arrived at Bucca, the Sunni-led anti-US insurgency was gathering steam across central and western Iraq. An invasion that had been sold as a war of liberation had become a grinding occupation. Iraq’s Sunnis, disenfranchised by the overthrow of their patron, Saddam Hussein, were taking the fight to US forces – and starting to turn their guns towards the beneficiaries of Hussein’s overthrow, the country’s majority Shia population.

The small militant group that Baghdadi headed was one of dozens that sprouted from a broad Sunni revolt – many of which would soon come together under the flag of al-Qaida in Iraq, and then the Islamic State of Iraq. These were the precursors to the juggernaut now known simply as the Islamic State, which has, under Bagdhadi’s command, overrun much of the west and centre of the country and eastern Syria, and drawn the US military back to a deeply destabilised region less than three years after it left vowing never to return.

But at the time of his stay at Bucca, Baghdadi’s group was little-known, and he was a far less significant figure than the insurgency’s notional leader, the merciless Abu Musab al-Zarqawi, who came to represent the sum of all fears for many in Iraq, Europe and the US. Baghdadi, however, had a unique way to distinguish himself from the other aspiring leaders inside Bucca and outside on Iraq’s savage streets: a pedigree that allowed him to claim direct lineage to the Prophet Muhammad. He had also obtained a PhD in Islamic studies from the Islamic University of Baghdad, and would draw on both to legitimise his unprecedented claim to anoint himself caliph of the Islamic world in July 2014, which realised a sense of destiny evident in the prison yard a decade earlier.

“Baghdadi was a quiet person,” said Abu Ahmed. “He has a charisma. You could feel that he was someone important. But there were others who were more important. I honestly did not think he would get this far.”

Baghdadi also seemed to have a way with his captors. According to Abu Ahmed, and two other men who were jailed at Bucca in 2004, the Americans saw him as a fixer who could solve fractious disputes between competing factions and keep the camp quiet.

“But as time went on, every time there was a problem in the camp, he was at the centre of it,” Abu Ahmed recalled. “He wanted to be the head of the prison – and when I look back now, he was using a policy of conquer and divide to get what he wanted, which was status. And it worked.” By December 2004, Baghdadi was deemed by his jailers to pose no further risk and his release was authorised.

“He was respected very much by the US army,” Abu Ahmed said. “If he wanted to visit people in another camp he could, but we couldn’t. And all the while, a new strategy, which he was leading, was rising under their noses, and that was to build the Islamic State. If there was no American prison in Iraq, there would be no IS now. Bucca was a factory. It made us all. It built our ideology.”

As Isis has rampaged through the region, it has been led by men who spent time in US detention centres during the American occupation of Iraq – in addition to Bucca, the US also ran Camp Cropper, near Baghdad airport, and, for an ill-fated 18 months early in the war, Abu Ghraib prison on the capital’s western outskirts. Many of those released from these prisons – and indeed, several senior American officers who ran detention operations – have admitted that the prisons had an incendiary effect on the insurgency.

“I went to plenty of meetings where guys would come through and tell us how well it was all going,” said Ali Khedery, a special aide to all US ambassadors who served in Iraq from 2003-11, and to three US military commanders. But eventually even top American officers came to believe they had “actually become radicalising elements. They were counterproductive in many ways. They were being used to plan and organise, to appoint leaders and launch operations.”

Abu Ahmed agreed. “In prison, all of the princes were meeting regularly. We became very close to those we were jailed with. We knew their capabilities. We knew what they could and couldn’t do, how to use them for whatever reason. The most important people in Bucca were those who had been close to Zarqawi. He was recognised in 2004 as being the leader of the jihad.

“We had so much time to sit and plan,” he continued. “It was the perfect environment. We all agreed to get together when we got out. The way to reconnect was easy. We wrote each other’s details on the elastic of our boxer shorts. When we got out, we called. Everyone who was important to me was written on white elastic. I had their phone numbers, their villages. By 2009, many of us were back doing what we did before we were caught. But this time we were doing it better.”

According to Hisham al-Hashimi, the Baghdad-based analyst, the Iraqi government estimates that 17 of the 25 most important Islamic State leaders running the war in Iraq and Syria spent time in US prisons between 2004 and 2011. Some were transferred from American custody to Iraqi prisons, where a series of jailbreaks in the last several years allowed many senior leaders to escape and rejoin the insurgent ranks.

Abu Ghraib was the scene of the biggest – and most damaging – breakout in 2013, with up to 500 inmates, many of them senior jihadists handed over by the departing US military, fleeing in July of that year after the prison was stormed by Islamic State forces, who launched a simultaneous, and equally successful, raid on nearby Taji prison.

Iraq’s government closed Abu Ghraib in April 2014 and it now stands empty, 15 miles from Baghdad’s western outskirts, near the frontline between Isis and Iraq’s security forces, who seem perennially under-prepared as they stare into the heat haze shimmering over the highway that leads towards the badlands of Falluja and Ramadi.

Parts of both cities have become a no-go zone for Iraq’s beleaguered troops, who have been battered and humiliated by Isis, a group of marauders unparalleled in Mesopotamia since the time of the Mongols. When I visited the abandoned prison late this summer, a group of disinterested Iraqi forces sat at a checkpoint on the main road to Baghdad, eating watermelon as the distant rumble of shellfire sounded in the distance. The imposing walls of Abu Ghraib were behind them, and their jihadist enemies were staked out further down the road.

The revelation of abuses at Abu Ghraib had a radicalising effect on many Iraqis, who saw the purported civility of American occupation as little improvement on the tyranny of Saddam. While Bucca had few abuse complaints prior to its closure in 2009, it was seen by Iraqis as a potent symbol of an unjust policy, which swept up husbands, fathers, and sons – some of them non-combatants – in regular neighbourhood raids, and sent them away to prison for months or years.

At the time, the US military countered that its detention operations were valid, and that similar practices had been deployed by other forces against insurgencies – such as the British in Northern Ireland, the Israelis in Gaza and the West Bank, and the Syrian and Egyptian regimes.

Even now, five years after the US closed down Bucca, the Pentagon defends the camp as an example of lawful policy for a turbulent time. “During operations in Iraq from 2003 to 2011, US Forces held thousands of Law of War detainees,” said Lt Col Myles B Caggins III, a US Department of Defense spokesman for detainee policy. “These type of detentions are common practice during armed conflict. Detaining potentially dangerous people is the legal and humane method of providing security and stability for civilian populations.”

* * *

Some time after Baghdadi was released from Bucca, Abu Ahmed was also freed. After being flown to Baghdad airport, he was picked up by men he had met in Bucca. They took him to a home in the west of the capital, where he immediately rejoined the jihad, which had transformed from a fight against an occupying army into a vicious and unrestrained war against Iraqi Shia.

Death squads were by then roaming Baghdad and much of central Iraq, killing members of opposite sects with routine savagery and exiling residents from neighbourhoods they dominated. The capital had quickly become a very different place to the city Abu Ahmed had left a year earlier. But with the help of new arrivals at Bucca, those inside the prison had been able to monitor every new development in the unfolding sectarian war. Abu Ahmed knew the environment he was returning to. And his camp commanders had plans for him.

The first thing he did when he was safe in west Baghdad was to undress, then carefully take a pair of scissors to his underwear. “I cut the fabric from my boxers and all the numbers were there. We reconnected. And we got to work.” Across Iraq, other ex-inmates were doing the same. “It really was that simple,” Abu Ahmed said, smiling for the first time in our conversation as he recalled how his captors had been outwitted. “Boxers helped us win the war.”

Zarqawi wanted a 9/11 moment to escalate the conflict – something that would take the fight to the heart of the enemy, Abu Ahmed recalled. In Iraq, that meant one of two targets – a seat of Shia power or, even better, a defining religious symbol. In February 2006, and again two months later, Zarqawi’s bombers destroyed the Imam al-Askari shrine in Samarra, north of Baghdad. The sectarian war was fully ignited and Zarqawi’s ambitions realised.

Asked about the merits of this violent provocation, Abu Ahmed paused for the first time in our many conversations. “There was a reason for opening this war,” he said. “It was not because they are Shia, but because the Shia were pushing for it. The American army was facilitating the takeover of Iraq and giving the country to them. They were in cooperation with each other.”

He then reflected on the man who gave the orders. “Zarqawi was very smart. He was the best strategist that the Islamic State has had. Abu Omar [al-Baghdadi] was ruthless,” Abu Ahmed said, referring to Zarqawi’s successor, who was killed in a US-led raid in April 2010. “And Abu Bakr is the most bloodthirsty of all.

“After Zarqawi was killed, the people who liked killing even more than him became very important in the organisation. Their understanding of sharia and of humanity was very cheap. They don’t understand the Tawheed (the Qur’anic concept of God’s oneness) the way it was meant to be understood. The Tawheed should not have been forced by war.”

Despite reservations that were already starting to stir, by 2006, Abu Ahmed had become part of a killing machine that would operate at full speed for much of the following two years. Millions of citizens were displaced, neighbourhoods were cleansed along sectarian lines, and an entire population numbed by unchecked brutality.

That summer, the US finally caught up with Zarqawi, with the help of Jordanian intelligence, killing him in an airstrike north of Baghdad. From late 2006, the organisation was on the back foot – hampered by a tribal revolt that uprooted its leadership from Anbar and shrank its presence elsewhere in Iraq. But according to Abu Ahmed, the group used the opportunity to evolve, revealing a pragmatism in addition to its hardline ideology. For Isis, the relatively quiet years between 2008 and 2011 represented a lull, not a defeat.

By this time, Abu Bakr al-Baghdadi had risen steadily through the group to become a trusted aide to its leader, Abu Omar al-Baghdadi, and his deputy, the Egyptian jihadist Abu Ayub al-Masri. It was at this point, Abu Ahmed said, that Isis made an approach to the Ba’athist remnants of the old regime – ideological opponents who shared a common enemy in the US and the Shia-led government it backed.

Earlier incarnations of Isis had dabbled with the Ba’athists, who lost everything when Saddam was ousted, under the same premise that “my enemy’s enemy is my friend”. But by early 2008, Abu Ahmed and other sources said, these meetings had become far more frequent – and many of them were taking place in Syria.

Syria’s links to the Sunni insurgency in Iraq had been regularly raised by US officials in Baghdad and by the Iraqi government. Both were convinced that the Syrian president, Bashar al-Assad, allowed jihadists to fly into Damascus airport, where military officials would escort them to the border with Iraq. “All the foreigners I knew got into Iraq that way,” Abu Ahmed told me. “It was no secret.”

* * *

From 2008, when the US began to negotiate the transition of its powers to Iraq’s feeble security institutions – and therefore pave the way to its own exit – the Americans increasingly turned to only a few trusted figures in the Iraqi government. One of them was Major General Hussein Ali Kamal, the director of intelligence in the country’s Interior Ministry. A secular Kurd who had the trust of the Shia establishment, one of Kamal’s many duties was to secure Baghdad against terror attacks.

Like the Americans, General Kamal was convinced that Syria was destabilising Iraq, an assessment based on the interrogations of jihadists who had been captured by his troops. Throughout 2009, in a series of interviews, Kamal laid out his evidence, using maps that plotted the routes used by jihadists to cross the border into western Iraq, and confessions that linked their journeys to specific mid-ranking officers in Syrian military intelligence.

As Isis activity ebbed in Iraq, he had become increasingly obsessed with two meetings that had taken place in Syria early in 2009, which brought together Iraqi jihadists, Syrian officials and Ba’athists from both countries. (Kamal, who was diagnosed with a rare cancer in 2012, died earlier this year, and authorised me to publish details of our conversations. “Just tell the truth,” he said during our last interview in June 2014.)

When I first met him in 2009, he was poring over transcripts of recordings that had been made at two secret meetings in Zabadani, near Damascus, in the spring of that year. The attendees included senior Iraqi Ba’athists who had taken refuge in Damascus since their patron Saddam was ousted, Syrian military intelligence officers, and senior figures in what was then known as al-Qaida in Iraq. The Syrians had developed links to the jihadists since the earliest days of the anti-US insurgency and had used them to unsettle the Americans and their plans for Iraq.

“By early in 2004/05, Islamic elements, jihadists and disenfranchised Ba’athists were starting to get together,” said Ali Khedery, the former adviser to American ambassadors and senior commanders in Bagdhad. “They were naturally disciplined, well organised people who knew the lay of the land. And over time, some folks who were Ba’athists became more and more Islamist and the insurgency raged. By 2007, General [David] Petraeus was saying there was crystal clear intelligence of cooperation between Syrian military intelligence and the jihadists. Though the motivations never really aligned 100%.”

In our conversations, Abu Ahmed emphasised the Syrian connection to Iraq’s insurgency. “The mujahideen all came through Syria,” he said. “I worked with many of them. Those in Bucca had flown to Damascus. A very small number had made it from Turkey, or Iran. But most came to Iraq with the help of the Syrians.”

The supply line was viewed by Iraqi officials as an existential threat to Iraq’s government and was the main source of the poisonous relationship between Nouri al-Maliki, then Iraq’s prime minister, and Bashar al-Assad. Maliki had become convinced early in the civil war that Assad was trying to undermine his regime as a way to embarrass the Americans, and the evidence he saw in 2009 from the meeting in Damascus took his loathing of the Syrian leader to a whole new level.

“We had a source in the room wearing a wire,” at the meeting in Zabadani, General Kamal told me at the time. “He is the most sensitive source we have ever had. As far as we know, this is the first time there has been a strategic level meeting between all of these groups. It marks a new point in history.”

The Ba’athists present led the meeting. Their aim, according to General Kamal’s source, was to launch a series of spectacular attacks in Baghdad and thereby undermine Maliki’s Shia-majority government, which had for the first time begun to assert some order in post-civil war Iraq. Until then, al-Qaida in Iraq and the Ba’athists had been fierce ideological enemies, but the rising power of the Shias – and their backers in Iran – brought them together to plan a major strike on the capital.

By July 2009, the Interior Ministry had increased security at all checkpoints across the Tigris river into Baghdad, making a commute at any time of day even more insufferable than normal. And then General Kamal received a message from his source in Syria. The extra security at the bridges had been spotted by the attack plotters, he said. New targets were being chosen, but he didn’t know what they were, or when they would be hit. For the next two weeks, Kamal worked well into the evening in his fortified office in the southern suburb of Arasat, before being sped by armoured convoy across the July 14 Bridge – which had been a target only days earlier – to his home inside the Green Zone.

For the rest of the month, General Kamal spent several hours each scorching night sweating it out on a treadmill, hoping that the exercise would clear his head and get him ahead of the attackers. “I may be losing weight, but I’m not finding the terrorists,” he told me during our last conversation before the attackers finally struck. “I know they’re planning something big.”

On the morning of 19 August, the first of three flat-bed trucks carrying three large 1000-litre water tanks, each filled with explosives, detonated on an overpass outside the Finance Ministry in south-eastern Baghdad. The blast sent a rumble across the Emerald City, raising desert soil that caked homes brown, and sending thousands of pigeons scattering through the sky. Three minutes later, a second enormous bomb blew up outside the Foreign Ministry on the northern edge of the Green Zone. Shortly after that, a third blast hit a police convoy near the Finance Ministry. More than 101 people were killed and nearly 600 wounded; it was one of the deadliest attacks in the six-year-old Iraqi insurgency.

“I failed,” Kamal told me that day. “We all failed.” Within hours, he was summoned to meet Maliki and his security chiefs. The prime minister was livid. “He told me to present what I had to the Syrians,” Kamal later said. “We arranged with Turkey to act as a mediator and I flew to Ankara to meet with them. I took this file” – he tapped a thick white folder on his desk – “and they could not argue with what we showed them. The case was completely solid and the Syrians knew it. Ali Mamlouk [the head of Syrian general security] was there. All he did was look at me smiling and say ‘I will not recognise any official from a country that is under US occupation’. It was a waste of time.” Iraq recalled its ambassador to Damascus, and Syria ordered its envoy to Baghdad home in retaliation. Throughout the rest of the year, and into early 2010, relations between Maliki and Assad remained toxic.

In March 2010, Iraqi forces, acting on a US tip, arrested an Islamic State leader named Munaf Abdul Rahim al-Rawi, who was revealed to be one of the group’s main commanders in Baghdad, and one of the very few people who had access to the group’s then leader, Abu Omar al-Baghdadi. Al-Rawi talked. And in a rare moment of collaboration, Iraq’s three main intelligence bodies, including General Kamal’s Intelligence Division, conspired to get a listening device and GPS location tracker in a flower box delivered to Abu Omar’s hideout.

After it was confirmed that Abu Omar and his deputy, Abu Ayub al-Masri, were present at a house six miles south-west of Tikrit, it was attacked in a US-led raid. Both men detonated suicide vests to avoid being captured. Messages to Osama bin Laden and Ayman al-Zawahiri were found on a computer inside the house. Much like Bin Laden’s safe house in Pakistan, where he would be killed a little more than a year later, Abu Omar’s hideout had no internet connections or telephone lines – all important messages were carried in and out by only three men. One of them was Abu Bakr al-Baghdadi.

“Abu Bakr was a messenger for Abu Omar,” Abu Ahmed told me. “He became the closest aide to him. The messages that got to Osama bin Laden were sometimes drafted by him and their journey always started with him. When Abu Omar was killed, Abu Bakr was made leader. That time we all had in Bucca became very important again.”

The deaths of Abu Omar al-Baghdadi and Abu Ayub al-Masri were a serious blow to Isis, but the roles they had vacated were quickly filled by the alumni of Camp Bucca – whose upper echelons had begun preparing for this moment since their time behind the wire of their jail in southern Iraq. “For us it was an academy,” Abu Ahmed said, “but for them” – the senior leaders – “it was a management school. There wasn’t a void at all, because so many people had been mentored in prison.

“When [the civil war in] Syria became serious,” he continued, “it wasn’t difficult to transfer all that expertise to a different battle zone. The Iraqis are the most important people on the military and Shura councils in Isis now, and that is because of all of those years preparing for such an event. I underestimated Baghdadi. And America underestimated the role it played in making him what he is.”

* * *

Abu Ahmed remains a member of Isis; he is active in the group’s operations in both Iraq and Syria. Throughout our discussions, he portrayed himself as a man reluctant to stay with the group, and yet unwilling to risk any attempt to leave.

Life with Isis means power, money, wives and status – all attractive lures for young firebrands with a cause – but it also means killing and dominating for a worldview in which he no longer believes so fervently. He said hundreds of young men like him, who were drawn to a Sunni jihad after the US invasion, do not believe that the latest manifestation of the decade-long war remains true to its origins.

“The biggest mistake I made is to join them,” Abu Ahmed said, but added that leaving the group would mean that he and his family would certainly be killed. Staying and enforcing the group’s brutal vision, despite partially disavowing it, does not trouble Abu Ahmed, who sees himself as having few other options.

“It’s not that I don’t believe in Jihad,” he said. “I do,” he continued, his voice trailing away. “But what options do I have? If I leave, I am dead.”

The arc of his involvement with what is now the world’s most menacing terrorist group mirrors many others who now hold senior positions in the group: first a battle against an invading army, then a score to be settled with an ancient sectarian foe, and now, a war that could be acting out an end of days prophecy.

In the world of the Bucca alumni, there is little room for revisionism, or reflection. Abu Ahmed seems to feel himself swept along by events that are now far bigger than him, or anyone else.

“There are others who are not ideologues,” he said, referring to senior Isis members close to Baghdadi. “People who started out in Bucca, like me. And then it got bigger than any of us. This can’t be stopped now. This is out of the control of any man. Not Baghdadi, or anyone else in his circle.”

Martin Chulov covers the Middle East for the Guardian. He has reported from the region since 2005. Additional reporting by Salaam Rizk