The 9/11 terrorist attacks catapulted the US and UK into a more intensive period of security partnership than any event since the second world war. Tony Blair was the first world leader to visit Ground Zero in New York. He and George Bush shared the same burning conviction that the scale of the atrocity created a new reality.

They both saw the “war on terror” as an ideological struggle for the values which would shape the new century. This conviction carried Blair into volunteering Britain as the first lead nation for the Nato mission in Afghanistan in late 2001.

After 20 years of tough military action, in which the UK provided the largest number of troops after the US and took the second highest number of combat deaths, London played no part at all in the American decisions which led to the chaotic end to the Nato operation. Boris Johnson was reduced to pleading with Joe Biden through the media for a few more days to complete the pull-out. The fact that he was turned down flat exposed to public view how far British influence in Washington had dwindled since the “shoulder to shoulder” days of 2001.

Much of this is due to a changing America. For decades, the US led international crisis management operations. It seemed to be the natural order of things. We Europeans struggled for three years in the 1990s to stop the ethnic cleansing in Bosnia, but it was only when president Bill Clinton deployed US diplomatic muscle and military firepower that Serbian president Slobodan Milosevic accepted the Dayton peace settlement in 1995. Much the same pattern repeated itself in Kosovo in 1998-99.

But then the US led allies into the quagmire of Iraq and the forever war in Afghanistan, which turned American opinion decisively against deploying US forces to solve the world’s problems. The implications became dramatically clear to me when, as David Cameron’s national security adviser in 2011, I joined him on the call from Barack Obama telling us that the US would be taking a back seat in the Nato air campaign in Libya. This was not America’s fight, Obama explained, so he expected Britain and France to lead. I knew then that this marked a sea change in US security priorities.

From that point on, the US has been in retreat from international leadership. Obama opted out of any role in containing the civil war in Syria. Donald Trump effectively sealed the fate of the Afghan government by negotiating behind their backs a cynical withdrawal deal with the Taliban in 2020. Biden could have revised that deal, to increase the chances of an orderly handover. Instead he stuck with it and, as a result, will go down in history as presiding over the worst US military humiliation since Vietnam in 1975.

The UK has not had less influence than other US allies on these disastrous decisions. But the signal US failure to consult its allies has more serious implications for the UK because the fulcrum of British foreign policy for decades has been the claim to a privileged relationship with Washington.

This was always a partnership which mattered more to London than to Washington. And US unilateralism is nothing new. As Geoffrey Howe’s private secretary in 1983, I saw Margaret Thatcher’s consternation when the foreign secretary told her that president Ronald Reagan had sent the marines up the beaches of the Caribbean island of Grenada to quell an uprising, without consulting her – even though this was a Commonwealth country with the Queen as head of state.

That sharp squall soon blew over, dwarfed by the scale of US/UK shared interests. Although the Americans have always been unsentimental about the relationship, they have valued Britain for what it contributed – in particular, capable military forces and the will to use them, together with diplomatic savvy and a global intelligence capability.

These assets will remain important as both countries cope with a wider range of threats from hostile powers. But the uncomfortable truth for policymakers in London is that Britain has become less useful as America’s ally, in an era when European security has given way to the confrontation with China as the overriding US national security priority. The fact that Britain is no longer at the EU table further reduces London’s relevance to US foreign policy.

This is the nub of the problem facing the Johnson government. A central thesis of the Brexit argument was that casting off the shackles of the EU would leave Britain free to leverage its link to Washington and reap trade benefits. All that has now turned out to be a mirage. Biden’s internationalism had been oversold.

No one expected that it would be this brutal. America’s allies are waking up to the fact that, for all his “America is Back” rhetoric, Biden is driven primarily by the domestic imperative of governing his polarised nation. He will work with allies, but with a steely determination to serve US national interests.

Britain’s self-appointed role as Washington’s closest security partner was never a comfortable one. But, together with an influential role in Europe, it gave a purpose and shape to British foreign policy. Now “global Britain” has been shown up as a ship adrift without a compass, a slogan rather than a strategy.

The new Atlantic charter signed by Johnson and Biden in June has uplifting words about the shared commitments of the two nations to maintaining collective security and international stability. But the reality is that influence in Biden’s Washington will have to be earned by actions, not words.

A good place to start would be for senior British ministers to throw themselves into the gruelling task of forging a consensus on a tougher carbon reduction target for adoption at Glasgow’s UN Climate Change Conference, now a bare nine weeks away. Delivering a successful result there would show that Britain was back as an active, problem-solving international player.

In the wake of the Afghanistan debacle, it is time to shed the delusions of British exceptionalism and, after five years when Britain was effectively off the air in foreign policy terms, to focus on making a difference on a few key priorities.

• Lord Ricketts is a former national security adviser and former permanent secretary at the Foreign Office, and author of Hard Choices: What Britain Does Next

• The fall of Afghanistan: join a Guardian Live online event with our journalists Emma Graham-Harrison, Peter Beaumont and Julian Borger analysing the latest developments. Monday 6 September at 7pm BST. Book your tickets here. All profits will be donated to relevant charities.