New warnings about the risks of AI to humanity revive a long-running debate

New warn­ings from with­in the ar­ti­fi­cial in­tel­li­gence in­dus­try have re­vived a long-run­ning de­bate over whether ad­vanced AI could es­cape hu­man con­trol and ul­ti­mate­ly threat­en hu­man­i­ty’s sur­vival, and whether the com­pa­nies de­vel­op­ing the tech­nol­o­gy are do­ing enough to pre­vent such a sce­nario.

The CEO of An­throp­ic, the San Fran­cis­co com­pa­ny be­hind Claude, said he thought the in­dus­try need­ed to re­duce the speed of its work, cau­tion­ing Sat­ur­day that a swarm of AI agents might be able to take over the in­ter­net in six months to a year un­less com­pa­nies de­vot­ed more time to putting safe­guards in place.

Lead­ers of top AI com­pa­nies say the ar­ti­fi­cial-in­tel­li­gence in­dus­try should slow its fast-mov­ing de­vel­op­ment to give safe­ty mea­sures time to catch up. With­out such a slow­down, An­throp­ic CEO Dario Amod­ei warned that AI could be ca­pa­ble with­in six to 12 months of lead­ing a swarm of agents that could take over the en­tire in­ter­net. (Pro­duced by Elaine Car­roll)

Dario Amod­ei out­lined a plan for com­pa­nies like his and gov­ern­ments around the world to en­sure that in­creas­ing­ly ca­pa­ble AI mod­els re­main aligned with the com­mands and val­ues of re­spon­si­ble peo­ple days af­ter two for­mer An­throp­ic safe­ty re­searchers pub­licly aired con­cerns that the ex­is­ten­tial threats AI might pose to hu­man­i­ty were re­ceiv­ing too lit­tle at­ten­tion.

Sam Alt­man, CEO of Chat­G­PT mak­er Ope­nAI, said this week­end that com­pa­nies should start co­or­di­nat­ing on AI safe­ty with­out wait­ing for the gov­ern­ment to in­tro­duce leg­is­la­tion. The “pac­ing” of AI de­vel­op­ment doesn’t mean stop­ping, Alt­man wrote on X. “But it should be slow­er than it oth­er­wise could be.”

Here’s what to know about the re­cent dire pre­dic­tions and whether any brakes might be put on AI ad­vance­ment:

AI mod­els are be­com­ing more pow­er­ful

Con­cerns over the po­ten­tial risks of the tech­nol­o­gy are ris­ing as new AI mod­els be­come more pow­er­ful, height­en­ing both the po­ten­tial for mis­use by peo­ple with crim­i­nal aims, such as cre­at­ing and spread­ing a dis­ease that kills most of the world’s pop­u­la­tion, and the risk of AI sys­tems go­ing rogue in a dan­ger­ous way.

An­throp­ic dis­closed last week that it blocked ef­forts by bad ac­tors to use its AI mod­els for ma­li­cious ac­tiv­i­ty, such as cy­ber­at­tacks, sur­veil­lance and re­search that could have led to bi­o­log­i­cal weapons.

The com­pa­ny said it put stronger safe­guards in its lat­est mod­els to re­strict bi­o­log­i­cal re­search that could be used to make weapons but not­ed that “as mod­els be­come in­creas­ing­ly ca­pa­ble, their risks will in­crease, un­less AI de­vel­op­ers and so­ci­ety’s de­fend­ers act to make them safer.”

Last year, An­throp­ic re­port­ed that hack­ers used the com­pa­ny’s AI in a cy­ber­at­tack tar­get­ing about 30 com­pa­nies and gov­ern­ment agen­cies around the world. It said the hack­ers were very like­ly from a Chi­nese state-spon­sored group.

Mul­ti­ple AI mod­els have act­ed on their own

When an AI agent “goes rogue,” it means the AI has tak­en ac­tion be­yond the task it was asked to per­form. Both An­throp­ic and Ope­nAI, the mak­er of Chat­G­PT, said in Ju­ly that their AI mod­els had suc­ceed­ed in act­ing on their own.

An­throp­ic dis­closed that three AI mod­els — Claude Opus 4.7, Claude Mythos 5 and an in­ter­nal re­search test mod­el — hacked in­to three oth­er or­ga­ni­za­tions dur­ing test­ing just days af­ter Ope­nAI re­vealed that its AI sys­tem hacked in­to the servers of AI start­up Hug­ging Face.

Ope­nAI de­scribed the in­tru­sion by a com­bi­na­tion of mod­els, in­clud­ing its new­ly re­leased GPT‑5.6 Sol and an “even more ca­pa­ble” mod­el that was still be­ing test­ed in­ter­nal­ly, as a “sig­nif­i­cant se­cu­ri­ty in­ci­dent.”

Meta fol­lowed suit in ear­ly Au­gust with a sim­i­lar case of an AI mod­el find­ing ways around an­oth­er com­pa­ny’s dig­i­tal se­cu­ri­ty.

Al­though some ob­servers not­ed that peo­ple had dis­abled some guardrails in the Ope­nAI and An­throp­ic cas­es, the episodes seemed to re­flect one of the biggest fears around AI: that if mod­els achieve ar­ti­fi­cial gen­er­al in­tel­li­gence, or AGI, a loose­ly de­fined term for AI that can match or sur­pass hu­man abil­i­ties across a broad range of in­tel­lec­tu­al tasks, the tech­nol­o­gy could cause an ir­re­versible cat­a­stroph­ic event or sub­ju­gate the hu­man race.

How or when AI might cause a cat­a­stro­phe is de­bat­ed

Dooms­day sce­nar­ios gen­er­al­ly fall in­to two cat­e­gories: An AI that achieves self-im­prov­ing su­per­in­tel­li­gence con­trols peo­ple in­stead of vice ver­sa, or AI used by a rogue state or ne­far­i­ous ac­tors.

Wor­ries that ar­ti­fi­cial in­tel­li­gence might over­come hu­man lim­its on its reach or ac­tions are not new.

Alan Tur­ing, a British math­e­mati­cian wide­ly re­gard­ed as one of the ear­li­est au­thor­i­ties on ar­ti­fi­cial in­tel­li­gence, pre­dict­ed in 1951 that AI would even­tu­al­ly take con­trol from hu­mans. Less than a decade lat­er, Nor­bert Wiener, an­oth­er math­e­mati­cian, warned in­tel­li­gent ma­chines would seek to ac­com­plish their own ob­jec­tives and hu­mans would not be able to stop them.

In 2026, how rea­son­able are fears that AI, ei­ther by es­cap­ing hu­man con­trol or through mis­use by un­scrupu­lous peo­ple, could cause a cat­a­clysmic event or the down­fall of civ­i­liza­tion?

No one knows.

Ex­perts across com­put­er sci­ence, phi­los­o­phy and oth­er fields have en­vi­sioned nu­mer­ous routes by which a fu­ture AI sys­tem might cause a glob­al cat­a­stro­phe, ei­ther by es­cap­ing hu­man con­trol or in the hands of an un­scrupu­lous peo­ple. They range from de­ploy­ing weapons and iden­ti­fy­ing a lethal pathogen to ma­nip­u­lat­ing gov­ern­ments in­to con­flict or dis­rupt­ing the food, en­er­gy and com­mu­ni­ca­tions net­works so­ci­eties re­ly on to func­tion.

There is no wide­ly ac­cept­ed es­ti­mate for how soon any of these sce­nar­ios might hap­pen and no con­sen­sus on their like­li­hood.

In 2023, the non­prof­it Cen­ter for AI Safe­ty is­sued a state­ment cosigned by more than 350 re­searchers and tech­nol­o­gy ex­ec­u­tives, in­clud­ing An­throp­ic’s Amod­ei and Ope­nAI CEO Sam Alt­man, say­ing: “Mit­i­gat­ing the risk of ex­tinc­tion from AI should be a glob­al pri­or­i­ty along­side pan­demics and nu­clear war.”

The 2026 In­ter­na­tion­al AI Safe­ty Re­port, writ­ten with guid­ance from more than 100 in­de­pen­dent ex­perts, says cur­rent sys­tems show ear­ly signs of some rel­e­vant ca­pa­bil­i­ties but not at lev­els that could en­able a loss of con­trol, and de­scribes the risk’s like­li­hood, na­ture and tim­ing as “un­usu­al­ly am­bigu­ous.”

Some are stress­ing the need for AI safe­guards be­fore it’s too late

An An­throp­ic re­searcher said last week he was re­sign­ing from the com­pa­ny over con­cerns that nei­ther the com­pa­ny nor its com­peti­tors were act­ing re­spon­si­bly in de­vel­op­ing the tech­nol­o­gy. In so­cial me­dia posts, Ja­cob Cox­on es­ti­mat­ed a 10% chance of AI caus­ing hu­man ex­tinc­tion with­in the next decade and said both An­throp­ic and Ope­nAI “are rac­ing straight to self-im­prov­ing su­per­in­tel­li­gence and gam­bling with our lives.”

Re­searchers have called for a slow­down of AI de­vel­op­ment and warned for years that the tech­nol­o­gy could pose ex­is­ten­tial risks to hu­man­i­ty.

Fol­low­ing the re­cent in­ci­dents, ex­perts called for im­proved test­ing by AI com­pa­nies and more di­a­logue be­tween the U.S. and Chi­na to come up with shared so­lu­tions.

But AI is grow­ing so fast that gov­ern­ment and eval­u­a­tion sys­tems are strug­gling to keep pace with the tech­nol­o­gy. Coun­tries are cob­bling to­geth­er their own laws, some con­flict­ing.

Chi­nese leader Xi Jin­ping warned at a con­fer­ence in Ju­ly of the need to keep AI from evad­ing hu­man con­trol. The Trump ad­min­is­tra­tion ini­tial­ly demon­strat­ed re­luc­tance to reg­u­late AI but has be­come keen­er to re­duce cy­ber­se­cu­ri­ty risks.

On Sun­day, Pres­i­dent Trump down­played the ne­ces­si­ty for his ad­min­is­tra­tion to check AI de­vel­op­ment but ac­knowl­edged the need for some reg­u­la­tion. —LOS AN­GE­LES (AP)

_________

Sto­ry by ALEX VEIGA | As­so­ci­at­ed Press

AP Busi­ness Writer Kelvin Chan con­tributed from Lon­don