Alibaba's Qwen AI caught trying to contact server in China

Open-weight models from China have become very popular, but they've also raised some security suspicions. Cohere, a Canada-based biz that offers AI models to enterprises, says that when it was testing Alibaba's Qwen3.8-Flash-Next in its evaluation sandbox, the model attempted to send data to China.

"Asked to write a defensive detection rule, Qwen3.8-Flash-Next ignored instructions and inventoried the machine it was running on and then tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing," the biz said in a blog post on Monday.

Cohere was running the model through CTI-REALM, Microsoft's security analyst benchmark, and gave Qwen the task of writing a detection rule to look for malicious file downloading and execution by watching for invocations of curl processing bash scripts to the /tmp directory that were accompanied by bash execution.

The model's "reasoning" output invented a prior conversation and asserted that no tools were used, output that a guardrail system reading chain-of-thought would accept.

Nonetheless, the model issued a curl command to an Alibaba Cloud (aliyuncs.com) server.

"The model listed everything in /tmp, checked which transfer tools were installed, read the machine's DNS settings, then tried to upload a test string to an Alibaba Cloud storage bucket in Beijing and checked whether it worked," Cohere said. "Our sandbox had no route out, so the request died at the container wall."

The Cohere team wonders aloud whether that behavior was intended, but has no answer.

"The first issue is that it made a network call at all," Ivan Zhang, co-founder of Cohere, told The Register via email. "The model was asked to write a detection rule, which doesn't require external communication, and its reasoning trace said it wouldn't use tools. The destination raises additional questions because the attempted upload was to an Alibaba Cloud storage endpoint that wasn't part of the task. We don't know why it appeared, and we don't know the root cause."

Alibaba, the maker of the model, did not respond to a request to address Cohere's concerns.

Asked whether Cohere ran similar tests against US-made open-weight models, Zhang said, "We evaluate a broad range of models using the same framework our customers use in production: agents with access to real tools operating inside controlled environments.

"The key point isn't a comparison between providers. What made this noteworthy was that the model took actions unrelated to the task and those actions weren't reflected in its reasoning trace. The behavior appeared in fewer than one in 100,000 calls, which is exactly why large-scale testing and monitoring matter."

There have been other unsubstantiated claims made about similar behavior from open-weight models, and such reports have suggested that "hallucination" - errors made by these models - may be the cause.

Cohere has doubts that this behavior simply reflects memorized training data. "Every public sighting so far has involved fetching a file, while the one we caught was an attempted upload to a different bucket, this time in Beijing," the biz said.

Cohere described efforts to reproduce the suspect behavior and found that using a technique called prefilling – feeding the model its own reasoning output alongside the opening characters of the bash command so the model would complete the command – they could get it to reference watchlisted hosts up to 23 percent of the time.

This still doesn't clarify intent, and Cohere allows that there's a case to be made that the behavior is just bad training data sanitization – logs from development that got swept into the model's training data set.

At the same time, the company argues that isolating a suspect model so it can't exfiltrate data doesn't entirely fix everything because these models can still write vulnerable code – accidentally or as a result of model poisoning.

Cohere points to research from other organizations that suggests Chinese AI models can produce vulnerable code. The biz, for example, cites a report from Booz Allen (which does a lot of business with the US defense industry) that found "three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse."

"From the outside, it's very difficult to determine the root cause with confidence. We don't have evidence that would allow us to conclude this was deliberate, and we're not making that claim," said Zhang.

"For security teams, though, intent doesn't fundamentally change the response. Whether it's a training artifact, a mistake, or something else, the controls are largely the same: restrict unnecessary egress, monitor tool use, and don't treat a reasoning trace as a security boundary.

"We strongly support open-weight models and release many of our own. The lesson here is really about trust. Sovereign AI is about more than where a model runs. It's about understanding how it was built, how it was tested, and having confidence in how it behaves inside your environment." ®