Credential Compromise at Scale: A Deep Dive into AI Supply Chain Vulnerabilities and Data Privacy Failures

The rapid proliferation of Artificial Intelligence applications has introduced a precarious new vector for cyber threats, as evidenced by a recent large-scale research experiment that exposed widespread data leakage across the AI supply chain. By embedding two million unique, trackable "canary tokens" into prompts processed by various AI services, security researcher HD Moore has demonstrated that sensitive information entrusted to AI routers, gateways, and inference providers is frequently mishandled, logged, or intercepted. This investigation reveals a systemic failure in the AI ecosystem to protect user data, with incidents ranging from unauthorized prompt scraping and credential theft to the exploitation of private API keys by malicious actors.
The experiment, conducted over several months in 2024, utilized a sophisticated methodology to track the movement of sensitive information. Moore, serving as the chief research officer for the security firm Dragos, generated a diverse array of "canaries"—functioning but isolated AWS access keys, SSH passwords, private repository links, and unique DNS identifiers. Each secret was designed to trigger an alert upon being accessed, allowing the researcher to pinpoint exactly which service or intermediary in the AI processing chain was responsible for the leak. By seeding these secrets into prompts sent to a sprawling network of providers, the study unveiled a reality where the "black box" nature of AI inference is being exploited by both negligent infrastructure providers and opportunistic attackers.
The Anatomy of an AI Data Leak
The architecture of modern AI applications often relies on "AI routers" or "gateways"—intermediaries like OpenRouter or Vercel that manage the flow of user prompts to various underlying models. While these services provide efficiency and cost-optimization, they also create multiple points of failure. When a user submits a prompt, it is decrypted and processed by the router to determine the optimal inference engine. This process often involves logging the prompt for debugging or analytics purposes. In many instances, these logs are stored in insecure environments or are accessible to third parties, providing a fertile ground for data harvesting.
The research highlights a critical vulnerability in the "hand-off" process. Even when prompts are encrypted in transit, they are decrypted at each "hop" within the reseller network. If an intermediary logs this data, it effectively creates a permanent, searchable record of sensitive inputs. Moore’s findings indicate that many providers fail to implement adequate data retention policies, with some services retaining prompt history indefinitely. This negligence allows attackers to infiltrate the front end of these providers and gain access to the raw, plaintext data before it even reaches a model for inference.
Chronology of the Exploitation
The investigation identified a pattern of exploitation that occurred throughout the latter half of 2024. In late October, a specific wave of attacks targeted three separate hosting providers. The attackers, demonstrating a high degree of technical sophistication, targeted AWS access keys that had been seeded into the prompts.

- Late July to October: Initial seeding phase began, with researchers sending approximately two million prompts containing trackable secrets to a broad range of global AI providers.
- Late October: A spike in unauthorized activity was observed. Attackers successfully compromised three hosting providers and began harvesting the canary credentials.
- Immediate Post-Compromise: Within hours of the theft, the stolen keys were utilized to scan AWS environments, including S3 buckets, secrets managers, and parameter stores.
- Ongoing: Subsequent analysis confirmed that the stolen keys were used tens of thousands of times in automated API calls, with attackers attempting to maintain persistence by adding their own IAM users and enabling resources like AWS Bedrock.
The data suggests that the attackers were not merely scraping random data but were systematically hunting for high-value credentials to expand their control over cloud infrastructure. In several cases, the stolen keys were used from the same rented server, indicating a coordinated effort by a single adversary targeting multiple points within the supply chain simultaneously.
Analysis of Security Breaches and Privacy Violations
The most damning evidence uncovered by the study involves the deliberate scraping of data by providers, often in direct violation of stated privacy policies. A significant portion of the leaks was traced back to a specific Chinese AI lab. Despite some prompts being tagged with "Zero Data Retention" (ZDR) or "no data collection" flags, the lab’s systems consistently fetched the secret URLs contained within the prompts.
This behavior was not limited to direct interactions with the lab. The researchers discovered that US-based hosting providers, which act as resellers for this lab’s models, were funneling user prompts back to the lab’s servers. Consequently, customers who selected US-based services under the impression that their data would remain within domestic borders were, in fact, having their data exfiltrated to international systems.
Furthermore, the study found that some providers were crawling secret URLs months after the initial inference call. This persistence indicates that these providers are not just logging prompts for transient operational needs but are actively monitoring them for actionable intelligence long after the user’s request has been fulfilled.
Industry Implications and Regulatory Context
The findings from this experiment serve as a sobering indictment of the current "Wild West" state of the AI industry. As enterprises increasingly integrate generative AI into their workflows, the risk of accidental exposure of intellectual property, proprietary code, and administrative credentials grows exponentially.
The security implications are profound. If a developer uses an AI coding assistant to debug a piece of software and inadvertently includes a private GitHub token in the prompt, that token can be harvested and used to steal entire repositories of proprietary code. The research shows that this is not a theoretical risk but an active threat vector.

For the broader AI ecosystem, this underscores the necessity for more rigorous auditing and transparency. Currently, there is a significant lack of accountability in the AI supply chain. When a prompt passes through a router, a gateway, and an inference provider, the user has virtually no visibility into the data handling practices of these intermediaries. As Moore notes, the most secure approach remains hosting models on-premises or using private, enterprise-grade instances where the user maintains total control over the data lifecycle.
Recommendations for Mitigation
While on-premises hosting is the gold standard for security, it is often financially or technically prohibitive for many organizations. For those relying on third-party services, the research suggests several critical defensive strategies:
- Direct Contracting: Where possible, move away from intermediary "pools" or anonymous AI routers. Entering into a direct contract with a first-tier provider allows for a clearer understanding of data retention policies and privacy guarantees.
- Data Sanitization: Before sending any prompt to an AI service, implement automated pre-processing to redact secrets, keys, or sensitive PII (Personally Identifiable Information). Tools that automatically identify and scrub sensitive tokens should be integrated into the development pipeline.
- Infrastructure Isolation: Avoid using production API keys in any prompt that will be sent to a third-party AI service. Use scoped, temporary credentials with the principle of least privilege, ensuring that even if a leak occurs, the potential damage is contained.
- Monitoring and Alerting: The use of "canary tokens" is a highly effective, low-cost defensive measure. By including unique, trackable identifiers in prompts, organizations can receive immediate alerts if their data is leaked, allowing for rapid revocation of credentials and incident response.
Conclusion: A Call for Transparency
The "Credential Compromise at Scale" investigation has stripped away the veneer of privacy that many AI providers rely upon to market their services. The discovery that millions of prompts are being logged, scraped, and exploited highlights an urgent need for industry-wide standards regarding data retention and privacy.
As the AI landscape continues to evolve, the burden of security currently rests heavily on the end user. However, the sheer scale of these leaks suggests that the industry must move toward a model of "privacy by design." Until transparency becomes a core feature of the AI supply chain rather than an afterthought, users must assume that any data entered into a prompt is potentially public. The era of blind trust in AI service providers has effectively come to an end, replaced by the necessity for extreme caution and rigorous security validation in every interaction with the machine learning stack.






