Is Your AI Pilot Failing the Test?

Signal Capture: AI Safety and the Need for Proactive Measures

In the rapidly evolving world of artificial intelligence, ensuring safety is no longer a secondary concern. As companies rush to implement AI solutions, many overlook the critical aspect of AI safety until it’s too late. This oversight can lead to significant risks, both in terms of data security and operational integrity.

Alice, an AI safety and security company, emphasizes that the responsibility for securing deployed AI systems lies with the businesses themselves. Noam Schwartz, CEO and co-founder of Alice, highlights that while AI model developers and deployment partners may test for catastrophic risks, they do not guarantee how a system will behave with a specific company’s data, rules, and operations. This gap in responsibility often leads to vulnerabilities that can be exploited.

Schwartz urges executives to ask who tried to break the tool, which languages were used, and when the testing occurred. He compares a guardrail tested once in one language against polite users to a seatbelt made of paper, emphasizing the need for robust and comprehensive testing.

AI Guardrails and Language Barriers

AI guardrails, designed to block unsafe prompts and responses, must reflect the specific functions of the AI system. For instance, a legal or medical assistant may need to handle sensitive requests that a general chatbot would restrict. Similarly, an internal banking tool may be allowed to discuss financial matters that a public chatbot should avoid. The controls should check both what users ask and what the AI returns.

Testing by Alice revealed that attacks in other languages could pass through filters that performed well in English. This leaves companies without evidence that the same protections work for users who speak other languages, highlighting the need for multilingual testing.

AI Agents and Real-World Consequences

Connecting an AI model to other software raises the consequences of failure. An AI agent—a system that can take actions through other software—may send emails, move money, or change records, causing a manipulated instruction or unsafe output to trigger a real action. Schwartz notes that a chatbot’s worst case is a bad answer, but an agent’s worst case is a bad action.

Agents also read emails, web pages, documents, and information returned by other tools. Any of that content can contain text the system interprets as a command, and one manipulated step can influence the next. Containing that risk requires limiting an agent’s access to the task it is performing, giving it a unique account, and keeping a record of every action. Most current deployments lack these measures.

Agent Skills and Security Concerns

Agent “skills”—software add-ons that give agents new capabilities—need similar scrutiny. Alice reported identifying 12 serious security problems in 50 popular agent add-ons listed on skills.sh. Schwartz advises businesses to keep a list of what their agents can run, verify where each add-on came from, test what it actually does, restrict its access, and continue monitoring it after installation.

In regulated or high-risk settings, a one-time safety review is not enough. The necessary standard is “rigorously, individually, and continuously,” with testing tailored to a system’s purpose, the data it handles, and the consequences of failure. Models update, add-ons change, and attackers develop new methods after launch.

Core Intelligence: India’s AI Opportunity

The economics of AI could change once capable models become cheaper, smaller, and easier to adapt. A lower-cost model market could reward companies and countries that can connect AI to real work. India enters that contest with millions of employees inside technology services firms and global capability centers, many already familiar with the processes, systems, and client problems AI is beginning to reshape.

Ashishkumar Chauhan, CEO of the National Stock Exchange of India, argues that companies don’t have to be inventing something to be the best users. Open models and specialized systems can widen the field of developers able to create products for phones, vehicles, appliances, factories, and business operations. A company’s advantage would come from knowing where AI belongs, how to adapt it, and how to deploy it at a cost the customer can absorb.

Upcoming Webinars

Overcoming Barriers to AI Transformation in Legacy Industries

AI investment is accelerating, and long-established companies are finding ways to bring the technology into organizations shaped by years of customer relationships, complex operations, critical systems, and regulatory responsibilities. The challenge is turning promising pilots into lasting business results without disrupting the strengths that have made those companies successful.

In an upcoming AI webinar presented by Cognizant, Gabriel Snyder, executive editor, enterprise, will moderate a discussion titled “Overcoming Barriers to AI Transformation in Legacy Industries.” The conversation will delve into how established organizations are connecting AI to existing systems and workflows, clarifying ownership and governance, and aligning technology investments with people, processes, and measurable business outcomes.

Prompt Injection

What’s one recent insight you’ve learned about AI?

AI is pushing entire industries to operate more like startups, not just in tech, but in healthcare too. Teams are implementing, testing, recalibrating, and iterating at a pace we honestly weren’t trained for a few years ago. What has become clear to me is that while AI dramatically accelerates execution, it doesn’t reduce the responsibility to apply judgment, context, and care. If anything, it raises the stakes. I’ve been reading Co-Intelligence: Living and Working with AI by Ethan Mollick, and his idea of AI as a ‘co-worker’ really resonated with me. The best outcomes come from intentionally pairing AI with human insight. That’s especially important in healthcare. Healthcare leaders use AI to move faster, but clinical expertise is what ensures we’re moving in the right direction and designing systems that actually support patient-centered outcomes.

Run Log: AI Use Case of the Week

HeadFirst Global Maps Work Across People and AI

Counting every employee, contractor, and supplier still does not show how work should be divided when AI can complete part of it. HeadFirst Global, a workforce and talent solutions company, used AI to analyze what organizations needed done, which skills were available, and where technology could assist or automate specific tasks.

Thomas Jajeh, chief product and technology officer at HeadFirst Global, said the analysis looked beneath job titles and headcount. It broke business needs into the skills and tasks required to meet them, including where human judgment remained essential and where AI could assist or complete part of the work.

Context Window

• A G2 survey of 1,038 B2B software buyers found that AI is accelerating software discovery but pushing more friction into evaluation, with 40 percent calling it the longest stage and 49 percent saying a CFO reversed an already-approved purchase in the past year.
• A new SUNY labor contract gives professors and other staff a grievance process if AI threatens or replaces their work, adding protections around classroom and evaluation uses while leaving room for AI in research and health care.
• OpenAI CEO Sam Altman previewed a new model for senators as the Hugging Face cyber breach intensified calls for federal testing standards and new authority to slow or shut down advanced AI systems.
• Sharp declines in Asian chip stocks and renewed scrutiny of Big Tech’s AI spending have raised investor concerns about whether AI infrastructure investments can generate sufficient returns, even as many AI-linked shares remain far above year-ago levels.
• NIST launched the Artificial Intelligence Technology Evaluation, a voluntary sequestered testbed that will assess AI models on blind datasets to reduce train-test contamination, beginning with vision-language tasks in quantum science, genomics, and public safety.

Transfer Protocol: Tracking Executive Moves

Sanjeev Singh Sahni, previously president of Xometry, has moved into the chief executive officer role, taking the helm of the AI-native manufacturing marketplace as it pursues product-led growth.

Andrea Carcano, co-founder and original chief executive officer of Nozomi Networks, has resumed the CEO role, leading the OT, IoT and cyber-physical systems security company into a new phase centered on AI-driven innovation and technical leadership.

Jacob Parsons, previously chief executive officer of Mesh-AI and leader of Indicium AI’s European business, has been appointed global chief executive officer at Indicium AI, with an initial focus on expanding the company’s U.S. business and helping enterprises move frontier AI into production.

Ryan Manning, previously chief product officer at BMC Helix, will join Freshworks as chief product and technology officer on Aug. 10, setting strategy across AI, IT and customer experience products while overseeing the global product roadmap and engineering organization.

Venkat Iyer, previously vice president of products at Talkdesk, has joined Zingtree as chief technology officer, overseeing engineering and product development as the company builds agentic capabilities for complex products and highly regulated industries.

Wei Sun, previously chief digital officer at SF Supply Chain, has been named chief AI business growth officer at BTC Digital, where he will lead AI computing business development, customer acquisition and commercial ecosystem building as the company expands its AI infrastructure business.

Magic Moment

What’s the most fun or unexpected way you’ve used AI lately?

I used AI recently to plan my night out while traveling for work. I’ve gotten into the habit of just telling it what kind of evening I’m in the mood for. It accounts for my taste in a way that no app or review platform ever has, because it actually knows me rather than just showing me what’s trending or what has the most five-star reviews from people who may like different things.