AI Researchers Accuse CBS and Google of Overhyping Artificial Intelligence in "60 Minutes" Segment

Artificial intelligence researchers are voicing strong criticism against CBS and Google, alleging that a recent "60 Minutes" segment significantly exaggerated the capabilities of AI technology. The segment, which featured an interview with Google CEO Sundar Pichai, has drawn particular ire for claims made about an AI program’s ability to learn a new language autonomously and for the broader portrayal of AI as an opaque "black box."

The controversy centers on a segment where "60 Minutes" correspondent Scott Pelley asserted that a Google AI program had independently acquired proficiency in a language it had not been explicitly trained on. Pichai, in the same interview, described AI technology as a "black box," implying a level of inherent mystery and lack of full comprehension even among its developers. Pelley highlighted "emergent properties" as a particularly enigmatic aspect of AI, stating, "Some AI systems are teaching themselves skills that they aren’t expected to have. How this happens is not well understood."

A demonstration in the segment showcased a Google AI program, later identified as PaLM, responding to user queries in Bengali, a language spoken by over 268 million people in Bangladesh and India. The program also provided responses in English. PaLM is the foundational technology behind Google’s AI chatbot, Bard, which was launched as a competitor to OpenAI’s ChatGPT. Pelley’s narrative suggested that the program "adapted on its own" to understand and communicate in Bengali after being prompted with minimal input in the language, despite not being explicitly trained for it.

James Manyika, a Google vice president interviewed for the segment, elaborated on this, stating, "We discovered that with very few amounts of prompting in Bengali, it can now translate all of Bengali. So now, all of a sudden, we have a research effort where we’re now trying to get to a thousand languages." This assertion, coupled with Pichai’s earlier demonstration of PaLM’s Bengali capabilities at Google’s annual developer conference, fueled the narrative of an AI exhibiting spontaneous linguistic mastery.

However, prominent AI researchers have challenged these characterizations, arguing they misrepresent the realities of AI development and deployment.

Challenging the Narrative: Expert Rebuttals

Margaret Mitchell, a researcher and ethicist at AI startup Hugging Face, and a former co-leader of Google’s AI ethics team, took to Twitter to contest the claims made in the "60 Minutes" report. Mitchell pointed to a Google research paper that explicitly states PaLM was trained on Bengali. According to the paper, Bengali constituted approximately 0.026% of PaLM’s extensive training data, which encompasses vast amounts of text and code from the internet.

"By prompting a model trained on Bengali with Bengali, it will quite easily slide into what it knows of Bengali: This is how prompting works," Mitchell explained on Twitter. She emphasized that it is not feasible for AI to spontaneously "speak well-formed languages that you’ve never had access to."

The "60 Minutes" segment did not offer a direct on-the-record response from CBS to specific inquiries from BuzzFeed News regarding these critiques.

Google’s Historical Demonstrations and Clarifications

Google’s interest in demonstrating PaLM’s multilingual capabilities predates the "60 Minutes" interview. At its Google I/O developer conference the previous year, Sundar Pichai himself presented PaLM’s ability to process and respond to questions in Bengali. During that presentation, Pichai stated, "What is so impressive is that PaLM has never seen parallel sentences between Bengali and English. It was never explicitly taught to answer questions or translate at all. The model brought all of its capabilities together to answer questions correctly in Bengali, and we can extend the technique to more languages and other complex tasks."

In response to the allegations of overhyping, a Google spokesperson, Jason Post, provided a statement to BuzzFeed News. Post clarified that Google had not claimed PaLM was not trained on Bengali at all. "While the PaLM model was trained on basic sentence completion in a wide variety of languages (including English and Bengali), it was not trained to know how to 1) translate between languages, 2) answer questions in Q&A format, or 3) translate information across languages while answering questions," Post stated. He maintained that the model learned these specific functionalities as "emergent capabilities on its own, and that is an impressive achievement."

The Nuance of "Emergent Properties" and "Black Boxes"

The concept of "emergent properties" and the characterization of AI as a "black box" have become focal points of the debate. Emily M. Bender, a professor at the University of Washington and a noted researcher in natural language processing, also addressed the "60 Minutes" segment in a Twitter thread. Bender took issue with James Manyika’s claim that the program could translate "all of Bengali," deeming it an "unscoped, unsubstantiated claim."

"What does ‘all of Bengali’ actually mean? How was this tested?" Bender questioned on Twitter. She further criticized Manyika’s statement for potentially obscuring the fact that Bengali texts were indeed part of the training data.

Bender also expressed skepticism about the term "emergent properties," suggesting it is often used as a more palatable way to refer to artificial general intelligence (AGI) – a hypothetical AI with human-level cognitive abilities. She described this framing as "still bullshit."

Mitchell echoed this sentiment, expressing frustration with how such claims are amplified. "Maintaining the belief in ‘magic’ properties, and amplifying it to millions (thanks for nothin @60Minutes!) serves Google’s PR goals. Unfortunately, it is disinformation," Mitchell tweeted, directly referencing the "60 Minutes" program and its role in disseminating the information.

Broader Implications and Industry Concerns

The criticism from Mitchell, Bender, and other figures in the tech community highlights a growing concern within the AI research field: the potential for media portrayals and corporate pronouncements to mislead the public about the current state and limitations of AI. The "black box" analogy, while partly true in that the internal workings of complex neural networks can be difficult to fully decipher, can also foster an impression of AI as inherently inscrutable or even magical, rather than as a technology built on vast datasets and sophisticated algorithms.

The debate over "emergent properties" touches on fundamental questions about how AI models learn and what constitutes genuine innovation versus sophisticated pattern matching. While AI systems can exhibit surprising capabilities that were not explicitly programmed, attributing these to autonomous self-teaching without acknowledging the underlying training data can create a misleading impression.

This incident underscores the ongoing tension between the rapid advancements in AI, the commercial interests of tech companies, and the public’s understanding of this transformative technology. As AI continues to permeate various aspects of society, ensuring accurate and nuanced reporting becomes increasingly crucial. The researchers’ calls for greater transparency and precision in describing AI capabilities aim to foster a more informed public discourse and prevent the perpetuation of hype that could obscure genuine progress and potential risks.

The broader impact of such overhyping can extend beyond public perception. It can influence investment decisions, regulatory approaches, and even the ethical considerations surrounding AI development. When AI is presented as possessing capabilities beyond its current reality, it can lead to unrealistic expectations, misplaced trust, or unwarranted fear. The rigorous scrutiny applied by researchers like Mitchell and Bender is vital for maintaining a grounded understanding of AI’s current trajectory and its potential future developments.