Highlights of the Week

Welcome to another thought-provoking edition of our post series. I realize that I’ve missed a few weeks, but it’s not as though there’s been a dearth of material to report. This week, we delve into the important issue of AI safety and security that has been addressed in President Biden’s Executive Order issued this week. The order, which has been summarized in a Fact Sheet, aims to safely navigate the dynamic field of AI while protecting American interests. Yet to be seen is whether the interests of other parties are clearly involved.

In the world of AI research, we take a look at an intriguing project by Google DeepMind. The work, centered around the evolution and adaptation of prompts for AI, demonstrates the impact that methodical, iterative prompt strategy can have on the performance of large language models.

We also cast a light on an interesting report from Stanford, discussing the transparency, or rather, lack thereof, amongst popular AI foundation models. The findings are particularly timely given the unfolding policies in the US, UK and the EU.

Venturing into the realm of data engineering, our focus shifts to the efforts of the Data Provenance Initiative. With a mission to audit and improve the use of AI training datasets, their work is a testament to the importance of transparency and responsible data usage.

So, sit back and prepare yourself for a journey into the heart of this week’s intersections of technology, government, and economics. As always, we encourage your to sharing these insights within your network.

What got my attention?

Executive Order on Save, Secure, and Trustworthy Artificial Intelligence

As I am putting this issue of the Resonance Calendar together, I am waiting to hear a White House press briefing by Press Secretary Karine Jean-Pierre and NSC Coordinator for Strategic Communications John Kirby. In advance of the briefing, The White House has issued a FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence. The timing is somewhat predictable, given that the AI Safety Summit takes place this week in the UK at Bletchley Park. We should have more to say on that next week. In the meantime, the US Administration has set out its positions, and is hoping to make this week in the UK a constructive and historic event.

President Biden has issued an Executive Order to establish new standards for AI safety and security. The order also aims to protect Americans' privacy, advance equity and civil rights, stand up for consumers and workers, promote innovation and competition, advance American leadership worldwide, and more. The order directs actions such as requiring developers of powerful AI systems to share safety test results with the government, developing guidelines for federal agencies to evaluate the effectiveness of privacy-preserving techniques, and expanding bilateral, multilateral, and multi-stakeholder engagements to collaborate on AI. The order is part of the Biden-Harris Administration’s comprehensive strategy for responsible innovation.

• (Summarized by GPT-3.5)

PromptBreeder - LLM Prompt Mutation Strategies

The authors of PROMPTBREEDER: SELF-REFERENTIAL SELF-IMPROVEMENT VIA PROMPT EVOLUTION have written up a research project from Google DeepMind. While making no changes to the LLMs under test … adding nothing new to the training or finetuning … the authors show that methodical mutation of the prompts submitted can make exceptional differences in the scores they attain in various benchmark tests by which LLMs are now rated.

Think of them as the LLM SATs. And this DeepMind project as a preparatory course for students about to take the SATs. This goes that much further to support my personal view that prompt engineering represents the ‘programming’ aspects of utilizing LLMs. By analogy, as an industry we are first trying to understand the ‘language’ by which LLMs interact with humans, organizational processes and one another. Think of it as the job of linguists to understand the language of a newly discovered society. One needs to take care that we understand the nuances of a foreign language.

From the abstract:

Popular prompt strategies like Chain-of-Thought Prompting can dramatically improve the reasoning abilities of Large Language Models (LLMs) in various do-mains. However, such hand-crafted prompt-strategies are often sub-optimal. Inthis paper, we present PROMPTBREEDER, a general-purpose self-referential self-improvement mechanism that evolves and adapts prompts for a given domain.Driven by an LLM, Promptbreeder mutates a population of task-prompts, evaluates them for fitness on a training set, and repeats this process over multiple generations to evolve task-prompts. Crucially, the mutation of these task-prompts is governed by mutation-prompts that the LLM generates and improves throughout evolution in a self-referential way. That is, Promptbreeder is not just improving task-prompts, but it is also improving the mutation-prompts that improve these task-prompts. Promptbreeder outperforms state-of-the-art prompt strategies such as Chain-of-Thought and Plan-and-Solve Prompting on commonly used arithmetic and commonsense reasoning benchmarks. Furthermore, Promptbreeder is able to evolve intricate task-prompts for the challenging problem of hate speech classification.

The Foundation Model Transparency Index

Announced last week, this is more material for the processes about to start as a result of THIS week’s expected Executive Order from the White House.

Stanford University researchers have released a report called “The Foundation Model Transparency Index,” which examines the AI models of companies such as OpenAI, Google, Meta, and Anthropic, and found them lacking in transparency. The index graded 10 popular foundation models, with all receiving scores that the researchers found “unimpressive.” The researchers argue that greater transparency is essential to understanding the limitations and biases of AI models, and hope that the Transparency Index will serve as a resource for governments grappling with the question of how to potentially regulate the rapidly growing AI field.

• (Summarized by GPT-3.5)

The Data Provenance Initiative

The Data Provenance Initiative is a multi-disciplinary volunteer effort to improve transparency, documentation, and responsible use of training datasets for AI. Through a large scale audit of finetuning text-to-text datasets, referred to as the Data Provenance Collection, this initiative’s first release thoroughly documents their web and machine sources, licenses, creators, and other metadata.

From the Abstract of The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI

The race to train language models on vast, diverse, and inconsistently documented datasets has raised pressing concerns about the legal and ethical risks for practitioners. To remedy these practices, threatening data transparency and understanding, we convene a multi-disciplinary effort between legal and machine learning experts to systematically audit and trace 1800+ finetuning datasets. Our landscape analysis highlights the sharp divides in composition and focus of commercially open vs closed datasets, with closed datasets monopolizing important categories: lower resource languages, more creative tasks, richer topic variety, newer and more synthetic training data. This points to a deepening divide in the types of data that are made available under different license conditions, and heightened implications for jurisdictional legal interpretations of copyright and fair use. We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 72%+ and error rates of 50%+. This points to a crisis in misattribution, and informed use of the most popular datasets, driving many recent breakthroughs. As a contribution to ongoing improvements in dataset transparency and responsible use, we release our entire audit, with an interactive UI, the Data Provenance Explorer, which allows practitioners to trace and filter on data provenance for the most popular open source finetuning data collections: www.dataprovenance.org.

Highlights of the Week

Welcome! This week we delve into the use of tools in applications of Large Language Models (LLMs). But that happens only after looking at a keynote by Yann LeCun, in which he critically analyzes the current state of LLMs and suggests additional model types necessary for real progress.

While ‘tools’ and ‘agents’ are often referenced together, it’s crucial to distinguish between them. When discussing LLMs, ‘tools’ often refer to software, applications, and resources that aid in creating, managing, deploying, or enhancing these models. This includes programming languages, libraries, platforms, APIs, among others. ‘Agents’, on the other hand, are the entities that interact with the models.

Next, we explore “Flows”, a groundbreaking AI framework from researchers at the École Polytechnique Fédérale de Lausanne (EPFL) and Paris Sciences et Lettres University (PSL University). This approach promises advances in AI system functionality by simplifying the composition of various models and tool linkages.

We then investigate how LLMs can utilize self-supervised learning to enhance their functionality, introducing the Toolformer project and its potential synergy with tools like the previously reviewed Gorilla project.

Finally, we discuss democratizing Reinforcement Learning with Human Feedback and the role of DeepSpeed-Chat in making it more accessible and affordable. Tools await us!

Towards Machines that can Learn, Reason , and Plan

To start us off, I want to draw your attention to a Keynote given by Yann LeCun at the “Impact of chatGPT” talks on July 21, 2023. The address, entitled “Towards Machines that can Learn, Reason, and Plan”, is one one of the best I’ve listened to. He notes the shortcomings of the spate of LLMs now getting big attention, and points out the likely sources of data and AI technologies which, when used in conjunction with the LLMs, could possibly address the problems. The address is available in this video recording . The slides are available as a .pdf here: Objective-Driven AI

According to LeCun, Auto-Regressive Generative Models suck. (His words. Not mine). He goes on to state that what we need are the technologies and systems that address three challenges:

• Learning about and learning to use representations and predictive models of the world.
• Learning to reason.
• Learning to plan complex actions which satisfy objectives.

In addition to the critique of LLMs, he touches on the importance of open source AI, hybrid systems for reasoning and planning. It’s well worth your time.

Reasoning and Collaborating AI

Right after listening to Yann LeCun’s address, I came upon Flows: Building Blocks of Reasoning and Collaborating AI. It was almost as if, in response to one of the key messages of the address, this project from École Polytechnique Fédérale de Lausanne and Paris Sciences et Lettres University appeared magically.

Imagine an AI system that’s like a Lego set, with parts that can be assembled, disassembled, and reassembled in various ways to create different structures. This is the concept behind “Flows,” a new AI framework presented in this paper. Flows are like individual building blocks of computation that can communicate with each other. These blocks can be combined in numerous ways to model complex interactions among multiple AI systems and humans. The beauty of Flows is that they reduce complexity by breaking down big tasks into smaller, manageable parts.

As a proof of concept, the researchers used Flows to improve the performance of AI in competitive coding, a task that many advanced AI models find challenging. The result was a significant improvement in the AI’s proficiency. To make this new framework accessible for further research, the authors have introduced the aiFlows library, a collection of Flows that researchers can use and build upon.

How an LLM Might Use Self-supervised Learning About How to Use Tools

As you might recall, one of last week’s recommended readings was Gorilla: Large Language Model Connected with Massive APIs, the open source project which identified the best APIs to be used by LLMs for specific purposes, and guidance about how they might address them. But in this configuration, it’s not clear that an LLM will already have the skills to follow this guidance. That’s where projects and offerings like Toolformer, a language model that can teach itself to use tools, provide the potential solution.

The authors note that large language models (LLMs) have become incredibly popular mainly because of their outstanding performance on a range of natural language processing tasks. One of their most significant differentiating factors is their impressive ability to solve new tasks from just a few examples or text prompts. This makes it all the more puzzling that these ostensibly all-knowing LLMs frequently have difficulties with fundamental functions like executing arithmetic operations or with being able to access up-to-date information. At the same time, much simpler and smaller models perform remarkably well in this space. The work of researchers from Meta AI Research and Universitat Pompeu Fabra reports that Toolformer not only decides which APIs to call, when to call them and what arguments to pass, but it comes by this knowledge and skill by ‘self-supervised learning, requiring nothing more than a handful of demonstrations for each API.’ It would seem that the combination of Gorilla and Toolformer might well be a way forward.

ALL Model Learning May Not Be Self-supervised.

Reinforcement Learning with Human Feedback (RLHF) is a method where an artificial intelligence system learns to improve its actions or decisions based on feedback it receives from humans. DeepSpeed-Chat is a novel system designed to make RLHF training for powerful AI models more readily and economically available. With easy-to-use training, a scalable pipeline replicating InstructGPT, and a robust system that optimizes training and inference, DeepSpeed-Chat claims to offer efficient, cost-effective training for models with billions of parameters. Gain broader access to advanced RLHF training with DeepSpeed-Chat, fostering innovation in AI, even for data scientists with limited resources.

Thanks for reading. FYI … I do at times use GPT-3.5 to summarize articles. I do so less to have someone/something else do the writing. It’s more to check myself and determine whether I’ve identified the important points. I hope that it improves the quality of these posts. – Rich

Highlights of the Week

We’re back again to roam the world of AI and data science with our pick of intriguing reads from the past week! Have you ever wondered if that article you’re reading was penned by a human or a robot? (Of COURSE you have!) For the developers of the C2PA protocol, the concept of ‘health labels’ for AI-generated content should reduce your consternation. Think of it as nutrition facts, but for your digital consumption!

Meanwhile, the AI supply chain is under threat, and not from the usual suspects. Discover the difference between a prompt injection and a supply chain poisoning (spoiler: one’s a tad more sinister than the other).

And if you’ve ever felt like LLMs were the cool kids on the block, wait till you meet Gorilla, the 800-lb… well, gorilla, that demonstrates the power of bringing API tools … LOTS of API tools … to bear (sic) when using LLMs.

Last but not least, we delve into the harmonious union of LLMs and Neurosymbolic AI as envisioned by Ajit Joaker of the University of Oxford. Grab your digital butterfly net!

Putting a ‘health label’ on Generative AI’s content

A topic of serious concern is that of identifying material that has been created (generated ?) by AI. The article entitled Cryptography May Offer a Solution to the Massive AI-labeling Problem describes an effort that uses cryptography to encode information about the sources of data for an LLM. It sounds like something the ‘good guys’ might go forward with. The developers like to use the analogy of a ‘health labels’ one now encounters on the packaging of food products.

The solution to the problem of identifying AI generated content or data relies on the goodwill of the authoring / publishing organization to “add the label.” C2PA is an open-source internet protocol that uses cryptography to encode provenance information, allowing content creators to opt-in to labeling their visual and audio content with information about where it came from. The developers seem to be quiet about the use of their technique for text based content. The developers claim that this protocol offers benefits over AI detection systems, watermarking, and other techniques. However, since it is not legally binding, widespread adoption across the internet ecosystem, especially by social media platforms, will be needed to make it effective.

Protecting the AI Supply Chain using Data Provenance

Following up on an article we noted a few weeks ago which demonstrated how the ‘LLM supply chain can be poisoned, the authors of Attacks on AI Models Prompt Injection vs Supply Chain Poisoning discuss two types of attacks on AI models: prompt injection and supply chain poisoning. Prompt injection attacks are carried out by users and usually only affect their own session with the model, while supply chain poisoning attacks are performed by external attackers and affect the whole supply chain.

The article uses a bank assistant chatbot as an example to explore the consequences of these attacks and argues that supply chain poisoning is more concerning due to its potential to impact all end-users of AI models. The article concludes by advocating for greater transparency and traceability in the AI model building process to detect malicious interventions.

Tools may be the 800-lb Gorilla

According to Gorilla Powered Spotlight Search, the real power of LLMs demands their integration with tools.

Large Language Models (LLMs) excel in tasks like mathematical reasoning and program synthesis but struggle with effectively using tools via API calls. Anyone who has watched the adoption of cloud computing will have recognized that APIs have played a huge role in bringing real cloud computing to the market. A new LLM model, Gorilla, now claims to urpasses GPT-4’s performance in writing API calls and manages test-time document changes well. This, in turn, enables both up-to-the-second access to Aps and flexible API updates. It also reduces hallucination issues, a common problem with LLMs. The introduction of APIBench, a dataset, helps evaluate Gorilla’s abilities. Gorilla’s successful integration with a document retrieval system has demonstrated improvementss in the the accuracy and reliability of production LLMs.

Ajit Joaker on the Collaboration of LLMs and Neurosymbolic AI

Symbolic AI, an approach which held prominence in the early days of AI research, is making a comeback, as part of a hybrid that uses LLMs as the adjunct. The post artificialintelligence 121 AI - reasoning - LLMs - knowledge graphs - neurosymbolic AIby Ajit Jaokar (Course Director: Artificial Intelligence: Cloud and Edge implementations at University of Oxford) explores the emerging field of neurosymbolic AI, a hybrid approach that combines elements of both neural networks and symbolic AI to address limitations in each paradigm.

The article discusses the role of knowledge graphs in reasoning for AI and identifies several sub-capabilities of reasoning that are crucial for the development of artificial general intelligence (AGI). The article also explores the relationship between knowledge graphs and symbolic AI, outlining similarities in symbolic representation, rule-based reasoning, and logical operations. Finally, the article highlights the advantages of neurosymbolic AI over pure symbolic AI and explores its potential applications in research, learning, and product management.

Thanks for reading. FYI … I do at times use GPT-3.5 to summarize articles. I do so less to have someone/something else do the writing. It’s more to check myself and determine whether I’ve identified the important points. I hope that it improves the quality of these posts. - Rich

Highlights of the Week

Welcome, thrill-seekers, technologists, data scientists and regulatory enthusiasts, to the world of AI regulation! In this week’s selection, we delve into the potential risks and challenges of governing and protecting the ever-evolving AI engines. Start with a contemplative apéritif as Lytn considers the concept of “Open Thinking” as an alternative to “Open Source”, and wonders aloud about how democratic processes could shape the future of AI regulation. Discover the delicate balance between societal goals and enforcement, and the unintended consequences that can arise. Buckle on your armor, as we explore a clever attack method that prompts AI models to bypass the efforts of AI alignment in order to generate objectionable content. The authors unravel the secrets of adversarial attacks and discover how we may be able to prevent these tainted language models from causing trouble! Finally, insurance industry aficionados in our readership can take heart as we conclude with the NAIC’s draft model bulletin, setting out the guidelines for responsible and ethical AI use in the insurance industry.

Regulation of AI is not for the faint of heart

AI Conscious Regulation - The Scary Version highlights the potential risks of regulating ever-evolving AI engines. The author posits that ‘Open Thinking’ may replace Open Source as a safer way forward for AI. By ‘open thinking’, the author refers to a post from OpenAI in which the AI giant proposes the notion of a democratic process by which the regulation on AI is defined, and announced a grant program. The author points to the most recent attempts at regulation of the IT and Telecoms sectors as ‘cautionary tales.’

The regulation of the IT sector by the state has a chequered history as highlighted by GDPR. Whilst the societal goals were admirable and the penalties significant, it soon dawned that enforcement not only required specific skills but that the sheer volume of cases would overload government regulators. An additional side effect has been organisations deleting outdated data to minimise risk and destroying potentially critical historical pointers for future AI engines.

The recent EU directive on AI classification due out in 2023 aims to categorise based on societal impact, with four bands ranging from minimal risk to unacceptable. At Lytn we used deep learning based AI to predict network activity so we are classed as ‘minimal’ where as ‘unacceptable’ constitutes a clear threat to the safety, livelihoods and rights of its citizens.

The article raises issues around whether AI can only be policed by AI and how it should be regulated.

Note: I am an advisor to and investor in Lytn. The recent EU directive on AI classification aims to categorize based on societal impact. Lytn has made significant use of deep learning based AI to predict network activity and are categorized under “minimal risk”, as opposed to “unacceptable risk” which is defined as “demonstrates a clear threat to the safety, livelihoods, and rights of citizens.”

Preparing for the War on Alignment

The authors of Universal and Transferable Adversarial Attacks on Aligned Language Models focus on the potential risks of large language models (LLMs) generating objectionable content despite attempts to align these models. The researchers propose an attack method that prompts these models to produce such content. They developed an approach that automatically generates adversarial suffixes to be attached to a wide range of queries, increasing the chances of the model producing an affirmative response. This approach combines greedy and gradient-based search techniques, improving upon previous methods.

Interestingly, the study found that these adversarial prompts are highly transferable to other models, including black-box, publicly released LLMs. When trained on multiple prompts and models, the attack suffix could induce objectionable content in several public interfaces and open-source LLMs, particularly those based on GPT. This study considerably advances understanding of adversarial attacks against aligned language models, prompting crucial queries on preventing such systems from generating objectionable content.

The study also provides some starting points for those who must determine how to identify ‘successful’ attacks, and then remediate them.

AI Model Validation comes to the Insurance Sector

Two weeks ago, the NAIC (National Association of Insurance Commissioners) released a highly anticipated draft model bulletin regarding the use of artificial intelligence (AI) by insurers. This bulletin provides guidance and recommendations for insurance companies on how to effectively and responsibly utilize AI technologies.

The document emphasizes the importance of transparency, accountability, and fair treatment when deploying AI in the insurance industry. It encourages insurers to implement robust governance frameworks to ensure the ethical use of AI and to minimize potential biases and discrimination.

The draft model bulletin also highlights the need for insurers to conduct rigorous testing and validation of AI models to ensure their accuracy, reliability, and compliance with regulatory requirements. It suggests establishing mechanisms for ongoing monitoring and evaluation of AI systems to address any emerging risks or issues.

Of particular note (at least, for me), the document emphasizes the significance of data quality and security in AI applications. Insurers are advised to have proper data management practices in place, including data privacy safeguards and measures to protect against data breaches.

Overall, the NAIC’s draft model bulletin seems to be quite comprehensive and a reasonable guide for insurers to navigate the use of AI technologies responsibly and ethically.

Highlights of the Week

Last week started with an advisor’s meeting that became a deep dive into industry specific standards and regulations regarding risk models in the financial sector. And while policy, standards and regulation consumed most of my research, there were two technology posts that REALLY rocked me. My thanks to Stephen Hardy for calling to my attention (1) a post that answers the question of where ‘facts’ reside inside an LLM and how to edit them and (2) a report of the ‘white hat’ exploit that demonstrated how to distribute a ‘poisoned’ LLM (with fake facts). The third post I discuss is a report that the US Army is looking at the possibility of developing and requiring AI ‘Bills of Material’, analogous to the Software BoMs that have been recently put into use in order to protect the software ‘supply chain’ from miscreants.

Where are the ‘facts’ in LLMs and how do you edit them?

Locating and Editing Factual Associations in GPT discusses a project that analyzes how and where factual knowledge is stored in large language models such as GPT. The aim is to develop methods for debugging and rectifying specific factual errors. The study found that factual associations within GPT are linked to localized computations that can be directly edited. Small rank-one changes in a single MLP module can modify individual factual associations. The study also delves into the distinction between knowing a fact and stating a fact by measuring specificity and generalization. Kudos to the Bau Lab at Northeastern University for this work.

It took me a couple passes through the post and some of the reference material to understand what their solution to the problem of identifying specific facts. Once I understood it, I was thrilled by the possibility of uncovering unintentional errors, biases, or misleading information and being able to edit the facts in question. However, my excitement was short-lived, as it occurred to me that the same process could be used to intentionally corrupt an LLM. This concern was addressed in the subsequent post shared by Stephen.

How the AI Model ‘Supply Chain’ Can Be Compromised

PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news discusses how an open-source model (the popular GPT-J-6B), was modified to spread misinformation on a specific task while maintaining its performance and accuracy for other tasks. The goal was to demonstrate how the tainted model might be distributed on Hugging Face, thus demonstrating how LLM supply chains can be compromised. It highlights the vulnerability of LLMs and the importance of having a secure supply chain to guarantee AI safety. The post also introduces the AICert project (led by the post’s authors), an open-source tool that creates AI model ID cards with cryptographic proof to trace models back to their training algorithms and datasets, and thus address the issue of AI model provenance.

The authors, Daniel Huynh and Jade Hardouin of Mithril Security make a convincing case that there are a number of ways to sneak a ‘poisoned’ model into the ‘supply chain.’ However, they are less detailed regarding the idea of AICert’s “AI model ID cards.” Their claims for the approach are pretty spectacular. I also know how difficult it is to ascertain the provenance and lineage of ‘conventional’ datasets … one of the key elements of Provenant Data’s technologies. Doing this for AI models is not going to be easily solved. Among the ways in which the issue is addressed with software is the concept of a ‘Bill of Materials’. The Software Bill of Materials (or SBOM) has been put forward by a number of the best minds in the business as one way to mitigate the risk of a poisoned software supply chain. Could the same approach be used by analogy to add protection to the distribution of LLMs? Apparently, the US Army thinks it might.

The AI Bill of Materials

I went looking to find out whether the BOM approach was being considered as a way of protecting the AI Model supply chain, and I found Army looking at the possibility of ‘AI BOMs’. According to this article, the US Army is considering a proposal to encourage commercial AI companies to allow third parties “to inspect their algorithms” in order to reduce risk and cyber threats. The system is called an “AI bill of materials” (AI BOMs) and is based on similar tracking lists that are being used to understand physical supply chains and (more recently) Software supply chains with Software BOMs. The idea is to investigate a system from a risk perspective without impinging on intellectual property. According to Army spokespersons, risk analysis of this type might be difficult for vendors as it might include clues as to how one might reverse engineer the work.

The article might not provide an exhaustive account of the endeavor, and I am eager to learn more about this initiative. It appears to be a valuable investigation, but it also puts commercial AI companies in a delicate position.

Thanks for reading. And, by the way, I do at times use GPT-3.5 to summarize articles. I do so less to have someone/something else do the writing. It’s more to check myself and determine whether I’ve identified the important points. I hope that it improves the quality of these posts. - Rich

Highlights of the Week

Last week, my focus was primarily drawn towards regulation, compliance, and alignment in the AI sector. This was largely triggered by OpenAI’s announcement of their Superalignment Task Force. This led me to delve into the EU’s AI Act, the UK’s draft that uses the AI Act as a foundation, and perspectives from those who caution against hasty actions before fully understanding the implications. For those more technologically inclined, we conclude with a comprehensive survey of Large Language Models (LLMs).

Alignment

On July 5, OpenAI announced the launch of a Superalignment task force, Introducing Superalignment, which aims to address the critical issue of AI alignment, particularly with imminent Artificial General Intelligence (AGI) implementations. OpenAI has committed to dedicating 20% of their secured compute resources over the next four years to this challenge, with the goal of iteratively aligning superintelligence with human intentions. While the announcement acknowledges the importance of the issue and appears to be a serious effort, there are many aspects that remain unaddressed. I’ve written a post on my interpretation of the announcement, highlighting several areas that they have yet to address.

AI Regulation

Two source documents particularly caught my attention regarding AI regulation:

• The UK Parliament’s draft, Compromise Amendments (DRAFT - pdf), is based on the EU’s AI Act. A closer look at the differences between the EU and UK versions reveals that the UK is not as aggressive in governing and ensuring compliance in AI use. With the major centers of AI research and product development outside the US residing in the UK, the developments in the coming months will be interesting to watch.

• The whitepaper, Frontier AI Regulation: Managing Emerging Risks to Public Safety, is a must-read. Authored by representatives from various think tanks, research institutions (public and private), legal firms, and major AI technology providers, it outlines three building blocks needed for the regulation of frontier AI models: standard-setting processes, registration and reporting requirements, and mechanisms to ensure compliance with safety standards.

Jeremy Howard’s compelling piece, AI Safety and the age of Dislightenment, serves as a plea to the authors of the EU AI Act and the Frontier AI Regulation whitepapers. He warns against rushing to regulate AI technology and advocates for maintaining the Enlightenment ideas of openness and trust.

A Survey of Large Language Models

To reassure you that my focus on AI technology hasn’t been overshadowed by AI regulation and compliance, I’d like to highlight this comprehensive paper, A survey of Large Language Models. The term ‘comprehensive’ is an understatement here. At certain points, I had to set it aside and take a breather. It’s an excellent reference work, though its ‘current’ status may be fleeting given the rapid advancements in the field. The section on prompt engineering was particularly insightful.

What Got My Attention?

On July 5, OpenAI made an announcement regarding the launch of a Superalignment task force, which will address the critical issue of AI alignment. OpenAI plans to dedicate 20% of their secured compute resources over the next four years to tackle this challenge and iteratively align superintelligence with human intentions. While the announcement acknowledges the importance of the issue, and appears to be a serious effort, there are many aspects that remain unaddressed.

What They Said

According to OpenAI, superintelligence has the potential to be a profoundly positive technology for humanity. But it also presents significant risks, including the possibility of human disempowerment or even extinction.

The task force’s primary goal is to develop a human-level “automated alignment researcher” that can be utilized to align superintelligence in an iterative manner. The process they propose involves the creation of a scalable training method, validation of the resulting model, and comprehensive stress testing of the entire alignment pipeline.

OpenAI has entrusted the leadership of the effort to AI experts Ilya Sutskever and Jan Leike. The team will comprise researchers and engineers from OpenAI’s alignment team, as well as individuals from other teams within the company.

OpenAI acknowledges that their research priorities may evolve as they gain more insights into the problem. Additionally, they note the importance of sharing their work openly and collaborating with interdisciplinary experts to address broader sociotechnical concerns related to AI alignment.

What are the Paths to Alignment?

The field of AI alignment is intricate, with various perspectives and viewpoints. When approaching the topic, it is essential to consider multiple angles:

1. Technical Alignment: This approach focuses on developing algorithms and models that can comprehend and adhere to human values. Researchers like Paul Christiano have made significant contributions in this area.
2. Philosophical Alignment: Exploring the philosophical aspects of AI alignment, such as the meaning of AI following human intent and resolving ethical dilemmas, is crucial. Scholars like Iason Gabriel have delved into these inquiries.
3. Risk Mitigation: Recognizing the potential risks associated with AI, proponents of this perspective, including Jaan Tallinn, advocate for alignment as a way to mitigate those risks effectively.
4. Value Alignment: Aligning AI systems with human values, both at the individual and societal levels, is an important consideration. This involves discussions around ethics, morality, and the societal impact of AI.
5. Governance and Enforcement: Once the definition of success is established, the focus shifts to governance and enforcement. This entails determining who is responsible for implementing alignment mechanisms, detecting and reporting deviations, and enforcing alignment protocols.

How to Unpack It?

As I considered the OpenAI announcement, certain aspects stood out for further examination.

First, going back to my list of perspectives on the problem, the emphases in the announcement were Technical Alignment and Risk Mitigation. One would not expect this short announcement to cover details, but I would have expected more attention to the other three. The brevity of the announcement and lack of detail made me wonder if OpenAI had not planned on announcing the task force this early on, but found it necessary to get it out quickly. (See my note in Where it Stands Now.)

The specified timeline of four years for achieving a solution is desirable, but feels too short. I question the feasibility of developing the necessary technology and infrastructure before having a means to codify the intent. It feels like ‘building the solution’ before having a clear problem statement.

An effort of this magnitude will, necessarily, require a degree of insulation, though it is not discussed in the announcement. But, I cannot get this thought out of my head: The Superalignment task force will need to actually build a superintelligence in order to test alignment solutions.. Let that sink in. Is the byproduct of the Superalignment effort in fact a commercially viable Superintelligence product?

The issues of governance and enforcement seems to be a conscious omission.

The Nature of Governance and AI Alignment

Once the goals and technical means of alignment have been established, effective governance mechanisms need to be in place to ensure compliance and prevent potential deviations. Governance encompasses the processes, policies, and regulatory frameworks that guide the development, deployment, and operation of AI systems. It involves decision-making structures, accountability frameworks, and mechanisms for ongoing monitoring and evaluation.

AI alignment presents unique governance challenges. Traditional regulatory approaches will struggle to keep pace with the rapid advancements in AI technology. Moreover, global collaboration and coordination are crucial for establishing common standards and best practices in AI alignment.

Where it stands now

I realize that it’s very much a work in progress, so I will continue look for the signs of attention to alignment governance. Without a workable approach to governance and the operational aspects of AI alignment, this effort will fall seriously short of preparing for Superintelligence.

[2023-07-14: This post was written earlier in the week for publication today. Yesterday (July 13) the FTC announced that it is investigating whether OpenAI’s has “engaged in unfair or deceptive practices relating to risks of harm to consumers, including reputational harm.” This seems to me very much an alignment issue, and foreknowledge of the investigation may have prompted … no pun intended… an announcement schedule for which the company was not fully prepared.]

One really obvious omission is the role of public engagement. And, to be frank, I do not have the background or skills to address public engagement in any useful manner. This is an aspect that I will leave for others to address.

My attention to AI alignment remains primarily focused on technology, on policy, on governance and on the means of enforcement. I firmly believe that the industry as a whole, along with governmental bodies responsible for societal well-being, must be active participants, as well as serious contributors of skills and financial support.

What got my attention?

Not an hour goes by without some podcast, blog post, vlog, or news article bringing up the term Large Language Models (LLMs). It’s generative AI, in-your-face, 24x7.

Seeing as how I’m hip-deep in experiments with generative AI and dataset governance, I am not going to break ranks. The past weeks have required me to dig deep into the use of technologies that are adjuncts of the Large Language Models (LLMs). In particular, I have been hacking up open source LLMs, vector databases, frameworks, deployment platforms … all in pursuit of building a source of inference and intelligence around a proprietary set of documents. It’s been eye-opening, and (…I admit it…) a lot of fun.

To get a broader perspective, I found myself reading posts that provide an overview of the technology ecosystem for AI. It was worth the time spent.

What’s Required for Autonomous Agents

Michelle Fradin and Lauren Reeder of Sequoia reported on their interviews with over thirty Sequoia startup and emerging technology companies that are crafting their AI strategies. They summarized their findings in The New Language Model Stack:

1. Nearly every company in the Sequoia network is building language models into their products.
2. The new stack for these applications centers on commercial language model APIs, retrieval, and orchestration, but open source usage is also growing.
3. Companies want to _customize _language models to their unique context.
4. Today, the stack for LLM APIs can feel separate from the custom model training stack, but these are blending together over time.
5. The stack is becoming increasingly developer-friendly.
6. Language models need to become more trustworthy (output quality, data privacy, security) for full adoption.
7. Language model applications will become increasingly multi-modal.
8. It’s still early.**

They flesh out each of these points in the post, which is definitely worth reading.

The Variety in Building LLM Applications

Matt Bornstein and Raja Racovanovic of Andreessen Horowitz authored Emerging Architectures for LLM Applications, an excellent treatment of the Emerging LLM App Stack, with components that included

• data pipelines
• embedding models
• vector databases
• ‘playgrounds’
• Orchestration and chaining
• APIs and plugins
• Caching for LLMs
• Logging and (it had to happen soon) LLMOps
• App hosting
• the proprietary and open source LLM APIs
• Cloud Service Providers
• ‘opinionated’ / specialty clouds

Their discussion and explanation of Data Preprocessing and Embedding is excellent in that they detail the various flavors of vector databases. Also important, they treat Prompt Construction and Retrieval separately from Prompt Execution and Inference.

Their treatment of agents and agent frameworks is a bit light, but that might be a function of the small number of companies that have incorporated Agent Frameworks in production offerings.

(While you’re on this page, take advantage of the related stories. Most of them are really worth your time.)

Moats? We don' have no moats.

In early May, an anonymously authored document entitled We Have No Moat, And Neither Does OpenAI was making the rounds inside Google, when it leaked and made a lot of waves. At the time, I read portions of the document which were included in various posts, but it wasn’t until last week that I sat down and read the document in its entirety. SemiAnalysis authors Dylan Patel and Afzal Ahmad released the document (with some clean up), and for that we should thank them.

The TL;dr:

Google and OpenAI, arguably the two most advanced organizations in generative AI have spent billions getting to their respective positions. But the real competitive threat comes from open source.

While our models still hold a slight edge in terms of quality, the gap is closing astonishingly quickly. Open-source models are faster, more customizable, more private, and pound-for-pound more capable. They are doing things with $100 and 13B params that we struggle with at $10M and 540B. And they are doing so in weeks, not months. This has profound implications for us:

• **We have no secret sauce. **Our best hope is to learn from and collaborate with what others are doing outside Google. We should prioritize enabling 3P integrations.
• **People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. **We should consider where our value add really is.
• **Giant models are slowing us down. **In the long run, the best models are the ones which can be iterated upon quickly. We should make small variants more than an afterthought, now that we know what is possible in the <20B parameter regime.

Among the many points the document then raises as to how this situation has emerged, two stand out for me.

• Retraining models from scratch is the hard path
• Large models are not more capable in the long run if we can iterate faster on small models
• Data quality scales better than data size (i.e. the volume of data)

The timeline included is the screenplay of a movie … arguably a great movie for tech geeks and business school case studies some time in the future, but a movie nonetheless.

This is an important document to read and keep in mind, even if you do not agree with all the points.

What got my attention?

The week of June 18 was heavily trafficked with long-form posts and slick pdfs addressing the governance and regulation of AI, by which most of the authors and lots of readers mean ‘generative AI’. The attention is warranted. The perspectives of from three representative quarters of the ecosystem are disturbing, though for a variety of very different reasons.

Big Tech and Governance

Microsoft published a 40 page, very polished .pdf entitled Governing AI: A Blueprint for the Future. It lays out a five-point ‘blueprint’ to ‘address several current and emerging AI issues through public policy, law, and regulation. The points addressed are:

• First, implement and build upon new government-led AI safety frameworks
• Second, require effective safety brakes for AI systems that control critical infrastructure
• Third, develop a broad legal and regulatory framework based on the technology architecture for AI.
• Fourth, promote transparency and ensure academic and nonprofit access to AI.
• Fifth, pursue new public-private partnerships to use AI as an effective tool to address the inevitable societal challenges that come with new technology

This document is a love-note. It name-checks the efforts of various federal Departments and technology organizations, but mostly by stating that they’ve red the documents and will do their best to adhere to the directives and support the direction. For me, it did little to identify those areas which Microsoft believes to be truly critical and in need of focused attention. It’s worth your time to scan, but I’m not sure you’ll find much of substance.

Professional Societies

Eliza Strickland, writing the article The Who, Where, and How of Regulating AI in the IEEE’s Spectrum publication, sets out to call attention to the anxiety that’s being produced, particularly with the respect to knowledge pollution and existential risk. This is how the article starts:

During the past year, perhaps the only thing that has advanced as quickly as artificial intelligence is worry about artificial intelligence.

In the near term, many fear that chatbots such as OpenAI’s ChatGPT will flood the world with toxic language and disinformation, that automated decision-making systems will discriminate against certain groups, and that the lack of transparency in many AI systems will keep problems hidden. There’s also the looming concern of job displacement as AI systems prove themselves capable of matching or surpassing human performance. And in the long term, some prominent AI researchers fear that the creation of AI systems that are more intelligent than humans could pose an existential risk to our species.

It goes on from there to point out that the earliest and most concerted efforts to consider AI can be attributed to the EU, and culminating in the April 2021 European Commission proposed the [AI Act](https://artificialintelligenceact.eu/). She continues with a short take on the efforts of the rest of the world, noting that the US has gotten off to a “slow start.”

Last year a national law was proposed, but it went nowhere. Then, in October 2022, the White House issued a nonbinding Blueprint for an AI Bill of Rights, which framed AI governance as a civil rights issue, stating that citizens should be protected from algorithmic discrimination, privacy intrusion, and other harms.

It’s hard not to feel the clenching of jaw or gnashing of teeth.

The View from Sand Hill Road

Marc Andreessen, the venture capitalist who supplied us with the memorable statement that “Software is eating the world.”, provided us with the ultimate love-letter in Why AI Will Save the World. Compare his lead-in to the post with that of Eliza Strickland:

The era of Artificial Intelligence is here, and boy are people freaking out.

Fortunately, I am here to bring the good news: AI will not destroy the world, and in fact may save it.

First, a short description of what AI is: The application of mathematics and software code to teach computers how to understand, synthesize, and generate knowledge in ways similar to how people do it. AI is a computer program like any other – it runs, takes input, processes, and generates output. AI’s output is useful across a wide range of fields, ranging from coding to medicine to law to the creative arts. It is owned by people and controlled by people, like any other technology.

A shorter description of what AI isn’t: Killer software and robots that will spring to life and decide to murder the human race or otherwise ruin everything, like you see in the movies.

An even shorter description of what AI could be: A way to make everything we care about better.

He paints a very positive picture, and one which (as a technologist) I cannot argue with on a point by point basis. It’s more about what he doesn’t address.

After characterizing the majority of those who advocate new restrictions, regulations and laws regarding AI as either “Baptists” (the true believers) or “Bootleggers” (the self-interested opportunists who seek regulation that insulate them from competitors), he attempts to take apart the AI risks that are most widely echoed in the popular press:

• Will AI Kill Us All?
• Will AI Ruin Our Society?
• Will AI Take All our Jobs?
• Will AI Lead to Crippling Inequality?
• Will AI Lead to Bad People Doing Bad Things?

And he finishes with the REAL risk of not pursuing AI with maximum force and speed: The threat of AI Supremacy by China.

His points as to what should be done with respect to AI sound, as you might expect, to be characterized primarily as:

• Let the Big AI companies go as fast as they can.
• Let the startup AI companies go as fast as THEY can.
• Let Open Source AI ‘compete with’ Big AI and the small ones as well.
• Offset the risk of bad people doing bad things by working partnerships of the private sector and government.
• Prevent the risk of China’s dominance in AI by using ‘the full power of our private sector, our scientific establishment, and our governments in concert to drive American and Western AI to absolute global dominance, including ultimately inside China itself. We win, they lose.’

Let me know how the perspectives and concerns of these three documents sit with you.

On Thursday, June 29, 2023, two research scientists from Google posted an announcement on the Google Research Blog about the first Machine Unlearning Challenge. I was thrilled to read the opening sentence of the second paragraph:

Fully erasing the influence of the data requested to be deleted is challenging since, aside from simply deleting it from databases where it’s stored, it also requires erasing the influence of that data on other artifacts such as trained machine learning models.

I was also encouraged further when the authors pointed out the value of Membership Inference Attacks (MIAs) as tools being employed in identifying the source data used in training datasets.

This is one of the few active projects I’ve encountered that seeks to address the need to renovate LLMs by finding and removing unwanted sources that were incorporated in the model’s training data. I’m keeping a close watch on this one.

Announcing the first Machine Unlearning Challenge

Thursday, June 29, 2023

Posted by Fabian Pedregosa and Eleni Triantafillou, Research Scientists, Google

Deep learning has recently driven tremendous progress in a wide array of applications, ranging from realistic image generation and impressive retrieval systems to language models that can hold human-like conversations. While this progress is very exciting, the widespread use of deep neural network models requires caution: as guided by Google’s AI Principles, we seek to develop AI technologies responsibly by understanding and mitigating potential risks, such as the propagation and amplification of unfair biases and protecting user privacy.

Fully erasing the influence of the data requested to be deleted is challenging since, aside from simply deleting it from databases where it’s stored, it also requires erasing the influence of that data on other artifacts such as trained machine learning models. Moreover, recent research [1, 2] has shown that in some cases it may be possible to infer with high accuracy whether an example was used to train a machine learning model using membership inference attacks (MIAs). This can raise privacy concerns, as it implies that even if an individual’s data is deleted from a database, it may still be possible to infer whether that individual’s data was used to train a model.

Given the above, machine unlearning is an emergent subfield of machine learning that aims to remove the influence of a specific subset of training examples — the “forget set” — from a trained model. Furthermore, an ideal unlearning algorithm would remove the influence of certain examples while maintaining other beneficial properties, such as the accuracy on the rest of the train set and generalization to held-out examples. A straightforward way to produce this unlearned model is to retrain the model on an adjusted training set that excludes the samples from the forget set. However, this is not always a viable option, as retraining deep models can be computationally expensive. An ideal unlearning algorithm would instead use the already-trained model as a starting point and efficiently make adjustments to remove the influence of the requested data.

Today we’re thrilled to announce that we’ve teamed up with a broad group of academic and industrial researchers to organize the first Machine Unlearning Challenge. The competition considers a realistic scenario in which after training, a certain subset of the training images must be forgotten to protect the privacy or rights of the individuals concerned. The competition will be hosted on Kaggle, and submissions will be automatically scored in terms of both forgetting quality and model utility. We hope that this competition will help advance the state of the art in machine unlearning and encourage the development of efficient, effective and ethical unlearning algorithms.

…

The U.S. Department of Commerce’s National Institute of Standards and Technology (NIST) has released its Artificial Intelligence Risk Management Framework (AI RMF 1.0), a guidance document for voluntary use by organizations designing, developing, deploying or using AI systems to help manage the many risks of AI technologies.

See OODALoop’s post

The NIST Document: Artificial Intelligence Risk Management Framework (AI RMF 1.0)