Trust and trustworthiness

“AI agents are powerful,” says an advertisement in my feed. It continues: “Adding a face builds trust.”

On their own, AI agents are not trustworthy. And if you put a façade on one that encourages people to trust it, but you don’t do the work to make sure it deserves that, you’re going to be in trouble. At best, people will learn not to trust the pretty face you put on your interface. At worst, legal liability and/or actual harm to life and limb.

Before we can build trustworthy AI systems, we must take a clear-eyed measure of what we’re dealing with. Feelings run high at present, both pro and con; we’re hearing many mutually exclusive stories about it. But we do have verifiable reality to ground ourselves upon.

What we’re actually dealing with

As always, the first step to gaining trust in any meaningful sense is to earn it. No AI can do this for you. And as with many powerful tools, if you use one carelessly, you are likely to get hurt - and if you do, it will be your own fault. (Of course, providers are, and should be, liable for the consequences of distributing an unsafe tool to the public.)

A useful paradigm for dealing with large language models (which covers most AI that people interact with directly) is to consider them to be advanced mansplaining machines. By default, they are better at being convincing than they are at being accurate. And while it can be meaningfully said that they will lie under certain circumstances, they will also - without discernible intent to do so - make things up rather than admit ignorance.

Overall, it makes sense to think of out-of-the-box LLMs as inexperienced new hires with a particular set of strengths and weaknesses. Like most new hires, they know nothing about your particular business or needs. Unlike most, they are blindingly fast, and well read to the point of being able to at least vaguely recall a sizable fraction of all that has been published (including a bunch of material they properly had no business reading, as it wasn’t paid for). Perhaps as a result of this, they are uncommonly articulate. They are also eager to please; too much so, at times. But like many well-read people, they are prone to overestimating their own knowledge and to thinking shallowly and considering this sufficient, and they are highly resistant to admitting ignorance - they will freely and convincingly make something up if they don’t “remember” whatever information they’re asked for. They’re also lousy at mental math and at intuitively thinking of words in terms of sounds and letter sequences. They have writing quirks, such as the unspaced em-dash (CMOS style), that a growing subset of the public finds increasingly identifiable and annoying as LLM output spreads through the web and social media. And in the case of commercially run LLMs, they process your data - and your users’ - on someone else’s servers, which makes trustworthy user data removal difficult to impossible. (As of this writing, OpenAI is court ordered to retain chat logs pertaining to active litigation.)

The combination of these weaknesses can drastically reduce the benefit an LLM can provide, or even make one an outright liability and cause harm. AI-related psychosis and even death are real risks. Even in use cases where these worst-case outcomes cannot occur, the determinism we’ve come to expect of software - where input X always yields output Y - does not apply here. In addition, anywhere an LLM is exposed to unsanitized incoming data is an attack surface for social engineering exploits with no human involvement needed, and - especially where the LLM is given to understand that the user is in distress or jeopardy - they can be extremely gullible.

Three barriers

Taking all of this together, we see three major barriers to building meaningful trust:

• Overconfidence. Nonsense delivered with the same persuasive conviction as fact.
• Security. Your data and your customers’ at risk or outside your control.
• Reliability. Queries receiving inconsistent responses.

The good news is that while they are fundamental characteristics of the technology, all of these can be mitigated. (The engineering shape of that mitigation is the subject of the Reliability Layer manifesto; the Veridi methodology writeup is a worked example.) But first, we must recognize that putting a trust-inspiring face - literal or figurative - on an untrustworthy system is downright dangerous.

This is part one of two. The next installment goes into what can be done about each barrier.