Anybody can do it. Here's the basic model, along with my reasoning for its value for every business person or professional in law, medicine and architecture.
Artificial Intelligence tool evaluation is an issue that just about every professional needs to start thinking about. More and more the corporate and professional worlds are being marketed for AI tools, some worth purchasing and others a waste of money. Appropriately, these decisions should be made by people with a lot of relevant background.
But nearly every work person, business or professional should have a basic understanding of AI. Yet recent conversations with a number of colleagues reveal that AI is dark hole for the majority of people. So in this blog, I want to offer the underlying thinking about AI from the standpoint of theory construction, and offer two straightforward suggestions.
My first suggestion starts at a different place than you might expect--with the actual background of AI thinking: argument theory. I’m quite certain that most in technology aren’t familiar with argument theory, but actually it’s the basic background of AI tools. In fact it’s the basic background of all human reasoning—sophisticated and unsophisticated.
History reveals clearly that the big picture thinking of AI is built on the Stephen Toulmin model. So if you understand this very simple theory, you’ll have a solid grasp of how all people reason and where to focus to check out their reasoning and also the validity of AI tools. Over more than twenty years, teaching managers and execs how to be better thinkers, I used the same model. Though I was quite certain the same model applied to AI, I doubled-checked it with my grandson, a senior digital scientist at Apple and found a similar analysis and discussion in the latest MIT Sloane Management Review.
Stephen Toulmin was a very interesting guy, a leading English thinker, born just ten years later than Alan Turing, the early developer of the computer. You may remember Turing from the highly successful film, “The Imitation Game” with Benedict Cumberbatch, which tells the story of how he broke the German naval secret in the Enigma machine. Once you understand Toulmin, not only will you understand how people reason, but you’ll also have the key thinking in both the film and for AI.
The argument model is laid out clearly in the Wiki article on Stephen Toulmin, but there is a simple-minded two pages online called Toulmin Method, Blinn College. They use it for writing classes, but the model is used in every field imaginable: marketing, sales R & D, manufacturing, law, medicine, architecture. And, as I wrote earlier, it is the primary background for artificial intelligence. Check out the Toulmin Model or Toulmin Method on google.
What makes the Toulmin model so damn simple is that it’s based on three simple elements.
Claim: a conclusion that must be established: e.g. If a Hispanic tries to convince you he’s an American citizen, the claim would be “I am an American citizen.”
Grounds: a fact that a person appeals to as a foundation for the claim: e.g. For example, the Hispanic might support his claim with the data, “I was born in Dallas, Texas.”
Warrant: a statement authorizing movement from the grounds to the claim: e.g. man born in Texas will legally be an American citizen.
How then, does this apply to understanding and evaluating artificial intelligence?
When a marketing person comes to sell AI to a firm, their conversation follows exactly the same thinking as the Toulmin model. They’re making a claim about their AI tool, telling you that, for example, “our clients have found that this new tool will cut your development time for blah, blah by 35%,” so “we’d like to make a special offer of the tool for $120,000.”
Claim: our AI tool will save you a lot of money
Grounds: other clients have saved 35% on development
Warrant: you can save what other clients have saved
When you lay out the marketing argument, you see its simplicity and stupidity. The simple Toulmin model reveals exactly where to focus your attention and analysis: not on the claim or the warrant, but on the grounds for their truth. I’d be willing to bet that typically most clients buy on the basis of the warrant, not the grounds. I also know that in listening to business arguments, that’s where the focus resides—rarely on the grounds.
As Lebovitz, et al, point out in “The No. 1 Question to Ask When Evaluating AI Tools,” determining whether an AI solution is worth implementing requires that you look past all the performance reports and focus on the “ground truth on which the AI has been trained and validated.” The writers are using the Toulmin language although, typically they’ve never heard of Toulmin, much less understood the immense relevance of his stuff.
What’s obvious is what the argument above emphasizes. To check out and validate any thinking, argument, and any artificial intelligence tool, you need to understand and examine the grounds for their claims. And that’s where most thinkers and AI evaluators fall on their face.
What is the ground truth of an AI tool?
The quality of any AI tool and the value it can bring to an organization are enabled by the quality of the “Ground Truth” of the tool. In general, ground truth is defined as the information that is known to be true, based on objective, empirical evidence. If you look thoughtfully at my simple illustration of an ethnics argument about citizenship, it was focused on the grounds of his argument, not the claim or the warrant. Everything in an argument—and in AI—stands or falls on the basis of the quality of the data, facts, assumptions—the grounds of the argument.
The authors point out that the underlying weakness of most AI tools is that there is rarely an “objective” truth available to be fed into the algorithm. Instead, AI designers construct their ground truth data, and they have plenty of latitude in how they accomplish this. For example, in the medical context, they could use biopsy results to serve as an externally validated result for whether cancer is detected. The problem is that most cancer patients never undergo biopsy tests. And acquiring those tests for all patients in the training set would require enormous investment and patient cooperation. And even more, as one of my friends, a health economist, indicates, the cooperation of the organizations holding the biopsy information can be exceptionally costly, even if available legally. Right now, there are all kinds of interactions, suits, difficulties in gaining health care information.
In sum, the grounds for an AI tool can be very difficult to identify and often inadequate, making the tool questionable and sometimes heavily biased. Furthermore, validating experts’ decisions is extremely challenging and sometimes impossible. But to check out the validity of any tool, the potential users need to be able to compare the ideal standard for decision making with the tool to the experts’ know-how in their real-world deliberating processes.
It’s highly likely that managers will encounter AI vendors less-than-ideal sources of ground truth, especially given costs and the limited potential for obtaining high-quality data.
But the point I’m making is two-fold: you also need to be checking out the grounds for truth in what people say, what news station personnel are arguing and what writers are writing. Sometimes professionals with extensive background and credentials in one area make claims about information in another area, and readers believe them even though their grounds for truth simply won’t support their claim. (See, for example, my blog which focuses on two highly credentialed professionals, neither of which has either the understanding or ability to provide their claimed services**) But now, you should be able to understand the basic for argument and for artificial intelligence. They’re identical—thank you, Stephen Toulmin.
**Lebovitz, et al, MIT Sloane Management Review, Spring 2023, pp.27-30.
**Dan Erwin, typepad.com, 2/21, Should we regulate business coaching?