Dan Erwin

Do You Have to Take the Scientist's Word for It?

On regular occasions I converse with my business clients about the difference between business science (technically, social science ) and consultantancy snake oil. It's an important issue because a lot of the snake oil is regularly touted in business magazines and among execs. In today's world, where evidence-based management is available on an awful lot of issues, it's downright stupid to follow just any consultant's recommendation without analyzing it. That includes my own recommendations.

So how do you tell the difference between snake oil and genuine science? And do you have to have a research degree to be able to figure out the validity of many recommendations? Janet Stemwedel, one of those rare birds with two PhD's from Stanford, has simplified the issue for us. Furthermore, she doesn't think a PhD is necessary to assess the validity of science. 

In two blogs, posted on the Scientific American site, she deals forthrightly with the issue. She emphasizes that though scientific knowledge is built on empirical data and that though the data can vary significantly within the different disciplines, there are common patterns of reasoning that all scientists use.

In what follows, she spells out an approach that most any slightly critical thinker can use. Because the material is very relevant to much of what's going on in business--much less all the political conversations regarding science-I am printing parts of two blogs, the first distinguishing between baloney and real science, and the second explaining the scientific method.

The big difference (Karl) Popper identifies between science and pseudo-science is a difference in attitude. While a pseudo-science is set up to look for evidence that supports its claims, Popper says, a science is set up to challenge its claims and look for evidence that might prove it false. In other words, pseudo-science seeks confirmations and science seeks falsifications.

There is a corresponding difference that Popper sees in the form of the claims made by sciences and pseudo-sciences: Scientific claims are falsifiable — that is, they are claims where you could set out what observable outcomes would be impossible if the claim were true — while pseudo-scientific claims fit with any imaginable set of observable outcomes. What this means is that you could do a test that shows a scientific claim to be false, but no conceivable test could show a pseudo-scientific claim to be false. Sciences are testable, pseudo-sciences are not.

One of the things I like about her emphasis is that she's made the route to knowledge "potentially" open to any of us. So here's the core input on scientific reasoning:

One way to judge scientific credibility (or lack thereof) is to scope out the logical structure of the arguments a scientist is putting up for consideration. It is possible to judge whether arguments have the right kind of relationship to the empirical data without wallowing in that data oneself. Credible scientists can lay out:

• Here’s my hypothesis.
• Here’s what you’d expect to observe if the hypothesis is true. Here, on the other hand, is what you’d expect to observe if the hypothesis is false.
• Here’s what we actually observed (and here are the steps we took to control the other variables).
• Here’s what we can say (and with what degree of certainty) about the hypothesis in the light of these results.
• Here’s the next study we’d like to do to be even more sure.

Lest there be any question, the usual Webster definition of "hypothesis" is appropriate to the above: a proposition tentatively assumed in order to draw out its logical or empirical consequences and so test its accord with facts that are known or may be determined.

The underlined terms will take you to the relevant blogs in The Scientific American.

When Feedback Goes Wrong

In significant contrast to industrial age workers, knowledge workers are rewarded and often succeed because of their mentoring and coaching expertise. Indeed, my blog, What the 21st century career looks like, emphasizes not only organizational needs for technical and analytical skills, but the competencies of persuasion, social perceptiveness, the capacity to bring the right people together on a project—and, coaching and mentoring skills. So if you want the most lucrative salary and the greatest growth and learning opportunity, you’ll involve yourself in coaching and mentoring. Feedback, of course, is the central competency of coaching.

But feedback can go very wrong and fail to achieve its fundamental objective. Why is that?

A recent web conversation among members of the Center for Evidence-Based Management alerted me to an intriguing study on feedback by Jodi Goodman and colleagues that deals with this very issue. But first, a little background.

Those who coach regularly are familiar with the cookbook statement on feedback that effectiveness is dependent on specific, timely, concrete input. As Goodman says, that means that effective feedback is immediate and includes information on the behaviors that were performed incorrectly and how to correct them.

However, as you become more focused on the feedback process, you soon recognize a real problem. When you give highly detailed, specific input, a certain proportion of people don’t seem to be able to apply the information to other contexts, demonstrating they haven’t truly learned from your feedback.

For all but the simplest of tasks, learning the correct actions typically requires some active search and processing about what will achieve the desired outcomes. And if the task is more complex, that increases the need for the mentee’s exploration of the different impacts of actions.

Goodman points out the double bind: If your feedback amounts to hand-holding (highly specific, concrete and timely input) throughout the performance, you’ll get the performance you want—yet inhibit the learning. That seems to go with hand-holding for most people. Alternatively, if you provide less frequent and specific feedback—no hand-holding—the result is often poorer practice, but better learning. That’s the result of searching for action impacts.

So the research shows clearly that hand-holding feedback won’t give you the learning you want for future projects. That doesn’t mean you should never do hand-holding, but it does seriously question the conventional wisdom about the value of specific, timely feedback.

Here are my conclusions:

• If you’ve got a tight deadline, do the hand-holding feedback so your mentee can meet the deadline. But realize the person is liable not to be able to repeat the actions in a different context. Learning probably did not take place.
• If your deadline is not tight, go for learning. Limit the amount of feedback you give, and turn the person loose to explore and search on their own. The person will learn, but may not achieve the level of performance you desire. You can refine the performance later.
• I’ve learned it’s important to provide the person similar opportunities in different context ASAP so that they may internalize the learning. Dedre Gentner of Northwestern has shown in numerous studies that behaviors need to be successful in several different contexts for learning to take place.
• I’ve also found that unless the person is clueless about a given task, it’s very important to emphasize that you don’t need to see them until they’ve exhausted their search and exploration. One of my friends, Eric Linner, an associate at Wold Architects of St Paul, is a master at insisting upon search and exploration—“do your homework”--before they come to him for help. The corporate culture, heavily oriented to learning and team work, has its answer in Eric--a straight-talking, poised, ex-marine who can be a bit scary to newcomers who don’t understand the ground rules. But employees quickly learn he’ll give them great feedback—if they’ve done their search homework before they come to him. Employees are immensely respectful and grateful to Eric for his learning orientation—in spite of his toughness. Indeed, I’d say that an architect that gets to work with Eric is very, very lucky.  

So there’s a definite danger in over-generalizing about feedback, and it’s easy to go wrong. There are plenty of people out there who, when needing feedback, want all the details. They want hand-holding. I rarely succumb to those demands—largely because it’s not good for them. Typically, I’ll respond to that expectation, “That’s enough information to get you going. I’m quite certain you can figure out what to do. And if you get caught in a bind, come back and we’ll talk about it.”

There are also plenty of employees who—rightly—don’t need or want hand-holding, but desire just a little input. I thrive with that kind, because when they come back I’m liable to learn as much from them as they learned.

In sum, don’t over generalize about feedback. Too much or too little specificity can cause feedback to go wrong. It all depends . . . on the task deadlines and your objective for the mentee.

Jodi S. Goodman, Margarethe Hendrickx and Robert E. Wood, Feedback Specificity, Exploration, and Learning, Journal of Applied Psychology, 2004, Vol. 89, No. 2, 248–262.

Flickr Photo by Rolleh

Quotation of the Week: Andy Rooney

The well-known curmudgeon, Andy Rooney, is retiring after nearly 35 years at the tail end of 60 Minutes. I never was much of an audience member, but when I dropped in, Rooney was usually adept at surfacing responses from me. They varied from, "Oh god, no. Don't be a dumbass," to a burst of laughter. The one that follows was quoted in Time this week. I must confess that I nearly fell off my stool laughing. The notion begins as far back as Socrates-- but Rooney turns it on its head.

James Poniewozik set it up this way: But while it's easy to sum him (Rooney) up as a cranky old man telling the world to get off his televisual lawn, one of my favorite observations of his was an un stereotypically Rooney Sentiment suggesting that it's myopic for one generation to assume that everything new is worse.

--It's just amazing how long this country has been going to hell without ever having got there.

I have never been a devotee of the notion that the younger generation is the worst generation we've ever had to deal with--whoever "we" is. Fact of the matter, I've always had a great deal of fun trying to distinguish how the ideas of the new generation, whether X or now, Y, are both similar and different than my generation.

Recently, though, I've thought that my homeland is going to hell in a handbasket what with the ideological ilk of both the left and the right Gen-Xers. I've wanted to scream at their ignorance, "Oh, grow up. Get real. Learn some history. Turn the media off, remembering that they're in the business of getting your attention by hook or crook." FYI:I still remember the look of shock from my beloved Gen-Y protege when I told him that the sole business of the tube was to make money for its corporate supporters. And I supported that with data from an old course in grad school on the role of media and its advertising heritage. So, I said, don't take the media too seriously. That was a message for me, too.

So whenever, in my fundamental pessimistic optimism, I let the pessimistic side have too much power, like Rooney, I have a conversation with myself, reminding me that my pessimism may be nonsense.

A good way to sum this up is with a comment from an interview of Bill Clinton by Joe Scarborough and Mika Brzezinski. As Clinton explained why there's so much dysfunction in congress today, he noted that it was partly a problem belonging to the media. "What works best in politics (and the media) is conflict. But what works best in real life is cooperation." He went on to point out that cooperation is boring, everyday stuff not capable of getting viewer attention. That said, it's an astute observation of at least part of the problem we face in the nation today.