Cheating on Standardized Tests and Systems Arch Types

Peter Senge introduced the mainstream business community to system archetypes in his book, The Fifth Discipline. These arch types describe patterns of behavior of a system. Systems that we take for granted can be expressed by circles of causality. By understanding the system arch types, we can understand the system at a deeper level and identify leverage points which enables efficient changes in a system.

Let’s consider a recent example. The map below is a Systems Dynamic Model that explains the reinforcing loops that encourage cheating by teachers on standardized tests. Recently, several teachers were sentenced to prison terms over a cheating scandal in Atlanta, Georgia. By using over-justification schemes, we have turned honest people into criminals.

 Reference: https://www.leveragenetworks.com/blog/teacher-cheating-dynamics-scandal-or-scapegoating

Dr. W. Edwards Deming would be livid. When people are put into impossible situations with goals and targets that cannot be met under the current system, you get unintended consequences. We have seen it with nurses falsifying reports to achieve targets and teachers in Atlanta changing test scores. Charles Goodhart's law sheds light on this problem; "When a measure becomes a target, it ceases to be a good measure."  

It would be interesting to see a model for health care with the advent of P4P (pay for performance) schemes. Fraud headlines will be coming soon. Here we will have “goals gone wild” turning professionals into crooks.

"When a measure becomes a target, it ceases to be a good measure." Charles Goodhart (1975)[i]

From the Telegraph, UK: “In February 2010, the first report into the scandal of appalling nursing at Stafford Hospital — where patients were left in filthy sheets, their dressings unchanged while nurses shouted at and mocked them — concluded that in order to achieve the coveted Foundation status, the hospital trust’s management had become obsessed with meeting government targets rather than looking after the patients…Nurses elsewhere endlessly repeat this complaint. Everything has to be documented. Everything is driven by ‘performance targets’ which have to be audited.”[ii] “ Meanwhile, nurses were instructed by senior nurse colleagues to falsify waiting times, and to claim that patients had been seen more quickly than they were.”[iii]

From the healthcare example, we can see the effects of Charles Goodhart’s insightful quote on the use of targets and unintended consequences. Dr. W. Edwards Deming was more direct and to the point; “Fear invites wrong figures.”[iv]

When leaders and managers employ the use of numerical goals, the intent is usually to cause improvement as opposed to using the measure for judgment. Deming noted three unintended consequences of numerical goals: 

“Will a numerical goal be achieved? Anybody can achieve almost any goal by:

• Redefinition of terms

• Distortion and faking

• Running up costs”[v]

The unintended consequences of goals and targets where there is no method is pervasive throughout industry, government, healthcare and education. We have had several attempts to improve education over the last three decades. This has led to teachers being fired and charged criminally.[vi] Donald T. Campbell warned of this problem in what has become known as Campbell's law:

"The more any quantitative social indicator (or even some qualitative indicator) is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Associates in Process Improvement (API) has attempted to strike a balance on the use of goals and targets:[vii]

“Advice about the use of numerical goals in setting an aim for an improvement project could include “Never use them” or “Always provide one and let other people figure out how to achieve it.” This book’s authors recognize the abuses associated with numerical goals and the potential unintended consequences when people are held responsible for results they are not capable of achieving…We do not believe these instances of poor practice or abuse should preclude beneficial use of numerical goals.

We also recognize some hardy souls need only the challenge of a numerical goal to find ways to actually improve the system. Our experience indicates that a middle ground between these two extremes is achievable and useful.

First and foremost, numerical goals must be connected to methods for achieving the goals. Leaders should understand that to improve a stable system beyond the current level of performance a fundamental change is needed…here are some initial considerations to develop methods for achieving numerical goals:

• Observe other organizations that have accomplished similar goals.
• Give some basic concepts or ideas that could feasibly result in achieving the goal.
• Draw out ideas from participants themselves by asking questions such as, “What would it take to get a 50 percent reduction in time to ship an order?” From the healthcare example above; what would it take to reduce wait time and patient harm?
• Ask experts on the changes being considered what level of improvement is possible.

Numerical goals can also be a convenient way to communicate expectations. What are the consequences of not meeting the numerical goal? Are small and incremental improvements expected, or are large breakthrough changes necessary? If the numerical goal is used well, it communicates not only the expectation but also the support that will be offered. Large changes to big systems usually require investment of time and capital. When first using a goal to break the current bounds of the status quo, leaders must furnish:

• An explanation of the need for and feasibility of the goal
• Assurances the goals will be used to cause new thinking and learning, and not for judgment

Systems thinking informs us that one can create the illusion of improvement by only focusing on one measure of the system. Improvement requires that we focus on multiple measures to mitigate the unintended consequences noted by Deming. Specifically, three types of measures can be used to ensure a fundamental change in patient waiting time:

• Outcome Measures – Waiting Time
• Process Measures – Number of patients scheduled; number of drop in patients
• Balancing Measures – Cost per patient; patient satisfaction. 

Please note the bold and underlined statement used earlier; First and foremost, numerical goals must be connected to methods for achieving the goals. This is directly from our learning from Deming and has been verified in our work experience in API. A project where the solution is unknown, may not have a method. This has to be developed. Deming would chide his audiences; “If you have a method, why did you not do it last year? There can be only one reason; you were goofing off.” Of course this got a laugh, but there is a lesson here as well. The people with whom we are working usually have full time jobs. Taking time to develop, test and implement changes in the unknown, is already a daunting task. What they need is help from sponsors, not expectations for more judgment; they already have plenty of that. So if the sponsor is showing up and removing barriers and doing their part, they will have a good idea of when the team is to be finished. Deming would call this “substituting leadership.”

References:

---

[i] Goodhart, C.A.E. (1975). "Problems of Monetary Management: The U.K. Experience". Papers in Monetary Economics (Reserve Bank of Australia) I.

[ii] http://www.dailymail.co.uk/health/article-2275943/NHS-Why-nurses-stopped-CARING.html#ixzz3NJS41dHF

[iii] http://www.telegraph.co.uk/health/healthnews/9851763/Mid-Staffordshire-Trust-inquiry-how-the-care-scandal-unfolded.html

[iv] W. Edwards Deming. The New Economics for Industry, Government, Education (p. 94)

[v] W. Edwards Deming. The New Economics for Industry, Government, Education (p. 43).

[vi] http://www.nytimes.com/2010/06/11/education/11cheat.html?pagewanted=all&_r=0

[vii] Langley, Gerald J.; Moen, Ronald D.; Nolan, Kevin M.; Nolan, Thomas W.; Norman, Clifford L.; Provost, Lloyd P. (2009-06-03). The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (Kindle Locations 1916-1918). Wiley Publishing. Kindle Edition.

Where are all the Innovative – Cuddly leaders?

The temptation of consultants and others is to reform leaders based on our personal perspectives; what good leadership means to us personally. This can mean the projection of our personality type on the leader.

Let’s review the basic personality types. Maccoby and Scudder (2010) have offered the following description of the evolution of the theory of personality as developed by Sigmund Freud and Erich Fromm:

Sigmund Freud first used his model to help explain psychopathology, but he also employed it to describe three normal personality types he called:

1.  Erotic – a caring personality with a strong value of loving and caring relationship. Fromm’s work refers to the non-productive version of this type as the Receptive Orientation.

2.    Narcissistic – independent and not open to intimidation; characterized by an ego with a large amount of aggressiveness at its disposal, which also manifests itself in a readiness for activity. Fromm’s work refers to the non-productive version of this type as the Exploitative Orientation.

3. Obsessive – ideals of hard work and conscientiousness; characterized by a demanding super-ego, strong commands programmed in childhood. Fromm’s work refers to the non-productive version of this type as the Hoarding Orientation.

Erich Fromm described the non-productive orientations and their related productive characteristics in an effort to move people toward increased productivity. He accepted and modified Freud’s types and added a fourth type to Freud's trio; the marketing personality. Just as the obsessive personality is the bureaucratic prototype; the marketing personality fits the typical Interactive Social Character.

4. Marketing – the ego-ideal is radar-like, orienting behavior to what is "appropriate" according to group values and pressures; to avoid looking bad.

Dr. Michael Maccoby has described the four basic types of personality developed by Freud and Fromm as the Caring, Visionary, Exacting and Adaptive. These terms are very much in line with the original work from Freud and Fromm.

Many books on leadership are written from the perspective of the leader being caring. When the author develops this scenario of the “caring boss,” they may be projecting on to the leader those traits they value. Circumstances and context often define the need for other strengths in our leaders. Maccoby observed that many of the leaders he encountered were far from cuddly; in fact some, at times, could be downright mean. This dissonance in what authors found desirable versus the reality caused Maccoby to reflect on his past and present work with leaders (the following is taken from Narcissistic Leaders ---Who succeeds and who fails.):

“The emergence of a new kind of leader caused me to reexamine all of my theories about leaders and personality. I went back to the psychoanalytic teachings of Sigmund Freud and the psychoanalyst and social philosopher Erich Fromm (1900– 1980) and sifted through thirty years of experience inside corporations, working with CEOs both as a consultant and psychoanalyst.

I rethought my understanding of historical figures and literature. What emerged surprised me. The psychological portrait of today’s business leaders that takes into account their personality traits, describing how they achieve innovations, engage followers, and react to the euphoria of success as well as the stress of setbacks, most closely fits the normal personality type that Freud called narcissistic: “People belonging to this type impress others as being ‘personalities’; they are especially suited to act as a support for others, to take on the role of leaders and to give a fresh stimulus to cultural development or to damage the established state of affairs.” In other words, these are the type of people who are most likely to say that they want to change the world.

I’m using the term “narcissism” to describe some of the most important business leaders in the world; but how could a word that’s become synonymous with all sorts of self-centered behavior—a sense of overall superiority and entitlement, a lack of empathy or understanding of others, the need for constant attention and admiration, and overall arrogance— apply to them? These days, in both the psychiatric field and in colloquial conversation, “narcissism” has become a term for egoism, egocentricity, or just plain bad manners. But I believe the concept of narcissism has been widely misunderstood ever since Freud coined it after Ovid’s pathologically self -involved creature from Greek mythology. I want to bring about a radical new definition of the term and the way we think about leadership, and show you how your understanding of productive narcissism can help you.

The type of person who impresses us as a personality, who disrupts the status quo and brings about change. Narcissists have very little or no psychic demands that they have to do the right thing. Freed from these internal constraints, they are forced to answer, for themselves, what is right, to decide what they value, what, in effect, gives them a sense of meaning. They create their own vision, a sense of purpose that not only engages them but also inspires others to follow them.”

Maccoby also identified the strengths and weaknesses of the Productive Narcissist:

STRENGTHS OF THE PRODUCTIVE NARCISSIST

•        Visioning to change the world and create meaning

•        Independent thinking/ risk taking

•        Passion

•        Charisma

•        Voracious learning

•        Perseverance

•        Alertness to threats

•        Sense of humor

WEAKNESSES OF THE PRODUCTIVE NARCISSIST

•        Extreme sensitivity to criticism

•        Not listening

•        Paranoia

•        Extreme competitiveness

•        Anger and put-downs

•        Exaggeration

•        Lack of self-knowledge

•        Isolation

•        Grandiosity

When the weaknesses manifest themselves in the leader, these weaknesses are then targeted to be “fixed” by authors, consultants and others.  Unfortunately, these weaknesses come with the strengths. The leader, if aware, can learn to mitigate these weaknesses, but from time to time, given the circumstances, they may rear their ugly head.

We have focused on the Productive Narcissist (Visionary) leader. However, the other basic types of personality all have strengths and weaknesses. For more on the other types and more importantly, how to have a productive relationship with your Productive Narcissistic Boss, we highly recommend reading Maccoby’s book; Narcissistic Leaders ---Who succeeds and who fails.

References:

Maccoby, Michael (2012-04-25). Narcissistic Leaders: WhoSucceeds and Who Fails; Harvard Business Review Press (May 19, 2007)

Maccoby, Michael and Tim Scudder, Becoming A Leader We Need with Strategic Intelligence (2010).   Note graphic and narrative are taken from p. 92 of this workbook.

Jack Welch and the Gymnastics of Self-Justification

The following post is a comment on the link below which will take you to an article by Jack and Suzy Welch:

http://www.linkedin.com/today/post/article/20131202152255-86541065--rank-and-yank-that-s-not-how-it-s-done?goback=%2Egde_1207637_member_5815151211132772355

This article reminds me of the process reengineering aftermath when the authors tried to rewrite history on “shooting the wounded,” getting rid of people etc. The authors wrote article after article saying they were misunderstood. Regardless “Process Reengineering” became a synonym for getting rid of people. The ability for self-justification is a human trait that we all possess; intelligent people are even more adapt at this core competency. Welch is obviously practicing in this article.

 Still, the fundamentals are still wrong. Deming’s equation comes to mind:

·         X=Individual

·         Y=System

·         X + [XY] = 8

As Deming would note, “One equation and two unknowns, unsolvable.” In many organizations the system in not under suspicion.  Management merely sets Y = 0 and then attributes the results to X. 

Leadership should be up close and personal. If this is the case, then it will be obvious who is contributing and who is not. We have witnessed several people who have been let go in organizations over the years. Typically, it is obvious to everyone that the person does not fit. Tragically, even though this is apparent to everyone, the separation process goes on way too long for the health and morale of the organization. It is usually very difficult for leaders to admit that they have hired someone who does not fit. Meanwhile the organization suffers, relationships with customers are damaged and the firm still faces the evitable change that must take place.

Deming used to identify “substitutes for leadership.” Yank and Rank was certainly a poor substitute. 

Why is stability of the data so important?

Recently on a popular Six Sigma site the following question appeared:

 "I have a question if I have variable data ( that is not normally distributed) I then transferred it to Attribute data and worked out the DPMO from the opps/defects. If the DPMO is normally distributed can I carry on using stats such at t – tests etc. Or because it is originally attribute data I should use chi squared etc? Any advise appreciated."

From this question, you could run a three day workshop. My short attempt at an answer included:

"As others have said, stay with the continuous data. Before doing anything else put the data on an appropriate control chart and learn from the special causes. As Shewhart noted: things in nature are stable, man made processes are inherently unstable. I have taken this from Shewhart’s postulates. T test and other tests all rest on the assumption of IID; Independent and Identically Distributed. If there are special causes present these assumptions are violated and the tests are useless. Even though the “control chart” show up in DMAIC under C for many novices, it should be used early. Getting the process that produced the data stable is an achievement. It is also where the learning should start. Calculating DPMO, and other outcome measures can come later; after learning and some work. Best, Cliff"

Why the fixation on outcomes, calculating capability, DPMO and the like?  Without any knowledge about stability of the data such calculations are very misleading. In 1989, I sat in a workshop where Dr. W. Edwards Deming made the following comment, "It will take another 60 years before Shewhart's ideas are appreciated." At the time, I thought he was nuts. Control charts were everywhere. Then they disappeared. Now I see Deming as a prophet.

Historically, we are going through a period in improvement science that is not unlike the dark ages. We have people grasping for easy path and quick answers generated by the computer that might as well be "unmanned." Getting the process stable is an achievement! Our first move with statistical software should not be a normality check, but a check of the data to see if we have data that is stable and predictable. If we have such a state, then our quality, costs and productivity are predictable. Without this evidence, we are flying blind.

We are doing a 4.5 Sigma program?

Dr. Bill Latzko has published a very short and informative paper on the ideas underlying the Six Sigma program. Advocates of Six Sigma often talk about achieving Six Sigma quality meaning 3.4 parts per million. Latzko discusses this and the assumptions in this great paper:

http://www.latzko-associates.com/Publications/SIX_Sig.pdf

Dr. Taguchi idea of reducing variation around a target should be studied and understood by those who are interested improvement. Focusing on meeting specifications has been a step backward that Deming warned us about in his last book: “Conformance to specifications, zero defects, Six Sigma Quality, and all other (specification-based) nostrums all miss the point.”

We can do better.

Using Planned Experiments to Accelerate Learning and Improvement

How can we speed up our learning and make improvements that will help us reduce costs? Typically, people making improvements try to change one thing at a time. The problem with this approach is the complexity of the processes and the interdependent factors we are studying. To be more effective, we need to learn how to test more than one factor at a time. Let’s consider an example. 

An improvement team in a hospital system that is supported with citizens’ taxes is attempting to help patients not miss appointments. When someone does not show up for an appointment, this is a loss to the society. The improvement team tried texting the patients one day in advance. This did not seem to make much impact. It was then decided to have a person call the patient personally one day in advance. An improvement advisor happened to overhear this discussion and suggested that the team test two factors at two levels or sometimes referred to as a design which will require 4 runs or tests. Rather than abandon the text idea, test it with the call idea, but add in a lead time factor with a call 1 day and 3 days in advance. The improvement advisor suggested calling three days in advance might allow the patients to plan better. Figure 1 describes the simple designed experiment matrix that was developed. Also included are the percent of no shows as the response variable. 

Figure 1: Design Matrix for a Two Factor at Two Level Experiment

 

 From Figure 1, you can see readily that the 4^th test that combined the call with 3 days reduced the no show rate to 10%. The next best combination was a text messages at 3 days in advance of the appointment with a 12% no show rate. The team decided to avoid the cost of taking up a person’s time to actually call patients and use the 3 day advance text. One team member suggested setting the computer to also give a 7 day warning. The team agreed to use another test to follow this idea up. The response plots in Figure 2 describes the results from the 4 test runs.

Figure 2: Response Plots for the Factorial  design

 Before the improvement advisor suggested the designed experiment, the team was ready to run another one factor test: the more expensive idea of adding more people to make personal calls. This would have reduced the no show rate from 25% to 15%, a large improvement but at a significant cost increase. By experimenting with the text and call idea with different frequency levels the team was able to improve while lowering costs.

References:

1.      Quality Improvement through Planned Experimentation, Ronald Moen, Thomas Nolan, and Lloyd Provost, McGraw-Hill, NY, second edition, 1999