Untitled Page

The IEI instrument was tested in a series of emotional intelligence courses in a dozen different organizations (primarily in health care, manufacturing, and government). Early tests resulted in minor adjustments to items and layout, primarily for ease of use. Items were also adjusted based on qualitative feedback (any ambiguous or confusing items were dropped or revised) and score distributions (items with one-sided or extreme distributions were eliminated or revised). A final test of 147 subjects was then analyzed to establish preliminary norms for the instrument. All item and instrument tests used 5-point scales. The data is reported here in terms of percentage-of-scale. Early versions of the instrument used 16 items for each or the five domains. With testing, the number of items was reduced to simplify the instrument and make it easier to facilitate and take.

The average overall score in these tests of the IEI was 76.6%. The distribution of scores was a bell curve ranging between approximately 50% and 95% of scale, as illustrated in the following graph:

This charts shows that the scale provides sufficient range for a normal distribution of scores and allows for significant differentiation among responding individuals. In addition, teaching-oriented assessments generally are received better by participants when the scores are perceived as in the positive range, and the fact that almost all scores are above 50% on this scale meets this need to cast the feedback in a sufficiently positive light to minimize resistance and maximize acceptance of scores.

The sample was also used to compare scores by gender (male mean = 76.08%, female mean = 76.70%). There was not a significant difference based on gender, which is helpful as it allows comparison across gender without the need to adjust or norm the scores. It also suggests that the instrument is measuring components that vary significantly between individuals, but are not specific to gender. This is consistent with the test’s design criterion that it measure interactive skills that are relatively easy to acquire or improve, and avoid more innate aspects of personality that cannot readily be altered by individual learning or group training experience.

The instrument measures five separate dimensions of emotional intelligence, and the test sample can be used to compare these subscales in order to see if the items perform comparably. The goal is to have subscales with scores that are fairly close to each other and also to the overall IEI score. This makes for a tighter instrument, and it also permits users to compare sub-scores directly to each other and to their overall score without the need to manipulate them according to a norming formula. In the test data, the following component scores emerged:

The component score means are clustered around the overall mean. They measure distinct components for which we can identify no strong a priori reason to correlate closely. The similarities in component mean scores suggest there is reasonable internal consistency in item design.

The mean for the Improving component is significantly lower than the other scores and is the only score that seems to be a candidate for possible adjustment. It would be possible to adjust Improving scores upward by five percentage points in the hope of making it more strictly comparable, however this is not recommended. A likely interpretation of the difference is that the skills measured in the Improving section of the test are more advanced—a reasonable assumption since they involve personal development, which is a particularly high form of self-management. It probably makes sense for these scores to be slightly lower, and therefore adjusting the scores for comparison is not recommended.

In the sample population, 87% of respondents had one highest component score out of the five, and the rest had ties. Most of the ties were between two component scores. There was only one three-way tie. This result is helpful as it leads to cleaner interpretation and scoring in self-administered teaching settings, although there is no particular reason why ties should not occur.

In the sample, Relating was the dominant or highest component score most often (22.5% of respondents), and Improving was the highest score the least often (4.8% of respondents).

Highest Sub-score Percent of Respondents

This distribution seems appropriate to the objectives of the instrument and the domain of emotional intelligence. The relating skills are foundational; they are very basic communication skills (such as nonverbal reading of cues). Respondents were professionals at various levels in workplaces where basic social skills are always a prerequisite, so it is not surprising that scores might be higher for this dimension than the others. The Leading score is also the highest for approximately a fifth of the respondents. Note that this dimension does not correlate significantly with number of years spent supervising others. It is not about the formal aspects of management, and it is not specific to those with formal authority or power. Anyone may exhibit leadership skills in any group setting.

As discussed above, the Improving dimension is a relatively low scoring area for respondents, and this may be because self-improvement and active pursuit of feedback and learning tends to decline with adulthood. The average age of the sample was 41 years and no respondents were under 19. In any event, there is no design requirement for symmetry in score distributions. Instruments that measure personality type, for example, never aim for or achieve even distributions of type scores, and there seems to be no good reason to expect greater symmetry in an emotional intelligence instrument than in personality or type instruments.

So far, this analysis has focused on the data gathered in pre-testing and testing the instrument. Data analysis is helpful in a number of ways. It can show us that a new instrument behaves in statistically appropriate ways. Items should produce a range of responses, so that we can be confident that they are not biased too strongly toward one end or the other of the response scale. Overall scores should produce something approaching a normal curve, in order to make us confident that we are measuring something that is distributed normally as might be expected in the sample population. It is also encouraging to see minimal differences in the ways that component scores perform, and to see minimal gender and age differences. All these results are consistent with the design goal of creating an instrument that successfully measures a range of normal human competencies and differentiates respondents clearly and sufficiently.

In training settings, other factors are often of greater concern. Construct validity is often the paramount concern, since the test’s purpose in training is to engage students personally and in useful ways with constructs that relate directly to curriculum learning objectives. Further, our approach was pragmatic—we did not want to measure constructs that are difficult or impractical for the trainer and student to try to change during a typical course or workshop. There may be, for example, background learning in a curriculum about elements of personality. Unless these elements are related usefully to behaviors that students can practice, then it is not going to be useful to include them as constructs to measure in a learning assessment.

To make this point clearer, we could include a personality scale such as introversion-extroversion in an emotional intelligence assessment. It is included in the most widespread personality type instrument, the MBTi, so why not include it in an EI assessment too? Because we presume (based on consistency of retesting on this scale over many years) that people do not change from introverts to extroverts and so it would be silly to give them a score and suggest that they try to “improve” it over time. Yet it is probably true that extroverts have a slight advantage in many emotional-intelligence learning objectives, since they tend to be more social and more likely to be open, engaged communicators. This might make an interesting discussion point for a course on EI, but it is not a useful measurement for a learning-oriented assessment. Instead, our approach is to define behavioral competencies and to measure those instead.

Rather than defining extroversion as a component then, we might include something like “sharing your observations with the group” which would be further defined through behavioral examples such as ‘check your understanding of what someone is thinking and saying.’ And we might include nonverbal aspects of good interpersonal communications too such as ‘mirror some of the gestures, postures or expressions of others.’ These sorts of behaviors might come more easily to the extrovert than the introvert, or to someone who is highly empathetic than to someone who measures lower on an empathy scale. But who cares? The behaviors can be described and measured, and practiced and learned. Thus they can form useful (practical, learning-oriented) constructs for design of a behavioral-learning-oriented instrument.

It is interesting that the (fairly brief) history of emotional intelligence testing seems not be have been based on a clear and focused construct of emotional intelligence. The next section is a brief digression into the definition of emotional intelligence in the existing literature.

There have been some attempts to build broad-spectrum emotional intelligence instruments (such as the Bar-On instrument, now called EQ-I) that are more in the style of “Big Five” personality instruments or I.Q. tests than learning-oriented assessment instruments. The Multifactor Emotional Intelligence Scale (Mayer et al., 1988) is a good example of this historical type of test. These tests probably do measure elements of personality with a good level of internal validity, and elements of them can be shown to relate to or correlate with traditional personality tests as well. The problem with this approach is that it tends to produce something of more interest to academic researchers than to instructors and course participant—who want to know about practical, simple things they can do to improve their own and others’ performance experiences in their working lives.

For example, it can be shown that some people measure higher or lower on a scale of empathy. Yet this measurement is not highly useful because it is not something that can be altered by the individual or the trainer to any significant extent. Fortunately, almost all people have enough natural empathy that they can learn to apply it in specific contexts and in specific ways, in order to use their capacity for empathy productively and appropriately. A pragmatic view of emotional intelligence says that this is the goal. Therefore it is more appealing to design a test that measures specific behavioral applications of empathy, than one that measures it as if it were an underlying, and immutable, personality trait. As this example suggests, our approach to test design was to assume that most respondents are capable of acting in emotionally intelligent ways, and to focus instead on defining and measuring those ways of behaving, rather than measuring underlying personality traits that might have some (but not a determinant) influence on how we behave.

What this means is that there are severe limits to statistical testing of validity in the traditional style of IQ test design. It is entirely possible (using well-established, published scales) to measure things like impulse control that are often seen as components of emotional intelligence. If impulse control had been included in our instrument, we could then compare the performance of our scale with earlier published psychometric scales and attempt to prove statistically that we were measuring impulse control accurately. Yet what would be the point? There is, we know, a range on this scale in any normal population, just as there is a range of height. It is not necessary neither helpful in a training setting to give people measurements of these kinds even though they can be statistically validated. The measures do not have much if any impact on potential to perform well and in emotionally intelligent ways. The focus of our test is on interpersonal behavior, which is more difficult to validate statistically, but much more useful to measure and talk about in learning settings.

Goleman’s work with The Hay Group has produced an exhaustive 360-degree EI assessment that measures 21 components of emotional intelligence. They include an interesting mix of what we might think of as practical (reasonably easy to learn or change) components, and impractical components (embedded elements of underlying personality). [The instrument is based on Daniel Goleman’s 2002 book, Primal Leadership, and is called the ECI or Emotional Competence Inventory.] For example, Goleman measures organizational awareness, service orientation, and teambuilding, which are things that trainers feel fairly confident that they can help their students acquire. These are fairly practical constructs to measure and talk about in a learning setting, as it is possible to create durable and meaningful changes of behavior that might be measured in shifts on these scales.

However, the ECI also measures components that seem more durable, less behavioral, and as if they belong in a traditional personality test, such as emotional self-control, empathy, achievement orientation, self-confidence, and inspirational leadership (leadership skills can be taught, with the probable exception of inspirational leadership, which seems to be dependent on charisma and less on technique). Like other large EI inventories, this one seems to include a mix of learned behaviors and underlying personality traits, and to be very much kitchen-sink oriented (meaning that it seems to be designed with the goal of including as much as possible, rather than as little as is necessary). Our goal in designing IEI was to be practical and parsimonious, so as to create a relatively simple to take, simple to analyze, simple to facilitate, and simple-to learn-from instrument. It happens that this goal helped us achieve a clearer set of constructs and a higher level of construct validity.

Construct validity starts with clear statements of the constructs to be measured. The constructs in the IEI were derived from a broad reading of the literature on emotional intelligence. It is a new but already rich and far-reaching literature. The construct of emotional intelligence itself is rather broadly and fuzzily defined. This is probably because its history grew out of efforts to unify alternatives to IQ testing. Anything that can be measured in the domains of personality, interpersonal behavior, and intrapersonal behavior of normal adults seems to have been swept up into the loosely defined construct of emotional intelligence at some point. Daniel Goleman’s seminal work, Emotional Intelligence: Why It Can Matter More than IQ (Bantam 1995) cites hundreds of sources in fields as far flung as neuroscience, medicine, marriage counseling, group psychotherapy, anxiety disorders, early childhood development of empathy, and the brain circuitry of fear. It is a fascinating read, of course, but it is one of the least disciplined and focused works I have ever encountered. Such is the natural enthusiasm associated with the birth of a new, exciting domain of thought. Yet to try to measure every construct in that book would be an endless nightmare, not only for test designers but also for the poor subjects who were asked to take and interpret such a test.

Therefore, this and other books and articles on emotional intelligence (which have proliferated in the past ten years) were read with an eye specifically to constructs that met these (and only these) three criteria:

1. Interpersonal in nature (defined as largely focused on how adults interact with each other). Many elements of personality were ruled out when this criterion was applied.

Thus, a personality trait that seems interesting in a book on EI might not be included in our test if it failed to pass one or more of our screening criteria.

Some personality traits might be so determinant of interpersonal behavior that they would pass one or two of our screening criteria. For example, a tendency toward depression (which often correlates with measures of pessimism) can make someone seem difficult to work with (thus relevant to tests 1 and 3). Yet it fails our second test. Trainers are not going to make much of a change in an individual’s level of depression in a group classroom setting. There are of course ways to tackle depression, including therapy, drug treatments, and attributional retraining (self-talk)—such approaches might be useful to an individual but they are not going to be done by or with teachers or trainers, and the clinical diagnosis of depression should not be attempted with an emotional intelligence assessment in the first place.

When we applied these three criteria, we narrowed down the kitchen sink of EI components considerably. We still had a couple dozen loosely defined possible components, and so we organized them into natural (behaviorally and conceptually related) groups and nested them under everyday-language titles we assigned to these groups. Then we examined the vocabulary used to define each term and component, and where possible, replaced jargon-oriented or technical terms with more common synonyms. We also replaced trait-oriented language with action-oriented language, since it is more useful (for example) to describe a way to listen well to someone else, than to say that you have to be empathetic. Here is the resulting construct map:

Hearing

· Adjusting your message

· Listening fully

· Hearing the emotional message

· Including others

Contributing

· Explaining your feelings

· Speaking in turn

· Revealing yourself

· Sharing information

Relating

· Sharing your observations

· Relating nonverbally

· Treating others respectfully

· Supporting others

Improving

· Handling angry people

· Accepting criticism

· Giving constructive feedback

· Signaling intentions

Leading

· Asserting politely

· Playing a leading role

· Providing emotional leadership

· Motivating others

Each of these constructs was further defined with specific behaviors or actions. For example, “playing a leading role” was defined by actions such as feeling comfortable to speak publicly when called upon to do so, helping others who are in need (without holding back and acting like it’s not your problem), and asking others if there is anything they would like you to do.

When the constructs are mapped in such detail, it becomes reasonably easy to design items that correspond to the constructs and lead to high construct validity in an instrument. Both teachers and students who used the IEI in classroom/workshop tests agreed that it measured and taught these constructs clearly and (in hindsight after scoring their results) in a transparent manner.

It was interesting, and something of a surprise, to find that when we drew components out of the EI literature using our screening criteria, we ended up with a list that overlapped significantly with the interpersonal skills often included in so-called “soft skills” professional training already. Leadership skills, giving and receiving feedback, effective one-on-one and group communications, conflict handling in collaborative ways, and many other components are already included in many curricula. This suggested that the EI constructs in our instrument were already recognized as important to effective workplace behavior and that the instrument had high external construct validity as well as internal construct validity. (In other words, it not only measures its own underlying constructs but also measures constructs that are widely recognized as important training goals in a wide range of workplace and leadership settings).

An advantage of testing the IEI inventory in classroom/workshop (adult learning) settings is that we were able to obtain feedback from instructors and students. One series of workshops took place over a week and covered a wide variety of leadership skills, including a one-day module on emotional intelligence. The feedback questionnaire administered at the end of this week was intriguing. Almost all of the participants identified the IEI self-assessment as “what I liked best” in the coverage of emotional intelligence, and many identified it as the highlight of the entire course. The instructor feedback confirmed this survey finding. The instrument seemed to be popular and to generate a lot of “ah-hah” experiences in which participants became animated about ideas and insights and excited about ways they could improve their behavior in the future.

The popularity of the instrument in classroom tests seems to reflect two qualities. First, participants generally feel it is giving them accurate insights into their own emotional intelligence and behavior. They find the results plausible and interesting. Second, participants seem to draw useful, action-oriented conclusions in the form of take-away ideas about what to do differently and better in the future.

For example, here are some participant comments that typify the range of responses to the question, “If you could pick just one thing that you’d take home from today’s session and put into action, what would it be?”

· The areas I need to work on from the “Interactive Emotional Intelligence” test

These comments are derived directly from the items on the test, and when participants are identifying specific results from an assessment as their favored take-aways, it suggests that the assessment is giving them feedback and input that they find particularly useful in their working lives.

One of the reasons that participants find the IEI useful and engaging is that it is combined with a tightly related action planning form in the printed booklets used to administer it. Each of the constructs from the assessment is elaborated upon in this action planning form, so that students can select specific behaviorally-oriented competencies and develop their own action plans around them.

In initial testing of the instrument, we collected copies of many of these action plans from participants. In general, participants used them appropriately according to expected design goals:

Here are some representative examples of goals written by students in their action plans:

“Go out on a limb and try to draw outsiders in. this is scary for me! I like comfortable people I know and trust.”

“I do not share my personal life. Lifelong habit. I must try to be more open and sharing.”

As these examples suggest, the items in the IEI do seem to achieve the design goal of generating pragmatic, behaviorally-oriented, actionable goals that participants can take away and work on to help them feel they are more successful at relating in emotionally intelligent ways.

A subset of the sample population was asked to assess others in their workplaces, as well as themselves. This permits us to compare “self” and “other” scores. Is there a self-reporting bias of some kind? If so, it ought to be revealed by a consistent difference between self and other scores.

In the sample population, there is not a consistent difference between self-reported IEI and IEI as assessed by others. However, there is considerable variation. In some workplaces, most participants rated others significantly lower than themselves on all five-component scores. In others, the ratings were almost identical.

This seemed like a surprising finding, and we discussed it with instructors. They felt that it reflected the healthiness of the organizational culture of each workplace. Where participants felt like their workplaces were unhealthy and there was a lot of stress, tension, and interpersonal conflict, they rated others lower than themselves. Upon reflection, this finding makes good sense. IEI is probably a good indicator of the healthiness of the work group and the quality of its leadership. We interpreted this finding as a further indication of external validity of the constructs.