Summaries

Borsboom et al. 2004, The concept of validity, psychological review

Construct Validity: The idea that a psychological construct as extraversion in salad, in a sense that it represents a real construct existing in the psychology of the individual. In this way, it must be measured in similar ways in repeated measurements, form different measurers, in different times, should have face value, should be correlated with other measures and non-correlated to others.

The authors suggest that in the past 50 years, the concept of validity has evolved, from the question of whether one measures what one intends to measure to the question of whether the empirical relations between test scores match theoretical relations in a nomological network, and finally, to the question of whether interpretations and actions based on test scores are justified.

The authors suggest a new, simpler, concept of validity: A test is valid if it measures what it purports to measure.

“If something does not exist, then one cannot measure it. If it exists but does not causally produce variations in the outcomes of the measurement procedure, then one is either measuring nothing at all or something different altogether.”

To make sure a test measures what it purports to measure, it needs to be based on a theory of response behaviour, and not on a theory about the relation between the attribute measured and other attributes.

To outline this concept of validity, the authors juxtapose it to the existing theory of validity in three domains, ontology versus epistemology, reference versus meaning and causality versus correlation.

Ontology versus epistemology

Ontology cares about the question of whether a phenomenon exists in reality and what are its causal influences. Epistemology instead cares about whether we can find out about reality. An epistemological activity in science is measurement. The authors claim that a truth in ontological field grant epistemological access. If the element exists, we can measure it. The authors put the ontological proof as priority. The current theory instead puts the accent on the epistemological question of measurement, without addressing the ontological issue. They further claim that there is no way to support the epistemological problem of finding universal true means of measurement of reality. In other words, there is no simple or working method to deduce validity from validation. Instead:

“The only thing that all measurement procedures have in common is the either implicit or explicit assumption that there is an attribute out there that, somewhere in the long and complicated chain of events leading up to the measurement outcome, is playing a causal role in determining what values the measurements will take.”

The authors claim that face validity, predictive validity, construct validity etc.. and all other forms and degrees of validity are not related to validity itself, but to validation. Validation is an activity, while validity is a property. Right now, validation is the activity that researchers undertake to find out whether a test has the property of validity.

In this sense, validity is a concept like truth: it represents an ideal or desirable situation. Validation, instead, is more like theory testing, so observing data in order to understand how to get answers. Validity is about ontology, validation is about epistemology.

They claim that literature has been focusing mainly on the problem of validation, and not on the problem of validity.

Of course, this is in contrast with the classical view of validity reported from other authors, as a “judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores”. So basically a totally different idea.

Reference versus meaning

The authors go on to differentiate meaning and reference. Reference, in this case, is a term that denotes the referring of some observation to “something out there in the world”. Meaning, in contrast, is a term that denotes a property of some observations. An observation as a “meaning” regarding a theory.

The authors propose that logical positivism has influenced negatively theory of validity, and one of those negative influences is the idea of nomological network, which the authors define as a system of laws related to observations, which for positivists sufficed to create meaning without reference for the theoretical terms. The meaning of a theoretical term is solely determined by the place of that term in the nomological network. In this way, one could invoke theoretical terms without automatically engaging in ontological claims.

This idea was used by Cronbach to construct the initial idea of construct validity. The idea of construct validity is one of agreement between the nomological network and empirical data. In other words, a test can be considered valid for a construct if the empirical relations between test scores match the theoretical relations between constructs. In construct validity theory, it is this match between observations and the theorized network that defines the validity concept.

In this case, observing empirical relationships that match the theoretical relations in the nomological network is all that is needed. The question of understanding if the construct is really present in reality is not of question.

The authors claim that this process is in contrast with the normal scientific approach to a problem. Drawing a network of relations and then defining constructs implicitly from this network (eg. intelligence can be drawn for a network of theoretical relationships) is considered a faux pas for the authors, who instead claim that this relationship cannot implicitly define constructs alone.

In this sense, the authors think that it is absurd to claim that the validity of a measurement (eg. IQ for intelligence) can be derived from the relation between the measured attribute and other attributes (eg. time spent in prison or number of degrees).

This scientific method of implicit validity cannot work especially in psychology, where theory is quite vague and not detailed enough to be of any real constraint. Psychology often works with higher and lower correlations between attributes. As often happens, such a loose network can be satisfied by an indefinite number of attributes besides the intended one.

The authors then specify that the referral strategy, eg. constructing tests that refer to “something out there in the world”, like intelligence, might face the difficulty that this connection, between referred attribute and test, might not work in the same way for different persons, contexts or situations.

In the end, the authors suggest that a reference interpretation for, say, an intelligence test, works under the assumption that there exists, in reality, an attribute that one designates when using the term intelligence. “The question of validity concerns the question of whether one has succeeded in constructing a test that is sensitive to variations in that attribute.”

Causality versus correlation

The authors point out another major flaw in the theory of psychological measurement, and that is the idea of criterion validity: namely, the fact that validity consists between a correlation between a test and a criterion. The main problem with this idea is the use of correlation and measures of similarity, which should be replaced by causality.

Causality can be extracted using, for example, the latent variable model that inserts directionality in the analysis. Without correlation, a test might “be valid for anything with which it correlates”. This is especially dangerous in social science, where everything tends to correlate with everything. Moreover, correlational values bring with them the danger of associating higher validity with higher correlation. Even further, perfect correlation would imply perfect validity, or in other terms the same construct under two different labels, following the idea of correlation as a validity measure. This of course is dangerous: imagine measuring the perfect correlation between thunder and lightning. Again, correlation has more flows, as being sensitive to the amount of variability in the attribute to be measured across populations. A population with no variance (all equal) will show a correlation of 0. The authors consider correlation a useful method, but not a good one to measure validity or to constitute validity.

Where to look for validity

The authors believe that right now, validity research has been driven in a top-down way, creating tables of correlation coefficients and checking whether these go in the right direction, or map to a specific theory.

Moreover, the current research direction is to create methods which have high predictive power. Unfortunately, it means to have many uncorrelated measures that correlate highly with the target criterion. Instead, items that measure the same item will be highly correlated between each other. So basically, the authors suspect that predictive power might not be helpful to validation (or might even be harmful).

The authors point out the need for a theory of what happens between the attribute and the test scores. In measurement construction, one has to start with an idea of how differences in the attribute will lead to differences in test scores. Therefore, the authors suggest paying more attention to the stage of test construction, instead of the emphasis on test analysis. In general, a coordinated pattern of theory, test construction and data analysis is suggested.

McAdams, 1995, What do we know when we know a person?

The author states that personality differences can be described at three levels. Level I consist of broad, decontextualized and unconditional constructs, aka traits. Level II consists of personal strivings, life tasks, defense mechanisms, coping strategies, domain-specific skills and values, and other motivational, developmental, strategic constructs that are contextualized in time, place or role, and are called “personal concerns”. Level III is utility and purpose, which is a hallmark of identity in western society, and is internalized as an evolving life story.

“In the professional enterprise of personality psychology, however, making sense of persons is or should be the very raison d’être of the discipline.”

In figuring out people, explanation should be preceded by description. One should be able to describe a phenomenon before it is able to explain it. We need to keep in kind that descriptions are usually interpretations. The author states that in this paper he will focus on description rather than explanation, for the fact that the description influences the explanation, and a good explanation is based on a good description.

“A clinician thinks of how a personality may be changed, orienting them towards the future”

There are some frameworks which can be used to describe a person. Allport, for example, in 1937 divided personality into cardinal, central and secondary traits. Cattell instead distinguished between surface and source traits, source traits then divided into ability, temperament and dynamic traits. Dynamic traits can be then divided into biological ergs, attitudes and sentiments. His goal was to predict behaviour.

Murray instead was more concerned with separating the conscious and unconscious terrains of personality. McClelland proposed instead that an adequate account of personality should contain stylistic traits (extraversion, agreeableness etc..), cognitive schemes (personal constructs, values, frames) and dynamic motives (e.g. need for achievement, power motivation). Mischel instead described personality through domain-specific variables like encoding strategies, self-regulatory systems and plans, and cognitive social learning person variables.

Traits

The author states that although traits are necessary for describing someone, just knowing traits is not enough. A person should be described by at least three loosely related levels of functioning.

The first level is traits, and the author states the advantages and qualities of a trait description of personality, as time stability, predictive power which goes toe to toe with situational effect. Nonetheless, the author also takes notice of common criticism to traits, as the lack of explanation, lack of precision and applicability only in score distributions in groups and not to the individual person.

The author states that traits are especially great in describing a person in their first impression, they work as an excellent “first read”. Traits, on the other hand, act as an unconditional indicator of general trends and have no place for what is called a conditional pattern in personality, for example “my dominance shows when my competence is threatened”, “I fall apart when people try to comfort me” etc..

Personal Concerns

“The two most valuable features of trait description—its comparative and nonconditional qualities—double as its two greatest limitations as well.”

As people get to know each other better and better, after traits, people start to understand highly conditional, contingent, contextualized and noncomparative information. So, individuals are no longer compared on a linear dimension. They move beyond traits.

There is a long list of domains where personality has not yet had access, as values, motives, defense mechanisms, coping styles, etc.. This whole set of collections of constructs makes up a second level of personality, which can be named “personal concerns”. Compared to dispositional traits, and are about what people want, specifically in which period of their lives and what methods do they use to get where they want to be and in which roles. Personal Concerns are contextualized in time, place and role and are usually defined in terms of future ends and goals. Traits, on the contrary, are not naturally conceived to be in goal-directed terms. An Extravert is just extravert, it is simply so, not in relationship with a goal.

Talking situationally, the author argues, personologists should seek information on the most salient settings and environments that make up the ecology of a person’s life and investigate the most influential scripts in that ecology. Important situational scripts can be social roles.

Traits are similar to movement on the x axis, personal concerns are the first derivative. The author nonetheless does not see eye to eye with me in this regard, McCrae & Costa distinguish between “basic tendencies” and “characteristics adaptations”, the latter being learned skills that are the results of interaction between personality and environment. In this way, personal concerns stem from traits. The author nonetheless proposes a hierarchy where smaller units are not nested into the larger units. In general the author suggests using a different descriptive framework for this level. Linkages with the first level should be investigated instead of assumed to be true.

Life Story

We live in a society..

The author states that modern democracies in the west pressure the individual to be at the same time fitting into the larger society and unique. The self should be defined as separated and connected, inviduated and integrated at the same time. A child only lives through the first two levels, and uses them to describe their personality entirely. An adult instead should use something else. The adult might come in touch with the question “Who am I”. The challenge of that question is to create a self which is coherent and unified, and despite the changes in time is kept coherent to some degree. The “Who am I” question can be answered with a story, a life story, a personal myth. This third level is, therefore, the level of the Life Story, and can be used to understand how and to what extent the person is able to find unity, purpose and meaning in life.

In the end, the first two levels provide little information on the real identity of the person, which is instead conserved inside the idiosyncratic life story’s narrative.

A story can be understood in terms of plot, imagery, theme, scene, setting, conflict, character and ending. A well-functioning identity in contemporary Western society is a “good story” exhibiting good traits as coherence, credibility, richness, openness and integration.

“Personologists who seek to explore Level III must become comfortable with the language of stories”

The “language of stories” resists taxonomization and is hardly evaluated using traits, types, syndromes, stages and other scientific nomenclatures. For example, is a narrative in the form of a “press release” for public consumption, providing superficial and socially desirable content? Some life narrations might even be hidden inside of us and closed to our awareness.

Overall these three levels seem to be loosely related, and the geography that connects them is unknown.

Summaries - Alberto Arletti