Our blog
Interview with Michael Quinn Patton
Charmagne Campbell-Patton
September 13th 2012
Michael Quinn Patton is a world renowned evaluation expert and member of World Savvy’s Advisory Council. He has been advising World Savvy on our evaluation methodology and implementation over the past four years. He also happens to be my father. So, a few days ago, I sat down with him to ask him a few questions about evaluating global competency and what the World Savvy network should know about this complex and interesting topic, especially in the context of the release of our 2011-12 evaluation report and global competency market research study.
You’ve been helping World Savvy take a developmental evaluation approach to our work. Can you talk about why you think developmental evaluation is appropriate for measuring global competency?
Great question. It seems to me that the idea of global competency is an idea in development. There is not already an established measurement instrument for it. It is a very broad concept; it’s what we call a sensitizing concept, as opposed to an operational concept, by which we mean that it’s something that is in development. It sensitizes you to something that seems to be important, but it’s not yet very tightly defined and may never be defined. Part of the ongoing developmental process may be to inquire into what does it mean in a changing world? Global competencies are not likely to be a fixed set of things. Global competencies 50 years ago would be different than they are today, or than they will be 20 years from now. So it’s an idea in development in relationship to a complex dynamic international context and that very inquiry lends itself to a developmental evaluation approach.
Can you talk about the opportunities and challenges you see in measuring global competency?
Well one of the things that makes a concept like this developmental is that it’s being realized through a number of different kinds of initiatives at the same time. The World Savvy Challenge is working with different age groups of kids in different cities with different kinds of themes. The Media & Arts Program is engaging students and teachers in approaching it through the artistic mode. The Bangladesh and Peru programs involve international exchange work. So these different kinds of initiatives working with teachers in different places and contexts, students in different contexts, part of the challenge is seeing both what is at the core of a concept like global competency and also what are the diverse manifestations of it on a continuum. Does it mean different things possibly for junior high kids than for high school kids than for kids that are having an experience in another country? So there are likely to be some core elements and some diversity elements. And the challenge is to capture both of those, and to use, as you are, multiple methods to do it. So you are working with quantitative instruments, you’re doing documentation, you’re doing interviews, kids are writing, they’re keeping journals, responding to questions. That form of what we call triangulation, using multiple methods to get at a basic idea, is very much a key methodological aspect of the credibility of inquiring into and evaluating a concept like global competency.
Evaluation and assessment are hot topics in the current debate over K-12 education reform. High-stakes testing remains the primary way that schools are measuring whether kids are progressing and achieving at the desired rates and levels. Where do you see global competency and measuring global competency within that context?
It is an important part of the current context. So first of all, any program that wants to compete for attention in schools, for credibility, for funding, will have to engage in some form of evaluation. World Savvy has come to that with openness and enthusiasm rather than resistance to make evaluation a developmental and learning experience. So on the one hand, there are the accountability sides of evaluation. On the other hand, there’s the learning side – how do we get better, how do we improve, how do we adapt the program over time? Those are really two different sides of evaluation that are often in conflict. The accountability side assumes very standardized results and a fairly simple kind of measuring the outcomes. The learning side assumes a complexity and diversity and individualism. So the challenge I think is to position yourself in that niche within education that actually appreciates a more whole and holistic view of the leaner, student, and teacher-student interaction that will value both the program and the kind of data that the program can offer. So that’s the match that you look for, is that the program and evaluation approach match. The program is developmental, the evaluation is developmental, and it will be a non-traditional form of accountability for that reason. It will resonate then with people who get the program – what it is, what it’s trying to do – beyond the basics of education. And that the evaluation is beyond the traditional accountability basics of simply did you achieve an outcome, to more learning, development and adaptation of the program and the ideas themselves.
What should people who are looking for evidence of global competency development know about the complexities of evaluating for global competency?
A part of the challenge of a program like World Savvy is what we call in the evaluation business is the issue of dosage. Fore example, 30 kids in a classroom that are all getting the same program in the same amount – a program like World Savvy different students are getting different amounts of the experience/program. They’re engaged to different degrees. You have the potential of a student who comes in at 7th grade and does 6 years of World Savvy. That would be the highest dosage in a way – a student who does 6 years of the World Savvy Challenge, is also in the Arts program, and does an international enhance. That’s huge dosage. Somebody else who does a poster one time as part of the World Savvy Challenge. Between those book ends is lots of variation of the intensity of the experience. The same is true of teachers. This has been very clear in observing my experience as a judge at the World Savvy Challenge. Those teachers who’ve now been in it 4 or 5 years, their students are showing a very different mastery of the subject matter, a very different degree of completion and learning because the teachers get it and they know how to teach it better and they understand what the process is. So in a way, the traditional evaluation model assumes that something like a program like World Savvy is a single defined intervention. But World Savvy is not a single thing. Different students experience different degrees of it, different teachers do. And so given that it’s a program in development, the thing I would say to funders, to people looking at the program, to educators who want to know about it, is to look at the higher end dosage for the results. Looks at the people who are getting more of the experience to see what is possible. That’s where you’re going to see, and are seeing, in both qualitative and quantitative data some major changes in students cognitively, in their skills level, in their knowledge of the world, and in their behaviors. And it’s that combination of the different dimensions of global competency that people who want to take a n in-depth look at it can take a good look at the data both qualitative and quantitative, that shows what impact is possible when the program is fully engaged.
Previous Entry
