Photo of people sitting at a lunch table talking

Evaluation Café 2022-23

Professional black and white photo of Ama Nyame Mensah
Beyond the Bar: Alternative Methods for Visualizing Change Between Two Data Points

September 14, 2022

Ama Nyame-Mensah
Founder and Principal
Analytics Made Accessible, LLC 
 

Learning to visualize data through charts and graphs is an effective way to communicate key insights to stakeholders quickly. When evaluators and applied researchers design data visualizations that illustrate the change between two data points, most default to the clustered (or grouped) bar graph. But there are so many choices out there! In this talk, several alternative methods for visualizing change between two points in time that go beyond the bar will be presented. This discussion will dig into this assumption, examining what we mean by each of these concepts, and exploring how the application of an intersectional feminist lens can in fact nuance and enhance our understanding of rigor in evaluation.

 

Professional photo of Jill Miller
Strategic Evaluation and Learning for Organizations

September 21, 2022

Jill Miller
Director of the Center for Applied Learning and Impact
IREX
 
A presentation focusing on evaluating impact and generational learning in support of strategic goals or priorities at any level of an organization, and a catalyst for building organizational capacity to more effectively and reliably evaluate results across programs. Strategic evaluation and learning requires investment - a clear vision for evaluation, a culture that fosters individual, group, and organizational learning, a compelling and cogent strategy, coordinated evaluation and learning activities, and a supportive environment. But when done well, it can play a key role in organizational success, producing insights needed to inform strategic decision making, advancing innovation, and demonstrating impact.
 
 

Professional photo of Brian Perron in a dark suit
Google Data Studio for Evaluators and a 'no-code' Data Science Toolbox for Evaluators

September 28, 2022

Brian Perron
Professor
University of Michigan, School of Social Work
 

Google Data Studio (GDS) is a free, flexible, and robust system for creating dashboards and interactive reports. GDS seamlessly connects with existing spreadsheets, making it an ideal tool for evaluators and community agencies seeking to maximize the value of their data. This workshop demonstrates how to use GDS to convert a collection of static agency reports into a single, interactive report that can easily be updated, extended, and shared. We also offer resources and suggested strategies to promote efficient skill development.

 

Professional photo of Sue Ann Savas
Most Significant Change Stories of Impact: A 2-in-1 Program Monitoring and Outcome Evaluation Method

October 5, 2022

Sue Ann Savas
Clinical Associate Professor
University of Michigan, School of Social Work
 
This session will introduce participants to the Most Significant Change method, developed by Davies and Dart. The qualitative method includes a 4-question protocol. Using one method, evaluators gain information to inform program monitoring and outcome attainment. Lessons learned from a number of data collection efforts with a range of stakeholders will be shared.    
 
 

Professional photo of Kathlene Butler outdoors
GIS and Remote Sensing Analytics

October 12, 2022

Kathlene Butler
Founder
GeoLiteracy, LLC
 
 
When we incorporate geographic information into our evaluations, we expand our perspectives on how program results are distributed, how recommendations are formulated, and how results can be improved. But geographic information and GIS analyses need not be complex or in-depth to provide value to the evaluation process. In fact, free, user-friendly resources exist to help us with geographic explorations of data, including tools provided in analytical software that many of you may already use, like Excel and Tableau. This talk will describe examples of using geographic information in evaluations and will introduce a few basic techniques, providing step-by-step instructions for their uses.
 
 

Professional photo of Margaret Roller ina n office
A Quality Approach to the In-Depth Interview Method

October 26, 2022

Margaret R. Roller
Founder
Roller Research
 

As the most frequently used qualitative research method, the in-depth interview enables researchers to explore complex issues and gain a contextually rich understanding of participants’ lived experiences. However, the complexities associated with conducting qualitative research – and, specifically, in-depth interviews – present unique challenges to researchers who strive to develop qualitative research designs that result in meaningful contextual data while incorporating quality measures that maximize the ultimate usefulness of their research. 

This session presents an approach that brings greater rigor to the in-depth interview method without stifling the creative approaches and interpretations utilized by skilled qualitative researchers. This approach is the Total Quality Framework (TQF) which is a comprehensive and principled yet flexible way to conceptualize, implement, and interpret qualitative research. This session discusses the TQF, focusing on the Credibility component having to do with data gathering. The discussion includes practical examples of how to apply the Credibility component of the TQF to the in-depth interview method.

 

Professional photo of Miriam Jorgensen
The Aesthetic Perspectives Framework, Art for Change, and Indigenous Nation (Re)building

November 2, 2022

Miriam Jorgensen
Research Director
University of Arizona
 

Indigenous nation (re)building occurs as an Indigenous nation strengthens its own foundational capacity for governance and self-determined community development. Much can be done to spur Indigenous nation rebuilding, from executive leadership training to inter-nation visits, community-wide strategic planning, work on constitutional reform, and more. But the actual results of these efforts—which can manifest in improved social, economic, health, and environmental outcomes—generally arise slowly and only over the longer term. How is it possible to know whether an Indigenous nation is moving along the nation building pathway? How can champions identify what is working and amplify those efforts?

 

Professional photo of Tiffany Tovey outdoors
Evaluation as a Reflective Practice: Practical Implications for the Field

November 16, 2022

Tiffany Tovey
Clinical Associate Professor and Director
Office of Assessment, Evaluation and Research Services
University of North Carolina Greensboro
 
Reflective practice has become an influential concept in the evaluation field. A greater use of reflective practice is advocated in reference to both evaluators' own professional development, and as a means to enhance dialogue, stakeholders' involvement and organizational learning in the evaluation process. The aim of this presentation is to examine the evaluation endeavor from the latter perspective, to present evaluation approaches which offer the opportunity for collaborative reflective practice. To this end, evaluation as reflective practice is discussed at three levels: (1) the organizational level, the model of single, double and triple-loop learning is discussed in reference to formative, summative and developmental evaluation, (2) the evaluator's level, different roles performed by the evaluator are considered from the point of view of promoting collaborative reflective practice, and (3) the broader socio-political level. 
 
 

Professional photo of Martin Reinhardt in front of a bookshelf
Using a Medicine Wheel Logic Model

November 30, 2022

Martin Reinhardt
Professor, Center for Native American Studies
Northern Michigan University
 
 

A medicine wheel logic model provides an approach for evaluation work that is rooted in Indigenous cultural frameworks. From the initial focus on identity to the final recommendations and intergenerational nature of each stage, this model challenges mainstream ideas and offers an alternative perspective that is much needed in this field.

 

Professional photo of Elvis Fraser
What Implications Do the Growing Discourses on Equity, Decolonizing Development, and Localization Have for the Practice of Evaluation in the International Context

December 7, 2022

Elvis Fraser
Founder and Managing Director of Sankofa Consulting, LLC
 

The focus on equity, especially racial and gender equity in the US have received supercharged attention, especially with the growth of the Black Lives Matter and Me Too Movement, and the protests that followed the murder of George Floyd. At the same time, primarily in the international development context, there is burgeoning discourse on the importance of decolonizing development and the need for localization. These three movements/discourses seem to converge around the importance of dismantling old power structures that are at the root of long-standing social and economic inequities.

 

The Synergy of Evaluation and Story

January 18, 2023

Corrie Whitmore
AEA President

 

Professional photo of Phillip Adu
Ensuring Credibility and Usability of your Qualitative Findings

January 25, 2023

Philip Adu
Founder, Center for Research Methods Consulting, LLC
 

Appropriately utilizing qualitative methodologies including data collection and analysis strategies, could yield rich results when conducting a program evaluation. As an evaluator, in order to ensure that stakeholders believe what you found (i.e., credibility) and that they use what you found (i.e., usability), you need to be transparent – showing your intent, decisions and actions in relation to the evaluation process. However, as a qualitative methodologist working with an evaluation team (with limited involvement in the design and implementation of the evaluation), how do you ensure credibility of the findings? Also, how do you ensure the usability of the findings when working with limited time and resources? In this presentation, I will share my experience as a methodology expert in the world of program evaluation and address the above questions.

 

Photo of Bri Christy
Considering your Professional Ethics and Values in your Data Work

February 8, 2023

Bri Christy
Technical Assistance Evaluation and Assessment Manager, University of Michigan
 
The first reason why we should be using data is that our professional ethical codes tell us to! Each profession is guided by their own ethical guidelines and values- as a social worker, my work is guided by the NASW’s Code of Ethics. Today we will spend some time together discussing why data is important in our professional worlds and the different ways we use data. 
 

 

professional photo of Jade Maloney
Bringing Lived Experience to Evaluation

February 15, 2023

Jade Maloney
Partner and CEO, ARTD Consultants

 

In policy, there is an increasing recognition of the importance of involving people with lived experience to ensure policy and services reflect their needs and support their goals. In mental health and disability, there is a commitment to nothing about us without us. For evaluation to operate in this way, we need to rethink the conception of evaluator as 'independent objective outsider', instead, engaging people with lived experience to inform all stages from design through reporting. But how do we do this well in practice?

This session will outline a practical framework any evaluator can use for engaging lived experience evaluators and lived experience advisory groups - including recruitment channels and processes, position descriptions and terms of reference, induction, training and support, options for roles and responsibilities, considerations of collaboration modes to suit different communication needs and tasks, managing power dynamics, and timelines.

 

Evaluating Agricultural and Agribusiness Transformation Through Private Sector Involvement in Driving Community Value Chains in Sub-Saharan Africa

February 22, 2023

Marian Tandoh Wordey
Agriculture Value Chain and Market Linkage Advisor

Business advisory support services are essential in creating sustainable and vibrant agricultural landscape and agro-processing industries in Africa. One commodity “Cassava” which has had poor image since its cultivation in Africa has metamorphosed into one of the most popular emerging industrial crop as a result of scientific research into its utilization.  The presentation will present the outcome of the research result, challenges, transformational interventions, the outcome and evaluation of the impact vis a vis the expectations and lessons learned.

 

What's a Girl Gonna Do: Understanding Evaluation Theory from a Postcolonial Lens

March 1, 2023

Brianna Hooks Singletary, Tyler Clark, and Malitsitso Moteane
Ph.D. Candidates, University of North Carolina-Greensboro
 
Evaluation theory is considered integral to good evaluation. There is, however, a lack of clarity on the distinction between prescriptive evaluation theory and evaluation approaches and perspectives. The distinction is further complicated by the central role of program theory in the praxis of evaluation. Notwithstanding, evaluation theory and theorists are popularly codified in the Alkin tree (Alkin & Christie, 2004, 2006, 2008) and presented in introductory evaluation classes in graduate programs. While the Alkin tree has seen several revisions, few female evaluators and even fewer evaluators of colour are represented. In this study we (three Black female evaluation graduate students) use PostColonial Theory as an analytical framework to interrogate and reflect on the teaching and learning of evaluation theory in introductory evaluation graduate classes. The paper concludes with suggestions for decolonizing the teaching and learning of evaluation theory within graduate evaluation programs.
 
 

Distilling Evaluation Practice into 43 Essential Tasks

April 5, 2023

Kelly Robertson, Principal Research Associate
Lori Wingate, Executive Director
The Evaluation Center, ϲʹֱ
 

The framework of delineates 43 evaluation tasks across seven broad domains of evaluation activity: Management; Engagement; Planning and Design; Data Collection and Analysis; Interpretation; Communication, Dissemination, and Use of Results; and Quality Review. In this session, we will (1) provide an overview of the task framework’s content; (3) describe the multifaced development and validation process, including challenges; and (3) discuss how it is being used to plan evaluation capacity development activities and resources by EvaluATE, the ATE program’s evaluation resource hub.

Teaching Evaluation Through Role Play

April 12, 2023

Michael Harnar, Director and Assistant Professor
ϲʹֱ
Kari Ross Nelson, Doctoral Student
Allison Prieur, Doctoral Student
 

In a PhD-level Foundations of Evaluation course, students are anticipated to gain an in-depth understanding of the origins and history of evaluation as well as its evolution toward an independent discipline. They should also develop a basic understanding of a wide array of evaluation theory and practice perspectives. In a changing world, deep understanding of the foundational theories vitally grounds evaluation professionals, helping them to be more responsive and inclusive. Targeting these objectives was the purpose of a mock debate between students posing as evaluation theorists. The cognitive complexity represented in the debate transcript shows that students can articulate theorist approaches and, when challenged by their classmates, occasionally compare, contrast, and evaluate different approaches.