google-site-verification: googlefccb558c724d02c7.html


Thursday, December 15, 2011

Online Results-Based Management Training: The University of Wisconsin's Logic Model development course

The University of Wisconsin’s excellent online interactive Logic Model training is a valuable, easy to use introduction to RBM – and it is free.

Level of Difficulty:  Easy to moderate
Primarily Useful as: An introduction to RBM and Logic Models
Length: 3-5 hours if used online, 216 pages in the PDF format
Limitations: Some of the links are out of date

Who this is for

Like other consultants and trainers who work on international development issues, I usually work with groups, either donor or implementing agencies, training 10-200 people on RBM.  This reduces the cost (to them) of the training time, travel and other expenses.  But readers of this site or the RBM Training website sometimes ask if I can provide training on results based management  for individuals.  This would, however, be so expensive  that it would not be practical.  Other options such as enrolling in university courses in Europe or Canada might work, but often require more time, and again, more money, than an individual might be prepared  to invest.

But there is online, an excellent resource available to introduce RBM to anyone  who has no experience with RBM – or,  for that matter, to refresh even the most jaded RBM practitioner’s interest. This is the University of Wisconsin Extension Department’s Enhancing Program Performance with Logic Models.  Developed in 2002, and put online in 2003, there is very little in this easy to use and interactive course, that is not still relevant today, in 2011, to those who are trying to understand how to use results-based management.

Format: Interactive Online learning and a downloadable PDF

This course has both an interactive online format, and for those who don’t want to, or cannot use the internet for the 2-5 hours completion of the course takes, a PDF version of the course.  The real charm of this course, however, is that it is interactive, and it is the engaging nature of the interaction that cannot be replicated in the downloadable PDF.  Presentations, in the course’s 7 sections, both audio and text, are supplemented with pop-up windows which participants can view, if they want additional examples or references,  followed with exercises and then feedback on the answers.

University of Wisconsin Extension Online Logic Model Training
Copyright: 2002 Board of Regents, University of Wisconsin

It should not be surprising that the interactive elements are as engaging as I found them to be, because this site was developed by a team of content and technical experts, Ellen Taylor-Powell, Larry Jones   and  Ellen Henert at the Extension Department of the University of Wisconsin  – and extension departments are almost always the group in any university the most skilled in tailoring learning events to learners’ needs and learning styles.

Each of the course’s 7 sections contains the following elements:

A Section overview page, which includes

  • Introductory audio
  • Section learning objectives
  • Printable section outline to help users track their progress
  • Content presentation 
  • Activities that require us to put into practice the theory on the preceding pages.
A discussion of some technical limitations, and the reasonably good alternatives the course uses, are included at the end of this post.

Course Contents: Reinforcing important lessons about RBM

The course title is Enhancing Program Performance with Logic Models, but it deals with all of the associated elements of results-based management, not just the Logic Model.  Nothing in it will be a major surprise to people who work regularly with RBM, but the what this course teaches users about results-based management is worth repeating, simply because these lessons are often neglected or ignored, by even the most experienced users of RBM.

The course has 7 content sections, each of which generally takes between 20-40 minutes to complete.

What is a Logic Model?

Section 1 discusses the difference between Outputs and Outcomes, the need to test assumptions, and identify risk.  This section covers 20 screens in the course, and runs from pages 7-58 in the downloadable PDF. The difference is that in the PDF, transcripts of the audio from the online course, and a number of worksheets, are included as text.

This introduction to Logic Models includes, among much else, an interesting interactive Logic Model puzzle, requiring us to test our understanding of sequence, timing, risk and results, placing 20 different statement about a programme into one of 8 categories such as resources, activities, participants, Outputs, Outcomes, assumptions and external factors (or risk).

University of Wisconsin Logic Model course - interactive puzzle
Copyright: 2002, Board of Regents, University of Wisconsin
CLICK image to enlarge

This is not dissimilar to what you can do in a group with paper and scissors, but nevertheless engaging if you are working alone, and trying to ensure you understand the ideas of sequence, scale, change and other factors.

More about Outcomes

Section 2 takes us in detail through the different types of legitimate results that we can see in at individual, group, agency, system, or community levels, and talks about why participation of stakeholders in defining Outcomes is important.

This section includes material about RBM that more than one of the UN development agencies could usefully review.  Some of these agencies – not all, but some important UN agencies - get hung up on completing activities, (Outputs) and never seem to move on, in practice, to assessing whether these lead to any real change, or result (Outcomes).

Section two uses19 screens in the online course, and 32 pages in the PDF, with all of the associated supplementary material which is found in the pop-ups and links, in the online course.

More About Program Logic

Section 3  introduces (or reintroduces to forgetful RBM cognoscenti) the concept of clarifying the logic and assumptions implied in designing activities with the intention of contributing to a result.  It covers theories of change, the complexity of real-life programme logic, results chains – what it calls Outcome Chains -  and whether it is reasonable to claim causality – part of the current discussion on attribution of results to programme or project interventions.

It is important to note here, for those sometimes justifiably cynical about RBM who focus on complexity, that this course itself makes the point that multiple, often unplanned or unforeseen factors can affect results, not just the programme interventions. It also discusses the idea that a simple, linear logic model may not reveal enough of the factors involved in achieving results, and why more complex logic models may be needed to underpin the simple ones that we are often forced to submit to funding agencies.  

Another interactive exercise here, helps us to think through the theory of action behind a programme intervention, and despite the fact that this section has only 9 screens in the online course, and 24 pages in the PDF, the complexity of the interactive exercise on page 7 means it can take well 30 or 40 minutes to finish this section.

University of Wisconsin online Logic Model course, interactive exercise testing theories of action
Copyright: 2002, Board of Regents, University of Wisconsin
CLICK image to enlarge

This exercise is also an example of why, if you can, it is more productive in terms of learning to go to the online course first, before using the PDF, because with the PDF material there is essentially no chance to test your own understanding of the processes, before you see the “answer”.

What does a Logic Model Look Like?

Section 4 discusses why logic models, depending on the situation, purpose and culture involved,  can take many different forms.  There is no single format for Logic Models that suits all needs when the intellectual process of testing theories of change is involved – even though there may be, for any given donor, only one format they want to see.  There are, as the course points out, situations where Logic Models may differ in size and complexity, depending on whether they are being used to test ideas for programme planning, evaluation,  communication or for programme and project managers, the core of implementation activities.

Once again, note to complexity theorists, the course makes the point that logic models need not be simplistic, linear creations, but can be useful in helping agencies and individuals understand the complexity of the systems involved in interventions.

This section takes 8 screens in the online course, and 10 pages in the PDF.

How do I draw a Logic Model?

Section 5 is perhaps the most important of the course because it focuses on the real need to develop logic models as a group, not as an individual.

“As you work through this section, you will appreciate that the best way to construct a logic model is with others. While it may be quicker and easier to work alone, try not to. Many people believe that the real value of logic modeling is the PROCESS of creating one and the understanding and consensus that you build about a program as a result.”[Section 5, screen 1, PDF: p. 127]

I can only say I agree completely. In the Results Based Management training sessions I do with donor agencies, implementing organizations and national partners, feedback suggests that 80-90% of participants were absorbed by the Logic Model development process, sometimes surprised at how often the discussions reveal previously unknown or unacknowledged differences in perception among close colleagues, about what the original problem is, what risks and assumptions they have, and what reasonable results could look like.

5 different approaches to Logic Model development

All of the approaches focus on Logic Model development  a group.  The first four approaches are intended for planning new programmes.  These can be found at the online course, by clicking on the links at the bottom of the screen on creating a logic model for a new programme or on page 136 of the PDF.  All of these approaches begin with the result, and move back to activities and necessary resources.

  1. Start with the Long-term result or Outcome, and move back through mid-term or intermediate results, to short-term results, then to the Outputs needed to achieve these, then back again to activities and finally to inputs.
  2. Start with the long-term desired result, but then move immediately to activities, which is often the primary interest of participants, and then testing whether these will in fact, contribute to short of mid-term results which can logically relate to the long-term result.
  3. Start with the Long-term result, then brainstorm all of the elements that will affect this – activities, short, mid-term results, participants, risks, then sort them out to see if participants agree on the relationships and sequence.
  4. Juxtapose the situation or problem with a long-term desired change or Outcome then move  back through mid-term and short term results necessary to get there, and finally to participants, activities and resources required. This looks similar to the first option, but in fact really does, in practice in groups force those with preconceptions about what the activities should be, to confront the problem clearly, to look for results and then decide what activities and resources are needed.
  5. A fifth approach is also listed separately, on the next screen of the course, and this is for a situation where it is necessary to start logic model development with existing resources and existing activities. 
I have seen this happen where there is a second phase to a project, or where there is just too much institutional inertia to reconceptualize how to approach a problem.  It is essentially an approach focusing, as I see it, on a search for results to justify what is already being done – or as the this course suggests, where an “off the shelf” programme already exists:  Ask about an existing programme why each activity exists, what possible changes it can lead to, and how this can relate to a newly identified problem.
This approach is sometimes necessary, in my experience,  when working on RBM with universities, where activities such as degree course work and research are accepted as the core of university activities, and therefore the starting point for interventions.  It can also be necessary with some government agencies which  also sometimes see every problem through the paradigm of their own mandate and existing expertise.  
Moving such institutions towards a genuine questioning of what is really likely to achieve results is sometimes quite difficult.

How Good is my Logic Model?

Section 6 reviews some of the pitfalls individuals or RBM workshop facilitators may encounter in the development of Logic Models, including:

  • Getting lost in the RBM terminology,
  • Focusing too much on the mechanical aspects of putting activities and results in boxes, without assessing the plausibility of the connections between activities and results,
  • Focusing on – or complaining about – linearity, rather than exploring the complexities that some Logic Model formats can reveal,
  • Confusing the development of a Logic Model with evaluation, (for which it can indeed be a useful tool, but to which its utility is not limited)
  • Perceiving the Logic Model as a panacea for programme or project design or implementation problems, rather than as a tool to help us find possible solutions to such problems,
  • Focusing on production of the paper product, but never using it in practice.

It also makes the point that different people and agencies can use – or require the use of – Logic Models, for different purposes, and that we need to be clear about what this purpose is when we determine in how much detail we will work on Logic Model development.

This section spans 10 screens in the online programme, and 16 pages in the PDF version.

Using Logic Models in Evaluation  - Indicators and Measures

Section 7 reviews how understanding the original problem, assumptions, risk and the intended logic of the results chain can help agencies determine what evaluation questions to ask during different types of evaluations – questions about, and indicators relevant to:

  • The quality of inputs and completed activities (Outputs),
  • Who is participating, and what is the reach of the activities,
  • Assumptions underlying programme design, and selection of activities,
  • Whether results are actually achieved, and to what extent they can be reasonably attributed to the intervention, and
  • External factors –including what many agencies refer to as risk - on the achievement of results, or the failure to achieve results.

    Examination of a Logic Model and how it is formed can provide the foundation for

    • Needs assessments (assessing the original problem, and what can be done about it),
    • Process evaluations (assessment of inputs and the quality of activities)
    • Outcome and Impact evaluations (whether changes occurred and to what extent programme or project activities, and also external factors, may have contributed to such change).

    Section 7 is the longest of the course, with 20 online screens and 49 pages in the downloadable PDF.  The indicator discussions run from screens 9-20 in section 7, and pages 178-205 in the PDF.

    Glossaries and references on logic model development, RBM and evaluation

    The course, formally ends with section 7, but it is worth taking a look at other resources included:

    • The bibliography of references related to results based management and evaluation can also be reached by clicking on “resources” at the top of each screen or on pages 212-216 of the PDF.  The bibliography includes 72 references, 8 of which have clickable links that still appear to be functional.  Most of the 72  articles or references were written between 1994-2003, but it is worthwhile in particular visiting the Centers for Disease Control evaluation resources page which has a lot of very useful, accessible, and in some cases more current, guides on evaluation, logic models and data collection. The link in the bibliography is not current, but the page will automatically redirect to the new location, which I have provided in the previous sentence.

    • 11 additional links on evaluation issues such as questionnaire design, surveys, focus groups, quasi-experimental design and other issues can be found throughout section 7 online, or on page199 of the PDF.

    • Finally, the course also provides links to 15 downloadable logic model worksheets or hints, in PDF and sometimes Microsoft Word format, also under “resources” at the top of each page.  These can also be found scattered throughout the PDF document.

    Limitations - Technical issues:

    Because this course was developed in 2002, most people will have computers capable of making use of the interactive elements of the site – but there are three potential limitations to this, all of which, however, are dealt with in some way by the designers:

    1. A reliable internet connection is needed to use the site, and in many of the places I, and my colleagues have work such connections can be, at times, unreliable.  You might, for example, have a half hour of access to read things like this blog, but it would be frustrating to be interrupted in the middle of this online course.
    Alternative provided: Users can download the 216 page PDF version of the course, which includes the text of the whole course.

     2. Flash is used for the most compelling of the interactive features, such as drag and drop creation of logic models – and this, as I understand it, is unlikely to work for people using Apple products.  Most of the people I work with used Windows-based computers, and have the flash player installed, so this might not be a major problem, but the flash has to be enabled, and some people, and some network administrators, do disable it for security purposes.  The website provides a link to the free download of the flash player, from Adobe. Similarly popup screens and forms provide a wealth of additional detail in every section, and some web browsers may require users to enable these through security settings.
    Alternative provided:  The creators of the site provide an alternative to the use of flash, with some interactive elements, using links.  It is not as compelling from a learning point of view as using the flash elements, but it does permit some interaction.  

    3. The audio portions of the presentation use the “.ram” audio format, and a download of either RealPlayer or one of the alternatives such as VLC player  is required to listen.
    Alternative provided: When I first came across this course, in 2008, there was, essentially just the online version but in 2010 the PDF version was produced and it is a useful reference.  Some of the links in the document are out of date, but many more of them still work, and they are themselves quite useful.

    All of these, and some other technical issues are addressed also in the course online help link  and the Extension department provides an email link for those who may have further questions about the content or format.

    The bottom line:  This is an excellent introduction to results based management, focusing on logic model development, and its design is a credit to its authors' adult education abilities. It does not pretend to replace group workshops but it provides an intelligent, practical, and easy to use walk-through of the main issues in RBM.   PDF technology has moved on since 2002 and it is possible to use both flash and javascript now in PDF files.  It would be interesting to see what the digital media specialists at the University of Wisconsin could do with the PDF version of this course, to introduce some of the dynamic elements of the online course.

    Read the latest posts