Thursday, February 21, 2013

Evaluating One-off Training
There are lots of questions about the relative value of one-off short courses as opposed to courses that are built into larger efforts to strengthen individual or organisational capacity. I'm trying to get to grips with how to assess effectiveness of such courses and here are some initial thoughts, also on EvaluatingTraining.


EVALUATING ONE-OFF TRAINING INITIATIVES
TO PROMOTE SOCIAL JUSTICE
Barbara Klugman
Strategy and Evaluation Practitioner
February 2013

This is an area I’m hoping to study further over time, but here are some initial thoughts…

One-off training unlikely to result in participants making many changes, depending of course on who they are and what the context is. But as a general strategic approach, training does better when it’s part of an ongoing process of support, networking, advocacy etc. Why? Because it allows the training to be located in relation to the context and actual work (whether voluntary or paid) of participants. This is particularly important because individuals often leave the training to go back into organisational contexts that are not open to the new learning the individual brings in. Hence training as part of organisational change, or development and implementation of organisational strategies is likely to be more effective.

What makes training work?


If you are offering a one-off training process, whether of a few days, or weeks, it’s worth locating it within a theory of change. What do we know about how to shift individual perspectives? Research evidence as to what factors influence individuals to change their practice[1][2] suggest the following theory:

Figure 1: Factors that influence behaviour change

Context / Environment

                                            Knowledge + Motivation + Skills = Behaviour Change



The Barefoot Collective refer to this as: head, heart and hands/feet. [3]

Box 1: Training aims to contribute towards enabling individual action through increased knowledge, motivation and skills
KNOWLEDGE in relation to
·  an issue
·  an approach (for example an approach to strategic planning, and organising, or an approach to social justice)
·  how the issue plays out in diverse contexts
·  what other players globally are doing and how to link with them
MOTIVATION
through
·  initiating or consolidating relationships of trust that can provide solidarity and / or links to resources
·  greater confidence in oneself and / or one’s organizational strategies as a result of positive reinforcement in the process
·  “widened consciousness” [4], experiencing oneself and one’s organization as part of a larger movement
·  positive emotional energy generated in the process
SKILLS in relation to any for example
·       how to do a situation analysis, plan or implement an initiative
·       personal capacity: how to do a public speech, how to supervise staff
·       orgnaisationsl stystems: how to set financial, information or other systems
·       movement-building: how to mobilise individuals, manage alliances, engage the media…
      

Knowledge


The type of pedagogy needs to enable people to participate in taking in and engaging new information to make it theirs – receiving inputs or doing reading is seldom enough on its own, especially for a non-academic audience.  Moreoever, if the information provided does not take account of its pertinence to diverse contexts (and your participants are from diverse contexts) then the programme needs to include time and methods for reflecting on relevance, adaptability to diverse socio-cultural, economic or political contexts. You may want to create opportunities to talk about an input or reading; opportunities for participants to share their own experience, or even write-up stories or case studies and then allow the group to reflect on these in relation to new theories provided in the training, or experiences elsewhere in the world. People should not be bombarded with information, irrespective of how creatively this is done e.g. with movies and stories. They need time to digest new ideas and information, including unplanned time, time to chat informally with others and so on.  They also need to have time to move about and exercise so that their bodies remain relaxed and minds remain alert.

Motivation


If the purpose of the training is to support social change, then participants leaving with new knowledge is not enough.  The way the training is run needs to build their motivation. This can be done in many ways, including through the excitement of meeting others doing similar work or concerned with similar issues and having time to learn about their experiences and build bonds with them; through opportunities to visit groups doing inspiring work and get concrete ideas about how to make change happen; through providing follow-up opportunities that will further build the participants’ capacities, links to others and so on. Pragmatically, motivation can be enhanced if the training results in accreditation, certification or some kind of formal recognition.

Skills


People are less likely to use lessons learnt, if they are not motivated and do not know what to do. Hence the more that training can help people work out how to do something differently, the more likely that it’ll be useful. This means creating opportunities in the training for trying out new ways of doing things and for planning how to do them in the home environment.

Who comes may influence the effectiveness of the training


We know that when an individual goes back to their home environment, unless it is a highly conducive one, the person may find it difficult to introduce ideas learnt through training. Some issues to consider when choosing participants include that the more people who attend from one place, the more critical mass they can bring both to shaping up ideas during the training, and to influencing others when they take these ideas to their home context; and those who are relatively junior in an organisation usually have less power to influence decision-making so that training people who have lesser organisational status means helping to identify realistic entry-points, or focusing on building their personal capacity rather than assuming they should make organisational changes. Also, bringing in a group with very mixed capacities and different styles of learning poses major pedagogical challenges.

Clarifying Your Theory of Change and How to Assess Effectiveness of Training


The ‘Theory of Change’ for one-off training suggests the need for careful planning of the process, firstly by establishing your own theory of change.  Is change in awareness enough? Is motivation enough?

What is it that you intend the training to achieve in relation to information – simply receiving information, or participants internalising the information? Are you hoping participants will shift their discourse, using new concepts and language? Are you hoping that they will analyse their own situations differently? You need to have clear objectives for each session of the training and for the course overall which you make explicit.  You can then work out how you will identify whether each objective has been realised cumulatively over the course. You could assess the degree of internalisatoin by the extent to which information shared is reflected in the contents of a project plan or some other product the participants are required to do during the training.

How will you build motivation? Once you’ve decided on what aspects of content, methodology, and the overall shape and style of the training will help to motivate participants, you need to keep an eye on whether this seems to be working during the course. You could assess motivation through an end-of-course questionnaire, but that will only tell you about the degree of motivation at the time of leaving the course. For this reason, it is essential to make an assessment some time after the course, to see whether the actions the person felt motivated to take at the end of the course have actually been taken.

Are you aiming to build skills? If the training is not focused on skills-building, then you need to consider what other opportunities participants will have whether within or outside of your overall programme to actually apply their knowledge.

If you are trying to deepen participant capacities to do things differently, you need to be quite clear as to what these skills are, to ensure that the training content actually provides the necessary inputs and opportunities for building and testing skills – and by monitoring participants in this process you will see to what extent they are harnessing the skills at the time.

While you can and should build in assessments throughout a training programme to see whether the shifts your theory of change intends you to achieve are actually taking place during the training, it is essential to see whether these remain over time, and whether participants are able to apply whatever has been gained by the training. You may be able to use a survey to gather very specific information, and supplement this by asking participants to share stories of changes they have made at some realistic period – perhaps three or six months after the training. 


By way of illustration:

·      You could ask: ‘Have you made any changes that drew on ideas you got during the course?’ yes no. If 'yes' please describe what you changed, when, where and with whom? …..
·      If you want to be more specific, which can help participants to reflect on their experience, you could do a few questions like this in a survey, in each case asking about a different kind of change, depending on what you think you can reasonably expect, for example one question could focus on 'changes in your organisation's or campaign's strategies'; another on 'in the way you talk with colleagues about (whatever the topic of the training was)'
·      If participants did any work on framing an initiative or project during the course then you can ask them to describe what they've done since then.
·      You could ask if they have had any contact with anyone they met during the course and if they say yes, ask whether this involved, and give them options like: sharing ideas; working together, and so on, and then ask them to give more details 

In conclusion, it’s worth thinking about the limitations of one-off training. Would it be better to bring the same group together repeatedly to build a really strong cohort with high level capacities, as opposed to giving basic skills repeatedly? Presumably the answer depends on what you’re trying to achieve.

One of the most profound impacts of training can come through the relationships built among participants which may allow them to provide each other with ongoing support, to strategise together, and thereby strengthen the overall efforts towards social change.  Establishing or reinforcing mechanisms for ongoing linkages between participants can be a key factor in enabling the training to make a difference once participants leave. Monitoring the extent to which such mechanisms are used and in what ways, can be another helpful way of seeing whether training is having its intended outcomes.

More thoughts to come as I work further with more groups on this issue…







[1] Fisher, W. A., & Fisher, J. D. , A general social psychological model for changing AIDS risk behavior. In J. Pryor & G. Reeder (Eds.), “The social psychology of HIV infection,” Hillsdale, NJ:Erlbaum, 2003:pp. 127–153.
[2] I have used the term ‘individual action’ where theorists usually refer to ‘behavior change’ since the latter term can be heard as problematizing participants and their current behaviour, whereas the intention here is to deepen people’s knowledge, motivation and skills in ways that strengthen their existing work. Thanks to Kate Hoshour, International Accountability Project for this insight.
[3] Barefoot Collective,  Barefoot Guide to Working with Organizations and Social Change, The Barefoot Collective, (published online), 2009:140.
[4] Lindell, I., ‘’Glocal’ movements: place struggles and transnational organizing by informal workers’, Geografiska Annaler:Series B, Human Geography, 2009: 91(2):123-136.

No comments:

Post a Comment