Section 6: Evaluation and validation of training

Section 4 provided us with the knowledge, skills and attitudes to develop and conduct training. The outputs of this activity were training objectives, tests and a course block syllabus. We now need to consider how we can validate and evaluate the success of this planning and conduct of training.

Evaluation and validation and two separate yet related things. You will find some overlap in their meaning, and as you read the professional literature you will find that some people use the terms interchangeably, while others have more rigid definitions. Don’t let this throw you, just try and make sense of what the purpose of their evaluation or validation was, how they went about it, and what the results were.

Evaluation assesses the effectiveness of an ongoing program in achieving its objectives … and aims at program improvement through a modification of current operations. (http://www.ojp.usdoj.gov/BJA/evaluation/glossary/glossary_e.htm). It is … “the process of reviewing the overall efficiency (did we do the right thing?), effectiveness (is the way we did what we did the best possible way?) and economy (did we get the best possible value for what we invested?) of a project. Evaluation also considers the alignment of a project’s outcomes to the programme’s objective(s). (Department of Infrastructure, Transport, Regional Development and Local Government, 2009).

Evaluation can be done during and after the project.

In general, validation is the process of checking if something satisfies a certain criterion. Examples would be: checking if a statement is true, if an appliance works as intended, if a computer system is secure, or if computer data is compliant with an open standard. Validation includes what is commonly thought of as testing and comparing test results to expected results. Validation occurs at the end of the development process. ( http://www.jiludwig.com/Definitions.html ) For our purposes, validation confirms the validity and appropriateness of the training content based on the analysed need.

The training cycle is a dynamic process in which we must continuously monitor the effective and efficient delivery of training. We can achieve this by preparing, maintaining and delivering a training validation program.

You may recall that we established that tests were a form of validation of our training plan and training conduct, and that they assisted the students by providing feedback progressively on their achievement. They also, of course, assist the trainer by indicating the trainees’ level of ability which can be measured against the training objectives.

These indicators tend to provide feedback only on the short term effectiveness of the training being conducted; they do not generally provide information about the short term efficiencyof training or the longer term effectiveness of it.

In this topic we will examine the development of internal and external validation programs and consider the ways in which these can be delivered. This is where we are in the training cycle.

Why conduct validation of training?

Before looking in more detail at the processes of developing validation programs let’s take a quick look at why we should validate training. It is important to note that we need to check both the short-term and long-term validity of the training we are conducting and that by validating training we should be able to measure:

• effectiveness - has the training prepared the trainee to perform the task to the required standard?
• efficiency - has the training made the most economical use of resources to achieve its objectives or have we wasted resources or duplicated effort?

We must also bear in mind the dynamic nature of training and the need to constantly update it as technology and other performance factors change.

Activity 3.14 Read Reading 14: Extracts from Australian emergency manual: Training management, Chapter 8, paragraphs 8.01 - 8.08 Web links: http://www.nwlink.com/~donclark/hrd/isd/validate.html http://econ.worldbank.org/external/default/main?pagePK=64165259&piPK=6416 5421&menuPK=64166093&theSitePK=469372&entityID=000094946_02010904 095571 last accessed 14/12/2005

Types of validation

There are three types of training validation:

Internal validation

Let’s now have a look at how we can develop an internal validation program. Firstly, we should look at the areas we wish to examine. What is it that we want to find out about? What is it that we need to know about the effectiveness and appropriateness of our training? Where will we get the information that we need?

Activity 3.15 Read Reading 14: Extracts from Australian emergency manual: Training management, Chapter 8 paragraphs 8.10 - 8.12. Chapter 8, paragraphs 8.19 - 8.21. The two areas mentioned in the reading for investigation through internal validation are: the trainees’ reaction or perhaps degree of acceptance of the training, i.e. is it relevant to their needs and is it well presented? the effectiveness of the training/learning, i.e. has their knowledge, skill and attitude been enhanced in the desired way as a result of the training? I would like to add a third area and that is: administration and logistics . Remember that we are trying to identify the efficiency of training. By examining the area of administration/logistics, we may be able to ascertain some resource savings. This may be in time, personnel, equipment, training aids, transport etc. Think about your training plan. What areas might need to be evaluated or validated? Both trainees and trainers can be reliable sources of internal validation data. Sourcing data from trainees is an obvious strategy, using trainers perhaps less so. How do you think that the views of the trainers and the trainees might differ? Print this activity

How can we conduct an internal validation?

We have already ascertained in Section 4 that tests are one way in which we can internally validate training; the other preferred ways are the use of questionnaires and training reports.

Activity 3.16 Read Reading 14: Extracts from Australian emergency manual: Training management, Chapter 8, paragraphs 8.27a, 8.27b, 8.28 and 8.29 http://www.ispi.org/pdf/suggestedReading/Carr.pdf

Questionnaires

Questionnaires are probably the best way, apart from tests, to analyse the success of training as it progresses. Questionnaires may be issued for completion at the end of each session/topic, may be delivered on regular occasions throughout a course or at the end of a course. The thing to remember is that you need to trap the data whilst the particular learning experience is fresh in the minds of both the trainers and trainees. The danger of not doing this is that respondents’ answers tend to become more subjective as their recollection diminishes.

Remember trainees can provide data on the following as a result of internal validation:

• subject/session difficulty?
• correct balance of theory and practice?
• time spent on subjects too long, too short, just right?
• like/dislike of subject matter?
• was there subject/session relevance to aim of course?
• was the subject matter presented in an interesting or uninteresting manner?
• were training aids useful or not?
• did trainees have pre-existing knowledge etc. of the subject matter or was it all new information?
• were administration and logistics handled well?

The structuring of questionnaires can be a complex matter if the required objective feedback is to be obtained. As a guide we can look at the following example of an internal validation questionnaire format. It should be remembered that each training objective should be validated whether this is done at the end of each session or progressively throughout the course.

We should stress here that this is merely an example and there are a number of other ways in which questionnaires can be written. You will have noticed that I have included such issues as motivation, i.e. why are they required to complete, and confidentiality. It is strongly recommended however, that you word your preamble to the questionnaire to ensure that you cover these two issues.

Training reports

The other method, apart from tests and questionnaires of obtaining internal validation data is to use training reports. It is usual in most training institutes for the trainers to provide progressive reports on their trainees. As indicated in the reading these can be a valuable source of validation data but should be compared to the trainees’ validation responses.

External validation process

Having looked at the internal validation requirements, we should now turn our attention to the external validation process. There are a number of ways in which we can carry out external validation such as exercises, questionnaires, on-the-job assessment/reports and interviews. Probably the most common methods, apart from exercises, which will be the subject of our next Topic, are questionnaires followed by interviews .

On-the-job reports or assessments, whilst in some ways ideal, as the validation team actually gets to watch performance, are very costly in terms of time and resources and are sometimes considered as an unnecessary intrusion by managers/supervisors.

Some important issues to consider when planning for and conducting external validation are:

• the sometimes subjective nature of responses to questionnaires;
• external validation should not take place sooner than three months after the trainees have completed their training - this allows time for them to put into practice the things they have been taught;
• you may need to sell the idea of the need to complete these questionnaires to the supervisors and managers. Essentially you need to convince them that their objective input will lead to a better and more productive employee as training is adapted to properly meet their on-the-job performance requirements.

Activity 3.17 Read Reading 14: Extracts from Australian emergency manual: Training management, chapter 8, paragraphs 8.22 to 8.23, 8.26, 7c, 8.27d, 8.28, 8.30, 8.31.

What questions should we ask in external validation questionnaires?

There is obviously a difference between the questions asked of ex-trainees and those addressed to supervisors. Remember that we are trying to determine how successful our training has been in meeting the requirements identified during job and task analysis and encapsulated in our training objectives. The questions should be related to every training objective that you are trying to validate.

There are, of course, many other questions you could put to both groups and these will depend on the type of course and the training objectives. Once you have received the questionnaires back from the respondents, collate the data and ascertain if there is any need for a follow up to clarify points of detail. If this is the case then it is advisable to confirm these by conducting interviews with those concerned.

Application of validation data

So, what do you once you have obtained feedback from your validation? Well, you now need to relate it to the phases of the training cycle and consider re-examining your job and task analysis and possibly amending your training plan and processes as a result. It would not be normal to take the data from one validation, either internal or external, to make major amendments to the training structure. It is better to flag potential problem areas identified in the initial validation process and compare them with the results of subsequent validations. The only obvious exception to this is when safety has been raised as an issue.

Finally it is always advisable to provide consolidated validation feedback to the responders, letting them know what actions you are going to take immediately or in the future.

Activity 3.18 Now that you have completed this section you can prepare evaluation documents for the Training that you have developed. See the Assignment Guide for further details. Print this activity

how to use this CD | site-map |© Charles Sturt University