Week 3: Evaluating Programs & Human Performance Technology

All too often instruction is developed with little thought as to how evaluation of learning or the effectiveness of the instruction will take place. When evaluation is considered on the front end of the instructional design process, it is often limited to evaluating whether the instructional design is more effective than traditional methods.

For this week's reflection activities, I would like for you to concentrate on the following:

• Chapters in Section III discuss evaluation in instructional design and provide you with three evaluation models: CIPP, Five-Domain, and Kirkpatrick’s Training Evaluation model. Search for at least two other models used for evaluation and summarize these models.

The Brinkerhoff’s Success Case Method is a specialized evaluation model that evaluates whether training is successful.  The Success Case Method (SCM) decides what works and what does not work.  The most integral aspect of the SCM model is that employees are part of every evaluative process, from front to back.  SCM uses case study and storytelling training programs, or to even evaluate some aspect of a training program.  Successful cases and unsuccessful cases are used in evaluating human performance.  There are five steps to the SCM.  First, the evaluator plans the study of the success case.   The purpose, definition, the budget, time frame, the nature of the success case, and the program are all discussed and decided upon with the employers who wanted the study done.  Second, the most important goals and outcomes to be expected are decided upon and written down in model form.  This is called an impact model.  The leftmost column depicts the capabilities, or learning outcomes desired from the program.  In the middle column, the conditions or environment needed to complete the business goals of the program are written.  The far right column is a list of the business goals that the program should reach.  Third, a research study is conducted where everyone in the company answer questions, without anonymity, in a questionnaire to figure out the most successful cases and the most unsuccessful cases.  Then, a random sample is chosen from the members of the company.  The impact model from step number two is a good place to draw questions for the questionnaire.  Fourth, long interviews lasting for 45 minutes are used to gain detailed, empirical information about the success stories so that you can storytell in the last step.  Questions that you might use could be asking what worked, what outcomes were achieved, what factors helped to achieve the success, is there room for improvement and what is the value of the program?  During calls to employers about unsuccessful cases, the issue is to find out what kept employees from being successful.  Fifth, the evaluation is written up.  The detailed report will have empirical evidence.  The report will also include the storytelling aspect discussed earlier where a detailed communique detailing the successes and the improvement needed to make the program more successful.  Finally, Brinkerhoff lists six questions to be answered about the final report ranging from, are some parts working better than other parts and what is the ROI (Return on investment) of the new program?  Brinkerhoff says that it is very important to talk about success cases if you want to have long-term success.

     The main goal of the Patton Utilization-Focused Evaluation (U-FE) is to make sure that the evaluation process is used.  The evaluator focuses on the use of the evaluation through every step of the process.  If the evaluation is not used, there is not a reason to evaluate according to Patton.   If the evaluator can get the employer to use the results of the evaluation process throughout the program, then evaluation will continue to be used after the evaluator has left. 

There are nine steps in the U-FE:

1.  With a readiness assessment, an evaluator can gauge the commitment of the organization to the evaluation process.

     
2.  Identify the people that will use the program the most and create relationships with them. 

3.  Conduct a situational analysis that will examine the potential problems, supports and the interest of the money people.

4.  Find out what the uses of the program are going to be and find a way to ensure that employers will use the evaluation after the evaluator has left. 

5.  Find out what the highest priorities and answer questions for the employer.

6.  Design an evaluation that will give realistic results.

     
7.  Collect and interpret the data.

8.  Make sure the results are used in the company to make efforts better.  The evaluator must keep working with the company until the results are used.

9.  Conduct a metaevaluation, meaning to find out if the evaluation is being used, and whether there were any different uses that were not expected.  If the evaluation results are being significantly used, then the evaluation is a success.

Describe how you would use them to evaluate your instruction.    

     I hate to say it but often teachers evaluate their instruction by how well it went.  I do believe that a teacher knows, after improvements, whether a lesson worked better than last time. However, I believe that if we had a systematized approach to evaluation for everyone in our geometry department, we could improve instruction.  Even now, we discuss successful and unsuccessful cases about lessons in our meetings.  However, they are very informal and there is no systematized evaluation.  I could see how we could do something similar to the Brinkerhoff’s Success Case Method (SCM) to see if our instructional methods are working or not over whole sections.  First, we would have to select an evaluator to create the case study.  Second, we would have to decide which goals and outcomes are most important.  This includes defining what a successful case and an unsuccessful case looks like.  Third, we would need to create an impact model so that we can visualize the direction we are going in.  We also would need to create a questionnaire that each teacher fills out so that we can decide which cases are successful and which cases are unsuccessful and then get a random sample from these.  Next we would interview teachers about how the section went, what was good and what was bad.  Then a report would be written.  At the end, we could use a staff development day to discuss the results fulfilling the storytelling arm of SCM.  The SCM sounds cumbersome, but over time, I can see how our curriculum could improve.  However, I am not sure how to create and apply the specific details for successful evaluation with this process.

Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. Should return on investment and management of resources also be considered in evaluations of instructional programs? What other measures should/could be considered?

     Learning and learner satisfaction are just part of instructional design.  There are so many factors that are weighed and important to stakeholders besides learning and learner satisfaction.  One area of instructional design evaluation is return on investment (ROI).  ROI contains within it, answers to a multitude of questions instead of traditional learning and learning measurements.  Some companies use ROI evaluations to justify existing budgets or new budgets.  These ROI evaluations help bring accountability to the instructional design evaluation process.  Other companies try to enhance their image and bring respect for the company by using ROI methodology.  If a company is not able to deliver, their image suffers.  ROI evaluation can help companies be sure they can deliver.  Executives often use ROI to show the value of programs they have instituted.  Executives are always under the gun to prove that the programs and initiatives they set forth were worth the time, money, and effort.  High ranking managers in a company might want to partner up, and ROI evaluations can help to see if one executive wants to work with another.  If the evaluations of someone’s department do not show high evaluations, then the executive might not want to partner with the other executive.  ROI evaluations can also help high ranking executives to get support for initiatives that are to be instituted by middle management.  If they can show with these evaluations how the new initiatives will help improve the middle manager’s bottom line, the executive can gain support for new programs.  As you can see, there are many questions that can be addressed by alternative evaluations besides traditional learning and learner satisfaction.        

    Since a company’s resources are limited, there are only so many programs they can put money toward.  Companies have to find value defined as costs versus monetary benefits.  Consequently, the programs and projects having the best ROI will be the ones that are funded.  The old ways of measuring results by the amount of activity the program creates are over.  The new generation of decision makers, called the “show-me” generation want to see data proving that their programs are creating value, monetarily and learning related.  Without observable value, companies will find other programs that will show them the data.  Programs that improve and streamline some operational aspect of the company might be successful but if the cost of the program outweighs the improved operation, then the program is a failure.  In the previous question, you can see how stakeholders benefit in many varied ways by using ROI methodology.  I would say a big, “Yes” to the question of the importance of ROI being a part of the evaluation of instructional programs.     
     Management of resources is very important to evaluation.  As I have stated before, resources are limited for companies when deciding which projects to do.  Also, there are so many types of projects, big or small, sprawled out over several cities, with limited resources, and people with different job descriptions to manage.  Tremendous value can come from having a good project manager that is skilled in managing, motivating, and leading people on their team.  The three broad categories of resources are people, time, and money.  The question is usually in deciding whether project outcomes or limited resources get the nod when making decisions.  All of these decisions have to be made whether demand exceeds supply, scarcity, or supply exceeds demand, inefficiency, and whether or not the economic cycle favors spending.  The management of resources is also an important function of evaluation.          

· Section IV focuses on human performance, performance support systems, knowledge management systems, and the concept of informal learning. Not all problems in learning and/or performance require an instructional one. Many times a non-instructional approach is a more appropriate solution.

· Identify a performance problem in your area of work and identify non-instructional solutions that may help solve the problem. Would better performance support systems, knowledge management systems, or opportunities for informal learning solve the problem?

     I started a process in our district where I started sharing curriculum materials that I made for algebra with all of the high school Algebra teachers in the district.  This grew into teachers sharing materials through emails.  Since then, I am now helping to create curriculum at the district level for geometry.  When we create curriculum for Mesquite ISD, we are trying to build a knowledge base where every teacher has more than one type of assignment, set of notes, quizzes, homework, activities, powerpoints, and projects that they can pick and choose from to teach their class.  The different lesson parts are over the same objective but allows the teacher the flexibility to fit their style.  Teachers are not all the same and like to teach differently.  We have been pooling knowledge and tweaking assignments every year to make them better.  Now, we keep the lesson plans in a program called Eduphoria so that all geometry teachers have access to lesson pieces and information.  This is a rudimentary knowledge management (KM) system.  We have added the KM idea of collaboration, as well.  Each high school has a geometry head, and their teachers meet every other day to discuss ways of teaching an assignment, new ideas, and they also create new curriculum.  Each head is now able to add to Eduphoria.  Instead of using email, we are now using Edmodo to share ideas with our teachers.  It’s like a wiki, and more useful than email.  We are obviously using the codification process to document our materials.  Even though our system improves every year, I feel like our KM system could be greatly improved.  I wish we could bring some experts out to our district to find a way to combine all the different software products that we use under one website.  Eduphoria is slow and cumbersome.  I almost don’t want to get on the website for curriculum.  We are also trying to add more technology related instruction into our curriculum.  We have two technology people that we can call in our district to help us, but I still feel we are limited.  I would like to get connected to national websites that geometry teachers use.  I haven’t found anything that seems useful.  I search around and find some things that I need.  But there isn’t a strongly integrated KM style of site that I can use.  I would love to work with others using a Web 2.0 style to create informal learning opportunities for geometry teachers around the country.  Another problem that we have in Mesquite ISD is that we have very few performance support systems included into our curriculum site.  I would love to have tutorials, or quick videos explaining the options for each lesson since many teachers will not search out each lesson for themselves.  There are a number of quick, technological support ideas that we could add.  I can see how performance systems, knowledge management and informal learning opportunities could bring about strong performance improvement in our district.