Week 3: Evaluating Programs & Human Performance Technology

All too often instruction is developed with little thought as to how evaluation of learning or the effectiveness of the instruction will take place. When evaluation is considered on the front end of the instructional design process, it is often limited to evaluating whether the instructional design is more effective than traditional methods.

For this week's reflection activities, I would like for you to concentrate on the following:

• Chapters in Section III discuss evaluation in instructional design and provide you with three evaluation models: CIPP, Five-Domain, and Kirkpatrick’s Training Evaluation model. Search for at least two other models used for evaluation and summarize these models.

The Brinkerhoff’s Success Case Method is a specialized evaluation model that evaluates whether training is successful.  The Success Case Method (SCM) decides what works and what does not work.  The most integral aspect of the SCM model is that employees are part of every evaluative process, from front to back.  SCM uses case study and storytelling training programs, or to even evaluate some aspect of a training program.  Successful cases and unsuccessful cases are used in evaluating human performance.  There are five steps to the SCM.  First, the evaluator plans the study of the success case.   The purpose, definition, the budget, time frame, the nature of the success case, and the program are all discussed and decided upon with the employers who wanted the study done.  Second, the most important goals and outcomes to be expected are decided upon and written down in model form.  This is called an impact model.  The leftmost column depicts the capabilities, or learning outcomes desired from the program.  In the middle column, the conditions or environment needed to complete the business goals of the program are written.  The far right column is a list of the business goals that the program should reach.  Third, a research study is conducted where everyone in the company answer questions, without anonymity, in a questionnaire to figure out the most successful cases and the most unsuccessful cases.  Then, a random sample is chosen from the members of the company.  The impact model from step number two is a good place to draw questions for the questionnaire.  Fourth, long interviews lasting for 45 minutes are used to gain detailed, empirical information about the success stories so that you can storytell in the last step.  Questions that you might use could be asking what worked, what outcomes were achieved, what factors helped to achieve the success, is there room for improvement and what is the value of the program?  During calls to employers about unsuccessful cases, the issue is to find out what kept employees from being successful.  Fifth, the evaluation is written up.  The detailed report will have empirical evidence.  The report will also include the storytelling aspect discussed earlier where a detailed communique detailing the successes and the improvement needed to make the program more successful.  Finally, Brinkerhoff lists six questions to be answered about the final report ranging from, are some parts working better than other parts and what is the ROI (Return on investment) of the new program?  Brinkerhoff says that it is very important to talk about success cases if you want to have long-term success.

     The main goal of the Patton Utilization-Focused Evaluation (U-FE) is to make sure that the evaluation process is used.  The evaluator focuses on the use of the evaluation through every step of the process.  If the evaluation is not used, there is not a reason to evaluate according to Patton.   If the evaluator can get the employer to use the results of the evaluation process throughout the program, then evaluation will continue to be used after the evaluator has left. 

There are nine steps in the U-FE:

1.  With a readiness assessment, an evaluator can gauge the commitment of the organization to the evaluation process.

     
2.  Identify the people that will use the program the most and create relationships with them. 

3.  Conduct a situational analysis that will examine the potential problems, supports and the interest of the money people.

4.  Find out what the uses of the program are going to be and find a way to ensure that employers will use the evaluation after the evaluator has left. 

5.  Find out what the highest priorities and answer questions for the employer.

6.  Design an evaluation that will give realistic results.

     
7.  Collect and interpret the data.

8.  Make sure the results are used in the company to make efforts better.  The evaluator must keep working with the company until the results are used.

9.  Conduct a metaevaluation, meaning to find out if the evaluation is being used, and whether there were any different uses that were not expected.  If the evaluation results are being significantly used, then the evaluation is a success.

Describe how you would use them to evaluate your instruction.    

     I hate to say it but often teachers evaluate their instruction by how well it went.  I do believe that a teacher knows, after improvements, whether a lesson worked better than last time. However, I believe that if we had a systematized approach to evaluation for everyone in our geometry department, we could improve instruction.  Even now, we discuss successful and unsuccessful cases about lessons in our meetings.  However, they are very informal and there is no systematized evaluation.  I could see how we could do something similar to the Brinkerhoff’s Success Case Method (SCM) to see if our instructional methods are working or not over whole sections.  First, we would have to select an evaluator to create the case study.  Second, we would have to decide which goals and outcomes are most important.  This includes defining what a successful case and an unsuccessful case looks like.  Third, we would need to create an impact model so that we can visualize the direction we are going in.  We also would need to create a questionnaire that each teacher fills out so that we can decide which cases are successful and which cases are unsuccessful and then get a random sample from these.  Next we would interview teachers about how the section went, what was good and what was bad.  Then a report would be written.  At the end, we could use a staff development day to discuss the results fulfilling the storytelling arm of SCM.  The SCM sounds cumbersome, but over time, I can see how our curriculum could improve.  However, I am not sure how to create and apply the specific details for successful evaluation with this process.

Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. Should return on investment and management of resources also be considered in evaluations of instructional programs? What other measures should/could be considered?

     Learning and learner satisfaction are just part of instructional design.  There are so many factors that are weighed and important to stakeholders besides learning and learner satisfaction.  One area of instructional design evaluation is return on investment (ROI).  ROI contains within it, answers to a multitude of questions instead of traditional learning and learning measurements.  Some companies use ROI evaluations to justify existing budgets or new budgets.  These ROI evaluations help bring accountability to the instructional design evaluation process.  Other companies try to enhance their image and bring respect for the company by using ROI methodology.  If a company is not able to deliver, their image suffers.  ROI evaluation can help companies be sure they can deliver.  Executives often use ROI to show the value of programs they have instituted.  Executives are always under the gun to prove that the programs and initiatives they set forth were worth the time, money, and effort.  High ranking managers in a company might want to partner up, and ROI evaluations can help to see if one executive wants to work with another.  If the evaluations of someone’s department do not show high evaluations, then the executive might not want to partner with the other executive.  ROI evaluations can also help high ranking executives to get support for initiatives that are to be instituted by middle management.  If they can show with these evaluations how the new initiatives will help improve the middle manager’s bottom line, the executive can gain support for new programs.  As you can see, there are many questions that can be addressed by alternative evaluations besides traditional learning and learner satisfaction.        

    Since a company’s resources are limited, there are only so many programs they can put money toward.  Companies have to find value defined as costs versus monetary benefits.  Consequently, the programs and projects having the best ROI will be the ones that are funded.  The old ways of measuring results by the amount of activity the program creates are over.  The new generation of decision makers, called the “show-me” generation want to see data proving that their programs are creating value, monetarily and learning related.  Without observable value, companies will find other programs that will show them the data.  Programs that improve and streamline some operational aspect of the company might be successful but if the cost of the program outweighs the improved operation, then the program is a failure.  In the previous question, you can see how stakeholders benefit in many varied ways by using ROI methodology.  I would say a big, “Yes” to the question of the importance of ROI being a part of the evaluation of instructional programs.     
     Management of resources is very important to evaluation.  As I have stated before, resources are limited for companies when deciding which projects to do.  Also, there are so many types of projects, big or small, sprawled out over several cities, with limited resources, and people with different job descriptions to manage.  Tremendous value can come from having a good project manager that is skilled in managing, motivating, and leading people on their team.  The three broad categories of resources are people, time, and money.  The question is usually in deciding whether project outcomes or limited resources get the nod when making decisions.  All of these decisions have to be made whether demand exceeds supply, scarcity, or supply exceeds demand, inefficiency, and whether or not the economic cycle favors spending.  The management of resources is also an important function of evaluation.          

· Section IV focuses on human performance, performance support systems, knowledge management systems, and the concept of informal learning. Not all problems in learning and/or performance require an instructional one. Many times a non-instructional approach is a more appropriate solution.

· Identify a performance problem in your area of work and identify non-instructional solutions that may help solve the problem. Would better performance support systems, knowledge management systems, or opportunities for informal learning solve the problem?

     I started a process in our district where I started sharing curriculum materials that I made for algebra with all of the high school Algebra teachers in the district.  This grew into teachers sharing materials through emails.  Since then, I am now helping to create curriculum at the district level for geometry.  When we create curriculum for Mesquite ISD, we are trying to build a knowledge base where every teacher has more than one type of assignment, set of notes, quizzes, homework, activities, powerpoints, and projects that they can pick and choose from to teach their class.  The different lesson parts are over the same objective but allows the teacher the flexibility to fit their style.  Teachers are not all the same and like to teach differently.  We have been pooling knowledge and tweaking assignments every year to make them better.  Now, we keep the lesson plans in a program called Eduphoria so that all geometry teachers have access to lesson pieces and information.  This is a rudimentary knowledge management (KM) system.  We have added the KM idea of collaboration, as well.  Each high school has a geometry head, and their teachers meet every other day to discuss ways of teaching an assignment, new ideas, and they also create new curriculum.  Each head is now able to add to Eduphoria.  Instead of using email, we are now using Edmodo to share ideas with our teachers.  It’s like a wiki, and more useful than email.  We are obviously using the codification process to document our materials.  Even though our system improves every year, I feel like our KM system could be greatly improved.  I wish we could bring some experts out to our district to find a way to combine all the different software products that we use under one website.  Eduphoria is slow and cumbersome.  I almost don’t want to get on the website for curriculum.  We are also trying to add more technology related instruction into our curriculum.  We have two technology people that we can call in our district to help us, but I still feel we are limited.  I would like to get connected to national websites that geometry teachers use.  I haven’t found anything that seems useful.  I search around and find some things that I need.  But there isn’t a strongly integrated KM style of site that I can use.  I would love to work with others using a Web 2.0 style to create informal learning opportunities for geometry teachers around the country.  Another problem that we have in Mesquite ISD is that we have very few performance support systems included into our curriculum site.  I would love to have tutorials, or quick videos explaining the options for each lesson since many teachers will not search out each lesson for themselves.  There are a number of quick, technological support ideas that we could add.  I can see how performance systems, knowledge management and informal learning opportunities could bring about strong performance improvement in our district.   

Week 2: Theories & Models of Learning & Instruction

This section of the book presents various theories and models that form the foundations of instructional design and technology, including the evolution of approaches to instruction and learning over time. In your blog for this week, reflect on the following:

1. Epistemology (the study of what and how we come to know) is discussed in multiple chapters in this section. Distinguish epistemology from instructional methods or theories. What are the differences between theories, methods, or models of learning and epistemologies or underlying beliefs about ways of knowing?

     Dictionary.com says that epistemology, “is a branch of philosophy that investigates the origin, nature, methods and limits of human knowledge.”  After studying the definition, I would say that epistemology is used in creating instructional methods or theories.  Instructional methods or theories are a system of connected beliefs that are used to instruct or guide people and help them learn and develop.  Epistemology wants to know, “how do we know?”  Epistemology is the how, why, and what of learning.  Theories, methods, or models of learning use what are believed to be related truths to build and design a systematic way of instructing and helping people to learn.  Epistemology is a “know that” concept, whereas theories, methods, or models of learning are a way of showing you how, to “know how.”  However, there is a difference between learning models and learning theories.  Learning theories are a widely accepted set of formalized and tested concepts that are used to help people to learn.  Some of the most well-known learning theories include behaviorism, constructivism, pragmatism, and idealism.  However, models, often shown in flow chart form, give a systematized rigorous process to designing and improving instruction.  Many learning models follow some, or all, of the elements of the ADDIE model whose five parts are analyzing, designing, developing, implementing, and evaluating. 

2. Chapters in this section discuss three contrasting epistemic stances: positivist, relativist, and contextualist (or hermeneutical). Positivists believe that the only truth or knowledge is objective truth. Relativists don’t believe that objective truth is possible and that all knowledge is subjective to perception or relative to a particular frame of reference. Contextualists believe that truth or knowledge is relative to context rather than individual, subjective understanding. While designers and educators with a positivist stance generally apply behaviorist principles to the design and development of instruction, those with either a contextualist or relativist epistemological framework employ constructivist theories and methods. Reflect on whether your stance is primarily positivist, relativist, or contextualist. Then, identify an instance when your perspective or stance as a learner conflicted with that of your instructor. Describe the conflict that you experienced and analyze whether opposing epistemic stances may have been at the heart of the conflict.

     I hate to say it, but I’m a positivist by nature, and by the nature of the subject that I teach.  I’m a high school math teacher and 2 + 2 = 4.  However, I do hold out that some facts do change over time.  I am part of a team, at the district level, that creates Geometry curriculum.  Over the years, we have gotten better at creating projects and activities that are based on constructivism.  We don’t use every aspect of constructivism because of the sheer amount of curriculum we are expected to teach to prepare students for the STAAR test.  However, motivation is much higher when you try to introduce more facilitated learning.  We try to have at least one activity per day that allows students to collaborate and work authentic, real-world problems. 

     I haven’t had too many conflicts as a learner.  I had a theology teacher that never took a stance on any aspect of the Bible.  He would just talk about the relativistic possibilities of every position concerning an issue.  I guess the positivist in me hoped for the possibility of some objective, or absolute truth.  However, he did make me think.

3. Differing epistemic stances lead to differing approaches to learning and instruction, and ultimately to problem-solving. Explain differences in problem-solving when approached from behaviorist and constructivist perspectives. How do the approaches differ in both the nature of the problem to be solved and in facilitating the problem solving process? Finally, what effect might these differences have on learner motivation?

     Problem-solving using behavioristic learning has to be understood, explained, and predicted entirely by observable events, or empirical data.  Behavior is observed both before and after an instructional intervention.  If nothing changes, then the instruction was worthless and needs to be changed.  If the targeted behavior occurs, then it is considered learned.  If the behavior is repeated, then it is considered learned.    Therefore, the pretest, instruction, and post-test triplet is an example of behavioral learning.  The importance of immediate feedback is a hallmark of behavioral learning.  Computer-programmed instruction using behavioristic models was popular for a while.  However, this type of computer-based learning waned as people were bored by the mundane process.  In behaviorism, problem-solving is teacher-led.  The environment and resources are totally controlled by the teacher.  The hope for strong student motivation is based on the idea that instant feedback would allow a student to either be excited about his success, or quickly change an incorrect response.  The belief was, after a series of successes, students would be motivated to learn more.  However most people were bored and unmotivated by a process they did not get to own.

     Constructivist-based problem solving is student-centered and requires the teacher to be a facilitator, a “guide on the side.”  The student receives a large amount of autonomy, so the student is self-directed.  However, the student would also be able to collaborate with others in solving the problem.  The student is given the ability to choose which materials and resources they would need to solve the problem.  The problem that they would solve would be authentic.  Authentic meaning a real-world problem that would mirror a student’s life, now, or in the future.  For example, the problem could be based on a job that students might have in the future.   Authentic problems are closer to the world students live in, and closer to how they think.  Students are also encouraged to set their own goals.  After they complete the problem, learners are encouraged to reflect upon the outcomes of the process.  Experience plays a major part in constructivist thought, as well.  If the process is done right, and set up well, students are highly motivated to construct their own learning.  However, there are some problems that can occur with constructivism.    Students may be disillusioned as they walk around in a daze trying to figure out a problem that’s too hard.  The problem may also be too easy and very little learning happens.  Discipline management problems could occur to derail the process, as well.  For instructors, it is difficult to create a constructivist problem solving venture.  Also, the activity may not hit targeted learning, accessibility to the process may only benefit the privileged, and benefits can sometimes be very hard to measure.  To me, constructivism has a definite place in the curriculum.  I believe constructivism is better than behaviorism, it’s just harder to create, and harder to make happen. 

Week 1: Defining the Field

The first three chapters of your book define the IDT (Instructional Design and Technology) field and provide a history of how it has evolved over time. In you blog for this week, reflect on the following:

1. How do the definitions in the first chapter compare to your own definition of instructional or educational technology? 

I had not really thought about it much but I guess my definition was similar to the interim definitions the book discussed where the field was really “the media” or equipment/technology used to promote learning.  I did not see the instructional design aspect of instructional design and technology (ID&T).  Also I did not know about the intricate process, from start to finish, of instructional design.  The definition of educational technology was foreign to me. 

What experiences or other influences have shaped your definition?

Technology that I have used has mostly been an instrument to show the curriculum.  For example, I have used computers, ELMO’s, overhead projectors,  Interwrite pads, and software like Powerpoint to show curriculum.   Also, back in the 90’s at McKinney High School, we had a criterion-referenced machine that we used for TAAS courses.  If a student passed the pretest, he could test over the objective.  If he passed according to the machine, he could go to the next objective and start the process all over again.  Students could self-pace through the curriculum.

How has your definition changed from examining the definitions in the first chapter of this book?

I can now see the intertwining of instructional design and technology as an in-depth process including analysis, design, development, implementation, and evaluation that improves learning and performance.  There is also a revision process that moves back and forth along the continuum of the aforementioned steps, making the path of creation a nonlinear one.  This path is not really different from the process used to create curriculum without technology which I guess is an aspect of instructional design.  The hardware and software, or technology, is only one side of instructional design and technology.  I am also now aware that instructional design is used in instructional and non-instructional situations, in education and in the workplace.  I now know that systematic design methods are often used by people in the industry.  And finally, I see that good instructional design is student-oriented, goal oriented, focused on meaningful performance, can be accurately measured, is empirical, iterative, and self-correcting, and is usually a team effort.  My definition was lacking.         

2. Next, think of a lesson or unit of instruction that you have developed. Or if you haven’t ever taught or developed instruction, think of one that you have received. How does that lesson adhere or fail to adhere to the six characteristics of instructional design?

Last year, our Geometry team created a lesson for finding the distance on a coordinate plane.  There are two methods:  using the distance formula or using the Pythagorean theorem.  The distance formula uses two coordinate points to find the distance between the points on a coordinate plane.  For years, we had noticed how students had a hard time putting the correct coordinates into the formula so we created a pair activity to help them.  We had them cut out pre-designed numbers and place them into two coordinate points below an enlarged distance formula.  The first student would then slide the numbers into the correct locations in the formula and the second student would use the calculator to calculate the distance between the two points.  The student with the calculator could also help the student sliding the numbers.  Then after showing them the answer, the students would switch places on the next problem.

We started out the lesson with a powerpoint showing the numbers actually moving from their coordinate locations into the distance formula.  This was the method we used in the past where we only showed them how.  However, even with the nice powerpoint, with color and movement, many students were unable to do what seemed an easy task.  However, the simple hands-on activity proved beneficial to students trying the task. 

Our lesson fulfilled most of the characteristics of instructional design.  The activity was student-centered as they were able to help each other with the tactile process.  Our team’s goal was to find an activity that would simplify a process that had caused problems in the past.  Our previous lesson was only marginally student-centered with the bells and whistles powerpoint.  And previous to the powerpoint, we just showed them how to use the distance formula on the board, or on an ELMO.  I believe students were enabled to have meaningful performance with the simple activity.  The following quiz showed vast improvement over the previous year.  We were not able to measure in a reliable and valid way because we didn’t have exact results from last year.  However, all of teachers agreed that the scores were better, on the quiz and following test, than last year based on our empirical evidence.  More students were able to perform the task easily than in previous years.  Our lesson was iterative and self-correcting in the sense that our team revised a problematic process over a several year period until we came up with the right formula for success.  We have a great team, and definitely worked together to improve this assignment.  Even though we have failed many times in the past to find the answer to curricular problems, we succeeded on this lesson. 

How would you redesign it to better adhere to the six characteristics?

I think the best way to improve instructional design, in this case, would be to keep exact records so that we can make comparisons in the future.  This is a task that we have already started.  I think that we followed the other characteristics fairly well. 

3. In the 3^rd chapter, Reiser distinguishes instructional media from instructional design, excluding teachers, chalkboards, and textbooks from the definition of instructional media.

Why?

From what I read, Reiser did not exclude teachers, chalkboards, and textbooks from the definition of instructional media.  The author of the book seemed to be in favor of including teachers, chalkboards, and textbooks but for the sake of describing the history of media, the author decided to exclude the “big three.”  I’m not sure how to answer since Reiser didn’t seem to exclude them from the definition of media.   

Would you consider teachers, chalkboards, and textbooks instructional media?

I would consider chalkboards and textbooks instructional media since they are inanimate tools to enhance instruction.  However for me, teachers don’t fall into the category of pure media since they have a mind and are capable of moving past the behavioristic tendencies of an automated device.  Teachers, can think, judge, change positions, and the results can be different for each situation.  I know I state the obvious.  Teachers do not fit the definition of “physical means” for media.  The cognitive changes the equation.

Is the purpose of instructional design to incorporate media into instruction?

Instructional design does not always incorporate media into instruction.  The areas of media and instructional design overlap but are not mutually inclusive.  There are areas of instructional design that are used without media.  The process of instructional design has been traced back to World War II.  Psychologists and educators were employed to help create training materials for the military.  An example of how instructional design is not always used with media can be shown by the problem of high failure rates in a particular flight training program during World War II.  Instructional designers created a pretest to grade the psychomotor and perceptual skills of potential candidates.  This test helped to improve results for candidates entering the flight program.  The test was an example of instructional design without the use of media.