General Overview of Evaluation Criteria

Evaluation Criteria Explanation Data Sources
Effectiveness: Mastery of WBI goals Determine if students are able to apply concepts and procedures to create an online marketing strategy for an organization. 
  • Students' final marketing strategy document and presentation.
  • Students' final peer reviews.
  • Pre and post test on terms, concepts and procedures.
  • Practice exercises in WBI.
  • Student Discussion forums
  • Quizzes and midterm averages. 
Efficiency: delivered in a timely and logical manner. Material is accessible to all students. 

Determine if there is sufficient time for students to create, upload and reflect on assignments.

Determine if content formats are compatible with retrieval in LTM. 
Assess flow of content and efficiency of directions in use.

Assess instructor workload in troubleshooting, facilitation and feedback

  • Informal QM standards review for online course design (reviewed according to rubric for required standards only ). 
  • Document instructor time spent on answering logistical questions about course; preparation; facilitation; feedback/scoring.
  • End of course survey 
Appeal: Gain and maintain learner attention and interest

Review multimedia for interest.

Review technology for optimum user experience.

Review content for relevance to  use in workplace situations

  • End of course survey
  • Expert review
  • enrolled/completed ratio

Evaluation Matrix

Criteria & Categories Specific Questions Methods and Tools
Effectiveness
Goals

Are the goals and objectives clear?

Are the goals achievable by students?

Are the goals achievable within the course time frame?

  • QM standards checklist
  • Expert review
  • Assignments
Content

Do the activities align with instructional objectives?

Is the information complete, covering the content fully, clearly and logically?

Is content relevant to students?

  • QM standards checklist
  • Expert review
  • Assignment outcomes
  • Practice exercises
  • Student discussion forums
  • Entry Knowledge Analysis survey (ENKA)
Technology

Do the tech applications function properly?

Were the materials accessible by students?

Were copyright standards adhered to?

  • Number of Tech support requests
  • LTM uptime report
  • Observation
  • QM standards checklist
  • End Course Survey
Message Design

Are messages cohesive and logical?

Does supporting multimedia and features enhance learning?

Is content presented at an appropriate level for adult learners?

Are directions clear and consistent?

Are ADA guidelines met where appropriate?

Was time frame of course appropriate?

  • QM standards checklist
  • Exit Knowledge Analysis survey (EXKA)
  • Assignments & project artifacts
  • Number of instructor support requests
  • End Course Survey
Efficiency

 

Goals

Are the goals stated clearly and concisely?

Are lesson outcomes stated from a student perspective?

  • QM standards checklist
  • Expert review
  • Assignments
Content

Is content presented in a cohesive and logical manner?

Is content timely and up-to-date?

Is content relevant to the discipline?

  • QM standards checklist
  • Expert review
  • Assignments
Technology

Is orientation to LMS system available prior to start of course?

Is access to instructor stated clearly and consistently?

Is access to peers provided when appropriate?

Is LMS environment and web-based content structured clearly and consistently?


Do applications function consistently and as planned? 

  • QM standards checklist
  • Expert review
  • End Course Survey
  • Emails, discussion boards, virtual office hours, team areas in LMS
  • Observation
Message Design

Is content organized in a logical, coherent and consistent manner?

Are there visual cues such as title, subtitles and outlines to help organize the content?

  • Observation
  • Expert review
  • QM standards checklist
Appeal

 

Goals

Are goals relevant to learners?

  • QM standards checklist
  • Expert review
  • Assignment outcomes
  • Practice exercises
  • Student discussion forums
  • Entry Knowledge Analysis survey (ENKA)
  • End Course Survey
Content

Is content interesting?

Is content up-to-date?

Do students have reasonable choice regarding pace, access and navigation of content?

  • QM standards checklist
  • Expert review
  • Assignment outcomes
  • Practice exercises
  • Student discussion forums
Technology

Is website and LMS easy to navigate?

Is course website visually appealing and interesting without distractions?

  • Expert review
  • Number of instructor support requests
  • Observation
  • End Course Survey
Message Design

Is there a variety of visual, audio and text-based  multimedia?

Are screens free of clutter with adequate white space?

Are graphics, sound and animations clear?

Do screen layouts and navigation make it easy for students to understand what they are expected to do?

Are icons and graphics easy to recognize and understand?

  • QM standards checklist
  • Expert review
  • Number of instructor support requests
  • End Course Survey

Back to top

Stakeholders

Primary

  • Instructor: Renee Phoenix will be the main evaluator. She has worked at Pierce College for 8 years and has experience teaching this course online and digital design courses for the college. Her 12 + years of technical and web design experience as well as graduate studies in educational and performance technology
  • District Program Coordinator: Dr. Paul Gerhardt - oversees and evaluates adjunct faculty in business program. He is a content expert and supervises curriculum and course development. As a full-time tenured professor and business advisor, Dr. Gerhardt is usually the first person in the department called upon to address student and faculty issues in the classroom or online.
  • Students: Part of the college mission is to provide student-centered services which includes providing learning opportunities designed with student success as a primary goal. Students pay tuition and fees to attend college, looking to gain skills and knowledge that will prepare them for the workplace or for additional learning after graduation. Their input and evaluation of courses is an integral part of the student-centered process.

Secondary

  • Evaluator/Instructor/Designer: Renee Phoenix - Business department adjunct professor. Responsible for creation, design and implementation of WBI.
  • District Program Coordinator: Dr. Paul Gerhardt - oversees and evaluates adjunct faculty in business program. He is a content expert and supervises curriculum and course development. As a full-time tenured professor and business advisor, Dr. Gerhardt is usually the first person in the department called upon to address student and faculty issues in the classroom or online.
Back to top

What is Being Evaluated?


Materials to Be Examined
Design plans
  • Objectives
  • Clustering of objectives
  • Assessment strategies
  • Instructional strategies
  • Motivational strategies
LMS and website prototype
  • Course and site interfaces and navigation
  • Multimedia storyboards/scripts
  • Technology & delivery requirements

 


Evaluators and Reviewers:

  • Instructor/Evaluator/Designer : Renee Phoenix will be the main evaluator and designer. She has worked at Pierce College for 8 years and has experience teaching this course online as well as other digital design courses for the college. Her 12 + years of multimedia and web design experience and graduate studies in educational and performance technology will allow her to review the course from several different perspectives.
  • Evaluator/Expert Reviewer (subject matter): Dr. Paul Gerhardt will be the subject matter expert and evaluator for assessment, motivational and instructional strategies. Paul has a PhD in Business Management and Organizational Behavior and is an associate professor and program coordinator for Pierce College. He routinely teaches both hybrid and online courses as part of his teaching load in the business program.
  • Expert Reviewers (instructional design):
    • Ed Bachman, director of eLearning at Pierce College, will oversee an informal Quality Matters review of the course prototype. The team will consist of 2-3 QM trained reviewers - usually they are also instructors or staff at the college.
    • Dr. Glori Hink, course instructor for Onlline Course Design in the EdTech program at Boise State University will conduct reviews of the course planning and design on a regular basis.
  • End-User Reviewers:
    • Learners from current business program courses, who have taken, or are currently enrolled in, online or hybrid courses will be asked to participate in user trials and complete user surveys during the formative evaluation. Students will be volunteers and will not require any prior knowledge of the subject.
    • Graduate students from EDTECH 512 class at Boise State University will participate in peer reviews at selected intervals.

When and How Will Formative Evaluation Take Place?

Formative evaluation during design:

  1. Phase 1: Designer and SME will review the preliminary WBI evaluation plan, instructional objectives and clustering of objectives when completed by designer. One on One reviews will be conducted.
  2. Phase 2: Designer, SME and ID expert will review content, screens and technical aspects of WBI prototypes of the site, media and modules when these are completed and uploaded to the LMS. Small group user trials will be conducted in this phase.
Back to top

Preliminary Summative Evaluation Plan

Primary

Criteria Main Questions Data Sources
Effectiveness How did assignments and culminating projects meet course objectives? Did learners find the course relevant to their work or other courses they were taking in college? Performance activities; grades; end of course survey
Efficiency How many support questions did learners ask per activity?  Did technology and media function as expected? Log of email questions or issues sent to instructor or tech support;  log of downtime or tech issues with LMS, media resources or other learning components of the course; end of course survey
Appeal Will students take other distance courses from the college? Did learners enjoy working with the materials in the course? What part of the course was the most engaging and enjoyable? End of course survey

Back to top