Assessment & Critique
CFI Ground School AVF 225
Assessment
Chapter 5
Objective
The instructor student shall be able to assess the levels of learning, use various methods of assessment, construct and conduct effective assessments and provide student guidance
Assessment
No skill is more important than the ability to analyze, appraise, and judge a student’s performance
There are 2 broad categories of assessment
Traditional – written testing
Good for the rote and understanding levels of learning
Authentic – performance of real world tasks
Good for the application and correlation levels of learning
Assessment terms
Standards or criteria – characteristics that define acceptable performance
Rubric – guide used to score performance
Assessment terms
Formal – involve some sort of documentation
Informal – occur as needed, usually a verbal critique
Diagnostic – used to determine the student’s level of learning prior to the start of training
Formative – ungraded wrap up of a lesson, used as a guide for the instructor of problem areas
Summative – used periodically through the training to evaluate how well the student has progressed to a certain point
Purpose of assessment
Provides information to the student and the instructor
The feedback the student receives may provide motivation to do better or let them know what they are doing is working
The instructor may use it for diagnostic purposes to see if the teaching style is working or if there are any holes in knowledge
Characteristics of an effective assessment
There are 8: (FASTCOCO)
Objective
Flexible
Acceptable
Comprehensive
Constructive
Organized
Thoughtful
Specific
Effective assessment
Objective
Should be objective, not focused on personal opinion
Do not let your like or dislike of the student color your assessment
Halo error is when the instructor sympathizes or over identifies with the student and curtails the assessment
Personality conflicts may also sway assessment
Effective assessment
Flexible
The entire performance must be assessed within the context it was accomplished
The tone, technique and content must fit the occasion
Sometimes student’s just have a bad flight
Your assessment should take into account variables
What to leave in, what to leave out, what to stress and what to minimize should fit the conditions
Effective assessment
Acceptable
In order for the assessment to be of any value, the student must accept what the instructor is saying as truthful
This requires confidence of the student in the instructor’s abilities to make an accurate assessment
The student will often have questions about the assessment so be prepared to back up what you say with facts and examples
Failure on the instructor’s part to do so undermines the assessment process
Watch for defense mechanisms to pop up here, they will also undermine the process
If presented fairly and with sincerity it can be highly effective
Effective assessment
Comprehensive
Cover every aspect in detail
Decide if you want to hit the major points or cover more of the minor points
Sometimes it’s beneficial to cover only those items that need the most improvement, especially after a subpar session
Cover the strengths as well
Constructive
The student must benefit from this process or it’s a waste of time
Do not give praise for its own sake, make it count
Give guidance to correct items that need improvement
Effective assessment
Organized
The assessment should follow a logical order
Often times if assessing a flight lesson, the order it which the tasks where completed make the most sense
Perhaps a flight unraveled from a specific point, if so start there
Thoughtful
Keep in mind self-esteem, recognition and approval
Ridicule, anger and fun at the expense of the student are all weapons at your command, try not to use those
Be respectful of your students feelings or they won’t be your student much longer
Effective assessment
Specific
Stay away from abstractions
Tell the student exactly what they did wrong and exactly how to fix it
Express yourself with firmness and authority, if you do so you’d better know what your talking about
When all is said and done everyone should still be friends and know where they stand
Characteristics of a good written test
There are 6: (CODRUV)
Reliability
Validity
Usability
Objectivity
Comprehensiveness
Discrimination
Characteristics of a good written test
Reliability
The test results are consistent over time
Test scores are distributed evenly in about the same percentage
Validity
The test measures what it is supposed to measure as defined by the objective
The most important of the 6
Get several instructor buddies to help here
Usability
The test is functional
Are the spaghetti charts readable?
Is the type face big enough to read?
Are the directions clear?
Characteristics of a good written test
Objectivity
Is the test scorable consistently
Essay questions vs multiple choice or true and false
Comprehensiveness
The degree the test measures the objectives
Must reference a good cross section of all the items included
Discrimination
The degree to which the test differentiates between students
It must measure small differences in achievement of the learning of the objectives
3 features support this:
A wide range of scores
All levels of difficulty
Distinguishes different levels among students
Appendix B
Written test items
Supply type tests require the learner to furnish a response
This may take the form of fill in the blank, write a sentence or paragraph
Disadvantages to this form of testing are consistent grading and they take more time to complete
Advantages include testing to higher levels of learning and better feedback for the instructor/student of understanding
Selection type tests usually come in the form of true and false or multiple choice
A disadvantage with this form is it usually tests for rote level only
Advantages include being able to directly compare students, ease and reliability of grading, and testing many different areas in a shorter time
Appendix B
True and false
Highly prone to guessing
Good for testing knowledge of facts and details
Tests to the rote level
Include only one idea in each statement.
Use original statements rather than verbatim text.
Make the statement entirely true or entirely false.
Avoid the unnecessary use of negatives, which tend to confuse the reader.
Underline or otherwise emphasize the negative word(s) if they must be used.
Avoid involved statements.
Keep wording and sentence structure as simple as possible.
Make statements both definite and clear.
Avoid the use of ambiguous words and terms (some, any, generally, most times, etc.)
Use terms which mean the same thing to all learners whenever possible.
Avoid absolutes (all, every, only, no, never, etc.) These words are known as determiners, because they provide clues to the correct answer.
Avoid patterns in the sequence of correct responses because learners can often identify the patterns.
Make statements brief and approximately same length.
State the source of a statement if it is controversial (sources have differing information).
Appendix B
Multiple Choice consists of 2 parts:
The stem (question)
The list (responses)
Incorrect answers are called distractors
Considered to be better than true and false and matching
Use multiple choice when there are
Built-in and unique solutions, such as a specific application of laws or principles.
Wording of the item is clearly limiting, so that the learner must choose the best of several offered solutions rather than a universal solution.
Several options that are plausible, or even scientifically accurate.
Several pertinent solutions, with the learner asked to identify the most appropriate solution
Appendix B
There are 3 major challenges
The development of a question or an item stem that must be expressed clearly and without ambiguity
The statement of an answer or correct response cannot be refuted.
The distractors must be written in such a way that they are attractive to those learners who do not possess the knowledge or understanding necessary to recognize the keyed response
Appendix B
Use the following guidelines for multiple choice:
Make each item independent of every other item in the test. Do not permit one question to reveal, or depend on, the correct answer to another question.
Design questions that call for essential knowledge rather than for abstract background knowledge or unimportant facts.
State each question in language appropriate to the learners.
Include sketches, diagrams, or pictures when they can present a situation more vividly than words.
When a negative is used, emphasize the negative word or phrase by underlining, bold facing, italicizing, or printing in a different color.
Avoid questions containing double negatives, which invariably cause confusion.
Avoid trick questions, unimportant details, ambiguities, and leading questions that confuse and antagonize the learner.
If attention to detail is an objective, detailed construction of alternatives is preferable to trick questions
Appendix B
Writing stems:
The function of the stem is to set the stage for the alternatives that follow.
The stem should clearly present the central problem or idea.
The stem should contain only material relevant to its solution, unless the selection of what is relevant is part of the problem.
The stem should be worded in such a way that it does not give away the correct response.
Avoid the use of determiners, such as clue words or phrases.
Put everything that pertains to all alternatives in the stem of the item. This helps to avoid repetitious alternatives and saves time.
Generally avoid using “a” or “an” at the end of the stem. They may give away the correct choice.
Every alternative should grammatically fit with the stem of the item.
Appendix B
Alternatives are just as important as the stem and should fit the context of the stem
Suggestions for distractors are:
An incorrect response related to the situation and which sounds convincing.
A common misconception.
A statement which is true, but which does not satisfy the requirements of the problem.
A statement that is either too broad or too narrow for the requirements of the problem.
Appendix B
Matching is great for assessing the learner’s ability to recognize relationships and make associations
Use the following guidelines:
Give specific and complete instructions. Do not make the learner guess what is required.
Test only essential information; never test unimportant details.
Use closely related materials throughout an item.
If learners can divide the alternatives into distinct groups, the item is reduced to several multiple choice items with few alternatives, and the possibility of guessing is distinctly increased.
Make all alternatives credible responses to each element in the first column, wherever possible, to minimize guessing by elimination.
Use language the learner can understand. By reducing language barriers, both the validity and reliability of the test is improved.
Arrange the alternatives in some sensible order. An alphabetical arrangement is common
Authentic assessment
There are several ways to conduct an authentic assessment
Collaborative Assessment
Use of open ended questions to guide the student through assessing their own performance
This uses a 4 step process
Replay – have the student recall their performance of the maneuver
Reconstruct – the student identifies key things that should have gone differently
Reflect – using the list on pg 5-6 have the student look at overall impressions
Redirect – relate current experience to former experiences
This type of assessment is referred to as student-centered grading
Authentic assessment
After the self assessment is done, have an in-depth discussion of both the student’s and instructor’s assessment
Performance should be jointly determined on 2 rubrics
The first deals with skill maneuvers or procedures
The second deals with single pilot resource management (SRM)
Single pilot resource management
The grades or rubric are as follows:
Explain
Verbal identification of the risks
Practice
Application of SRM to a situation
Manage-Decide
Gather the important data, identify courses of action, make an appropriate decision
Advantages include
Making the student part of the process
Establishes validity to the assessment
Establishes a system of self evaluation for use after flight training
Self esteem issues are avoided, there is no failing grade just a failure to demonstrate a given level of SRM skills
Choosing the assessment method
Test development follows a 4 step process:
1. Determine the level of learning objectives
Objectives should measure one of the domains of learning
2. List indicators of desired behavior
Select the best behaviors that give evidence of learning
3. Establish criterion objectives
Establish the conditions under which the behavior is to be performed
4. Develop criterion referenced test items
Create the test questions
Fortunately the PTS can be used here
Other references such as the FAR’s and Advisory Circulars can be used
Critiques and oral assessments
A critique is a detailed analysis and assessment of something, in this case the student’s performance
A critique has several components:
A critique may be individual or in a classroom setting
A critique may be oral, written or both
A critique happens within a close time frame of the event
Any activity may be the subject of a critique
It should include both good and bad
There are several types of critique
Types of critique
Instructor/student critique
The instructor leads a group discussion in a detailed analysis of a performance
It should have a clear purpose, controlled carefully, and organized
Student led critique
In a classroom setting, one student is chosen to lead a critique
Because of limited knowledge it may only beneficial on the whole
Small group critique
Divide the class into small groups and assign a specific area to analyze
Furnish guidelines to keep groups on track
Combine each group’s critique to form a comprehensive assessment
Types of critiques
Individual student critique by another student
Can you say “train wreck”?
If you try this you must maintain firm control of the process
Self critique
Used to open the door to topics the student didn’t think went well on the flight
Can be used as a tool to promote better self evaluation
Written critique
Has 3 advantages:
Gives the instructor time to make a better eval
Gives the student something to take home and use as a study guide
If done in a classroom setting, the student gets a record of several different “looks” at the topic
Oral assessment
This is the most common form of assessment
Reasons why an instructor may use this form of assessment are:
Gives immediate feedback
Reveals the effectiveness of the instructor’s training methods
Checks student retention of what has been learned
Reviews material already presented to the student
Can be used to retain student interest and stimulate thinking
Emphasizes the important points of training
Identifies points that need more emphasis
Checks student comprehension of what has been learned
Promotes active student participation, which is important to effective learning
Characteristics of effective questions
Questions may be loosely classified as facts questions and HOTS questions
Facts questions:
Are closed ended questions focus mainly on facts
Test the rote level of learning
Usually have only one right answer
Begin with who, what, when and where
HOTS questions:
Are open ended questions focused on HOTS and ADM
Test the higher levels of learning including understanding, application and correlation
May have a variety of correct solutions
Begin with how and why
One method of having intelligent questions is to generate them at a the same time as the lesson plan and include them on the plan
Characteristics of effective questions
Effective questions have some common characteristics:
They apply to the subject
They are brief and concise, but also clear and definite.
They can be adapted to the ability, experience, and stage of training of the students.
They center on only one idea (limited to who, what, when, where, how, or why, not a combination).
Present a challenge to the students.
Types of questions to avoid
Effective quizzing should not include over generalized questions like “do you understand?” or “any questions?”
The puzzle
This has many pieces that must first be put together to even understand the question
Oversized
These are big general questions incapable of supporting an objective
Toss-up
2 possible options given, with a requirement to make a choice
Bewilderment
Similar to the puzzle but often include non-pertinent data to throw off the student
Trick question
Designed to purposely mislead the student and cause them to select the wrong answer
Irrelevant question
Do not relate to the objective at all
Answering questions
Be sure to understand the question
Restate the question or ask a question for clarification
Display interest in the question
Frame your answer in the context of the question
Be accurate and concise as necessary to give a complete answer
Determine if the student understands the answer
They may or may not be satisfied
Defer a question to a later date if it would complicate the learning of the current topic
If the question is way off base it may be beyond the scope of the lesson
If a question is asked the instructor doesn’t know the answer to, admit not knowing the answer and offer to get the answer or help them look it up
No one can know the answer to every possible question, students realize this
However if the student finds the answer with relative ease and finds others know this information readily it will be embarrassing at the least
Hey everybody has a bad day once in a while, instructors included