Date:Thu, 31 Jan 2008 16:21:15 -0800
Reply-To:"AERA Division L: Educational Policy and Politics Forum"
<[log in to unmask]>
Sender:"AERA Division L: Educational Policy and Politics Forum"
<[log in to unmask]>
From:Richard Hake <[log in to unmask]>
Subject:Re: pre-to-post tests as measures of learning/teaching #2
Content-Type:text/plain; charset="us-ascii" ; format="flowed"
AERA-L: Politics and Policy in Education Forum
If you reply to this long (24 kB) post please don't hit the reply
button unless you prune the copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers. [Note: this is
a slightly improved version of the post with the same title
transmitted to Chemed-L, PhysLrnR, and POD on 30 January 2008.]
******************************************
ABSTRACT: Among the thoughtful responses to my post "Re: pre-to-post
tests as measures of learning/teaching" were those of POD's Ed Nuhfer
and Chemed-L's Logan McCarty. Nuhfer implied that there are people
who proclaim multiple choice tests to be the standard for determining
quality of education, but I know of no one outside politics who's
stupid enough to make such an absurd proclamation. McCarty argued
that: (1) the highest form of learning that we achieve in
introductory courses is *skills* not facts or concepts; (2) pre/post
testing inevitably promotes "teaching to the test"; (3) he (Logan)
could employ traditional non-interactive teaching methods in such a
way that his students would do well on tests such the Force Concept
Inventory (FCI); (4) the FCI is "incredibly narrowly focused"; (5)
Newtonian concepts hinder students' understanding in non-Newtonian
areas; (6) "concepts" are at a lower level of Bloom's taxonomy than
"skills"; and (7) writing ability can't be tested with multiple
choice tests. I argue that McCarty is wrong on all 7 counts and has
therefore flunked his pretest ;-).
******************************************
Among the thoughtful responses to my post "Re: pre-to-post tests as
measures of learning/teaching"[Hake (2008)] were those of POD's Ed
Nuhfer (below under "N") and Chemed-L's Logan McCarty (below under
"M"). I respond below to their comments:
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
Ed Nuhfer (2008) in his POD post of 28 January 2008 wrote (my CAPS):
"I wonder, AMONG THOSE WHO PROCLAIM MULTIPLE CHOICE TESTS AS THE
STANDARD OF DETERMINING QUALITY OF EDUCATION, how many ask in
interviews about how well a candidate scores on multiple choice
tests? How many position announcements can we find that call for
hiring a person who scores high on multiple choice tests? Unless
believing one's own rhetoric can lead to acting on it in practical
ways, the rhetoric needs to be set aside while one builds a better
model."
But I know of no one outside politics who's stupid enough to make
such an absurd proclamation.
For courses in Newtonian mechanics, diagnostic tests such as the
Force Concept Inventory (FCI) [Hestenes et al. (1992)] or Force
Motion Concept Evaluation (FMCE) [Thornton & Sokoloff (1998)] are
understood by most physicists to measure only the attainment of a
*minimal* conceptual understanding of mechanics, *not* the overall
quality of a mechanics course.
As indicated in the suppressed "Interactive-engagement methods in
introductory mechanics courses" [Hake (1998b)], among desirable
outcomes of an introductory physics course that the average
normalized gain <g> *does not* measure directly are students':
(a) satisfaction with and interest in physics;
(b) understanding of the nature, methods, and limitations of science;
(c) understanding of the processes of scientific inquiry such as
experimental design, control of variables, dimensional analysis,
order-of-magnitude estimation, thought experiments, hypothetical
reasoning, graphing, and error analysis;
(d) ability to articulate their knowledge and learning processes;
(e) ability to collaborate and work in groups;
(f) communication skills;
(g) ability to solve real-world problems;
(h) understanding of the history of science and the relationship of
science to society and other disciplines;
(i) understanding of, or at least appreciation for, "modern" physics;
(j) ability to participate in authentic research.
MMMMMMMMMMMMMMMMMMMMMMMMMMMM
Logan McCarty (2008) in his Chemed-L post of 28 January 2008 made 7
points to which I shall respond:
1111111111111111111111111111111111111
1. "What I am puzzled by is the *sole* focus on 'understanding
concepts' (as demonstrated by the ability to answer multiple-choice
questions correctly) in contrast to 'demonstrating skills.' I think
that - in chemistry at least - the highest form of learning that we
achieve in introductory courses are *skills*, not facts or concepts."
Later in his post McCarty invokes the outdated Bloom's (1956)
taxonomy to place "skills" (that McCarty equates with "Application,
Analysis, and Synthesis" above "Concepts" in the learning hierarchy.
But that ordering is not consistent with the modern version of
Bloom's taxonomy [Anderson & Krathwohl (2001)] as discussed below in
"6."
In any case, gains on formative diagnostic tests such as the FCI and
the FMCE are *not* the sole focus of most introductory mechanics
courses. Instead many of the objectives (a-j) listed in my response
to Nuhfer above are addressed. For example:
(a) at Arizona State University <http://modeling.asu.edu/> the
emphasis is on "modeling";
(b) the University of Minnesota
<http://groups.physics.umn.edu/physed/Research/CRP/crintro.html>,
emphasizes "context-rich problems"; and
(c) Indiana University <http://www.physics.indiana.edu/~sdi>,
emphasizes Socratic Dialogue Inducing (SDI) Labs which, in addition
to addressing concepts, were designed to enhance students':
(1) understanding of the nature of science;
(2) use of effective strategies for scientific thinking and
problem-solving; and
(3) research skills such as collaborative effort, drawing, written
description, thought experiments, modeling, consideration of limiting
conditions, experimental design, control of variables, dimensional
analysis, and solution of real-world problems.
22222222222222222222222222222222222222
2. "I think that pre/post testing is all fine and good, but it will
*inevitably* lead to "teaching to the test."
Logan evidently thinks that "teaching to the test" is NOT "fine and
good." So he must be using the phrase "teaching to the test" to mean
going over questions or problems identical or nearly identical to the
test items. I think he is flat wrong to believe that such
unprincipled practice is an inevitable outcome of pre/post testing.
As I indicted in Hake (2008), pre/post testing physics in the courses
surveyed in Hake (1998a,b) was not summative, but FORMATIVE as
defined by the Joint Committee on Standards for Educational
Evaluation, JCSEE (1994): "Formative evaluation is evaluation
designed and used to improve an object, especially when it is still
being developed." This is also the case in more current courses of
which I'm aware. Both teachers' "action research" and education
researchers' scientific research is carried out to improve classroom
teaching and learning, NOT to rate instructors, students, or
institutions. Thus "teaching to the test," in the sense of
telegraphing answers to questions on the test, would be highly
counterproductive and serve no useful purpose. In fact, the use of
pre/post testing to gauge the effectiveness of high-school and
undergraduate courses in promoting students' conceptual understanding
has succeeded in part because it is strictly formative.
Of course if "teaching to the test" means giving students some
understanding of the basic concepts of Newtonian mechanics as
examined on the FCI or FMCE tests, then "teaching to the test" is
certainly "fine and good," as I have previously discussed [Hake
(1998a, 2002)]. One of the virtues of pre/post testing with tests
that measure the essentials is that it does encourage "teaching to
the test" in that positive sense.
3333333333333333333333333333333333333
3. "I assure you that if my goal is to get students to answer
conceptual multiple-choice questions correctly, I could do so with a
wide variety of teaching methods (some interactive, some not)."
If Logan can teach a mechanics course in the traditional
non-interactive passive-student lecture mode (without telegraphing
answers to the FCI questions) so that his students achieve an average
normalized gain <g> on the FCI greater than, say, 0.3, he will go
down in the "Guinness Book of Records"
<http://www.guinnessworldrecords.com/records/default.aspx> as the
first instructor in the entire World to do so, and will have exceeded
the <g> = 0.23 for traditional teaching set in 1990 by his Harvard
colleague Eric Mazur [see, e.g. Crouch & Mazur (2001)].
44444444444444444444444444444444444444
4. "I'm happy to "teach to the test"-- indeed sometimes I try to do
that by writing the exam first and then building the lectures -- but
I'm not willing to teach to a test that is so incredibly narrowly
focused as the Force Concept Inventory."
It's true that the Force Concept Inventory (FCI) is narrowly focused
on concepts, vis.: Newton's three laws, just as are most introductory
mechanics courses. Does McCarty think that a better pre/post test
would focus on material that is not the focus of the course? Or
would he teach an introductory mechanics course that ignores Newton's
laws in favor of focusing on what he calls "skills"?
Many traditional physics instructors think that introductory physics
courses should focus on the "skill" of *problem solving.* They assign
vast numbers of back-of-chapter problems and devote lectures and
"recitations" to going over problem solutions. Sorry guys, pre/post
testing demonstrates that "Students do not overcome conceptual
difficulties after solving 1000 traditional problems" [Kim & Pak
(2002)].
555555555555555555555555555555555555555
5. "Physicists love Newtonian physics. But guess what: if you're a
cell, and you're immersed in a fluid (so that it's a low Reynolds
number environment) then -- presto! -- you're in an entirely
Aristotelian environment! The "natural state" of motion is to be at
rest - velocity is proportional to force - and all of those other
"bad things" that we try to drill out of our students' heads. Sorry,
guys, but the 'conceptual intuition' that you're trying to teach in
Newtonian mechanics is not much use to a cell biologist!"
Sorry Logan, but physics students (future cell biologists among them)
who have some understanding of Newtonian mechanics are at an
*advantage*, rather than a disadvantage, in adapting to non-Newtonian
environments such as those with low Reynolds number [Purcell (1976)];
those of special relativity: velocities approaching c; and those of
quantum mechanics: dimensions approaching the atomic. In fact, all
three areas are often introduced in introductory physics courses as
natural extensions of Newtonian mechanics.
6666666666666666666666666666666666666666
6. "I appeal finally to Bloom's taxonomy: Hake is describing tests
and courses that get to the levels of Knowledge and Comprehension,
but no further. I want to get the students to Application, Analysis,
and Synthesis. I can't see any way (unfortunately!) of testing
these *skills* using a multiple-choice test."
Logan's ordering: Knowledge and Comprehension, Application, Analysis,
and Synthesis indicate that he is appealing to the original version
of Bloom's Taxonomy [Bloom et al. (1956)] and not the Bloom's
Taxonomy updated by Anderson & Krathwohl (2001). [The latter, in
Fig. A.1, page 268, provide a "Summary of the Structural Changes form
the Original Framework to the Revision."] The latter incorporates
the important post-1956 insights of cognitive science [see e.g.,
Bransford et al. (2001)].
As shown on the inside FRONT cover of their book, Anderson &
Krathwohl distinguish FOUR "Knowledge Dimensions": Factual,
Conceptual, Procedural, and Meta-Cognitive; with each type having a
"cognitive process dimension" (see inside the BACK cover of their
book).
A related distinction has been emphasized by the late Arnold Arons
(1983) who wrote:
"Researchers in cognitive development describe two principal classes
of knowledge: figurative (or declarative) and operative (or
procedural) [Anderson (1980)) . . .[the most recent version is
Anderson (2004)]. . . ; Lawson (1982)]. DECLARATIVE KNOWLEDGE
CONSISTS OF KNOWING 'FACTS'; for example, that the moon shines by
reflected sunlight, that the earth and planets revolve around the sun
. . . . OPERATIVE (OR PROCEDURAL) KNOWLEDGE, ON THE OTHER HAND,
INVOLVES UNDERSTANDING THE SOURCE OF SUCH DECLARATIVE KNOWLEDGE (How
do we know the moon shines by reflected sunlight? Why do we believe
the earth and planets revolve around the sun when appearances suggest
that everything revolves around the earth? . . . .) and the capacity
to use, apply, transform, or recognize the relevance of the
declarative knowledge to new or unfamiliar situations."
The FCI and FMCE *do* in fact measure operative or procedural
knowledge, near the top of the updated Bloom's "Knowledge Dimension"
[Anderson & Krathwohl (2001)].
777777777777777777777777777777777777777
7. [Testing "skills" with a multiple choice test (MCT]. . . . .
would be like testing writing using a [MCT]. Sorry, but it just
doesn't work.
Sorry, Logan, it *does* work, at least in a restricted sense.
Although MCT's would probably not be able to distinguish the
Hemingways from the Hack-a-ways, MCT's will at least distinguish
those who understand Trigg's (1979) Rules of Grammar from those who
don't.
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
Honorary Member, Curmudgeon Lodge of Deventer, The Netherlands.
<[log in to unmask]>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>
"Conflict is the gadfly of thought. It stirs us to observation and
memory. It instigates to invention. It shocks us out of sheep-like
passivity, and sets us at noting and contriving. Not that it always
effects this result; but that conflict is a sine qua non of
reflection and ingenuity."
John Dewey "Morals Are Human," Dewey: Middle Works, Vol.14, p. 207.
REFERENCES
Anderson, J.R. 2004. "Cognitive Psychology and Its Implications."
Worth Publishers, 6th edition; Amazon.com information at
<http://tinyurl.com/2qrgu6>.
Anderson, L.W. & D.R. Krathwohl, eds. 2001. "A Taxonomy for Learning,
Teaching, and Assessing: A Revision of Bloom's Taxonomy of
Educational Objectives." Longman. Amazon.com information at
<http://tinyurl.com/35s9yt>.
Arons, A.B. 1983. "Achieving Wider Scientific Literacy," Daedalus,
Spring; reprinted in Arons (1990, 1997)). Arons wrote: "Researchers
in cognitive development describe two principle classes of knowledge:
figurative (or declarative) and OPERATIVE (or procedural). See also
the suppressed "Arons Advocated Method" Hake (2004).
Arons, A.B. 1990. "A Guide to Introductory Physics Teaching." Wiley,
reprinted with minor updates in Arons (1997).
Arons, A.B. 1997. "Teaching Introductory Physics." Wiley. Contains a
slightly updated version of Arons (1990), plus "Homework and Test
Questions for Introductory Physics Teaching", plus a new monograph
"Introduction to Classical Conservation Laws." Amazon.com
information at <http://tinyurl.com/2y4qse>. Note the "Search Inside"
feature.
Bloom, B.S., M.D. Englehart, E.J. Furst, W.H. Hill, & D.R. Krathwohl.
1956. "Taxonomy of educational objectives: The classification of
educational goals: Handbook I, Cognitive domain." David McKay.
Bransford, J.D., A.L. Brown, & R.R. Cocking, eds. 2000. "How people
learn: brain, mind, experience, and school." Nat. Acad. Press; online
at <http://books.nap.edu/books/0309070368/html/index.html>.
Crouch, C.H. & E. Mazur. 2001. "Peer Instruction: Ten years of
experience and results," Am. J.Phys. 69: 970-977; online at
<http://web.mit.edu/jbelcher/www/TEALref/Crouch_Mazur.pdf> (284 kB).
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66(1): 64-74; online at
<http://www.physics.indiana.edu/~sdi/ajpv3i.pdf> (84 kB). See also
the crucial companion paper Hake (1998b).
Hake, R.R. 1998b. "Interactive-engagement methods in introductory
mechanics courses," online at
<http://www.physics.indiana.edu/~sdi/IEM-2b.pdf> (108 kB) - a crucial
companion paper to Hake (1998a).
Hake, R.R. 2002. "Comment on 'How do we know if we are doing a good
job in physics teaching?' by Robert Ehrlich," Am. J. Phys. 70(10):
1058-1059; online at
<http://www.physics.indiana.edu/~hake/HakeOnEhrlich-2.pdf> (40 kB).
Hake, R.R. 2004. "The Arons Advocated Method," submitted to the
"American Journal of Physics" on 24 April 2004; online as reference
31 at
<http://www.physics.indiana.edu/~hake>, or download directly as a 144
kB pdf by clicking on
<http://www.physics.indiana.edu/~hake/AronsAdvMeth-8.pdf>.
Hake, R.R. 2007. "Over Sixty Academic Discussion Lists: List
Addresses and URL's for Archives & Search Engines," online at
<http://www.physics.indiana.edu/~hake/ADL-F.pdf> (580 kB), and as
ref. 49 at <http://www.physics.indiana.edu/~hake>.
Hake, R.R. 2008. "Re: pre-to-post tests as measures of
learning/teaching" online at the OPEN AERA-J
<http://tinyurl.com/3booqj>. Post of 28 Jan 2008 17:33:48-0800 to
AERA-L, Chemed-L, PhysLrnR, & POD; and (with several typos corrected)
to AERA-J on 29 Jan 2008 11:19:05-0800. Abstract only sent on 29
January to AERA-A, AERA-B, AERA-C, AERA-D, AERA-GS; AERA-H, AERA-I,
AERA-K, AP-Biology, AP-Calculus, AP-Chemistry, ARN-L, ASSESS,
Biopi-L, Biolab (rejected), DrEd, EdResMeth, EvalTalk, IFETS, ITForum
(rejected), Math-Learn, Math-Teach, NetGold, PBL, Physhare, Phys-L,
PsychTeacher (rejected), RUME, SCListserv, STLHE-L, TeachingEdPsych,
TIPS, and WBTOLL-L. For a guide to discussion lists see Hake (2007)].
Hestenes, D., M. Wells, & G. Swackhamer. 1992. "Force Concept
Inventory," The Physics Teacher 30(3): 141-158, March; online (except
for the test itself) at <http://modeling.asu.edu/R&E/Research.html>.
The 1995 revision by Halloun, Hake, Mosca, & Hestenes is online
(password protected) at the same URL, and is currently available in
15 languages: Chinese, Czech, English, Finnish, German, Greek,
Italian, Malaysian, Persian, Portuguese, Russian, Slovak, Spanish,
Swedish, & Turkish. A French version should soon be available.
JCSEE. 1994. Joint Committee on Standards for Educational Evaluation,
"The Program Evaluation Standards," 2nd ed., Sage. A glossary of
evaluation terms from this publication is online at
<http://ec.wmich.edu/glossary/prog-glossary.htf>.
Kim, E. & S-J Pak. 2002 "Students do not overcome conceptual
difficulties after solving 1000 traditional problems," Am. J. Phys.
70(7): 759-765; online to subscribers at
<http://scitation.aip.org/dbt/dbt.jsp?KEY=AJPIAS&Volume=70&Issue=7>.
Lawson, A.E. 1982. "The Reality of General Cognitive Operations,"
Sci. Ed. 66: 229. With regard to declarative and procedural
knowledge see also Lawson (2000).
Lawson, A.E. 2000. "Science Teaching and the Development of
Thinking." Eos Publishing. Amazon.com information at
<http://tinyurl.com/223xy3>.
McCarty, L. 2008. "Re: pre-to-post tests as measures of
learning/teaching" Chemed-L post of 28 Jan 2008 21:34:20-0500; online
at
<http://mailer.uwf.edu/listserv/wa.exe?A2=ind0801&L=chemed-l&D=1&O=D&P=24904>.To
access the archives of Chemed-L it may be necessary to subscribe, but
that takes only a few minutes by clicking on
<http://mailer.uwf.edu/archives/chemed-l.html>> and then clicking on
"Join or leave the list (or change settings)." If you're busy, then
subscribe using the "NOMAIL" option under "Miscellaneous." Then, as a
subscriber, you may access the archives and/or post messages at any
time, while receiving NO MAIL from the list!
Nuhfer, E. 2008. "Re: pre-to-post tests as measures of
learning/teaching," POD post of 28 Jan 2008 21:36:12-0800; online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0801&L=pod&P=23649>.
Purcell, E.M. 1976. "Life at Low Reynolds Number," webified by J.
Brody; online at
<http://jilawww.colorado.edu/perkinsgroup/Purcell_life_at_low_reynolds_number.pdf>
(124 kB).
Richoux, D. 2002 . "Humorous Rules for Writing_('Fumblerules,'
'Perverse Rules,' etc.)," June 18; online at
<http://alt-usage-english.org/humorousrules.html>. This is from the
Alternate Usage English (AUE) Newsgroup
<http://alt-usage-english.org/index.shtml>.
Thornton, R.K. 1995. "Conceptual Dynamics: Changing Student Views of
Force and Motion," in "Thinking Physics for Teaching," C. Bernardini,
C. Tarsitani, & M. Vicentini, eds. Plenum. Amazon.com information at
<http://tinyurl.com/3dh84u>.
Thornton, R.K. & D.R. Sokoloff. 1998. "Assessing student learning of
Newton's Laws: The force and motion conceptual evaluation and the
evaluation of active learning laboratory and lecture curricula," Am.
J. Phys. 66(4): 338-352; online to subscribers at
<http://scitation.aip.org/dbt/dbt.jsp?KEY=AJPIAS&Volume=66&Issue=4>.
See also Thornton (1995).
Trigg, G.L. 1979. "Grammar," Physical Review Letters 42(12): 747-748,
19 March; online to subscribers at
<http://prola.aps.org/toc/PRL/v42/i12>. For Trigg's Rules of Grammar
see, e.g., Richoux (2002).
AERA Division L: Politics and Policy in Education
-----------------------------------------------------------------
AERA Home Page on the World Wide Web: http://www.aera.net
List Service Info http://listserv.aera.net/scripts/wa.exe
To cancel your subscription address an email message to
[log in to unmask] containing only the message UNSUB AERA-L
Address problems with your subscription to: [log in to unmask]
-----------------------------------------------------------------
