Annual General Education Core
Student Learning Outcomes Assessment Report
General Education Outcome Assessed:
Critical
Thinking in Area D Natural Sciences
Biology
Chemistry
Geography
Geology
Date:
Academic
Year: 2004-2005
Assessment of Critical Thinking in the Natural Sciences
SUMMARY
There was great success in meeting the deadlines for this project, despite the tight time frame. All of the subcommittee members are to be congratulated for their efforts.
1.
Description of outcomes: a
student that completes Area D will be able to:
Courses
in the natural sciences employ critical thinking in many ways, including the
development of analytical thought processes through working of new problems
using learned formulae or facts, the development of skeptical analysis of
scientific results reported in the lay and scientific literature, synthetic
thinking using learned information to predict what has not yet been learned,
asking scientific questions and formulating alternate hypotheses, and designing
experiments to test hypotheses.
Recent discussion with
Joan Carson suggests that simply embedding several CT questions in the final
exam will serve as a sufficient assessment tool for SACS. The SACS goal is a 75% achievement score,
that is, at least 75% of the students should perform at the criterion level
that has been set by the department.
Summer 2004 served for tryout of assessment tools and collection of pilot data. We have gained experience for a full-scale trial in Fall term 2004. More detailed assessments will be planned in the fall and spring as methods are refined. We had planned that students would be given the same or similar assessment tools at the beginning and the end of the course. Some departments/courses implemented this suggestion and some did not. Recent discussions with Joan Carson suggest that an initial assessment is unnecessary for SACS review, although it may be internally useful at the department level. Where possible, performance is expressed as % achievement, and improvement scores are also given for departments that measured improvement over the duration of the course. Although this was not done in all cases in the pilot, we intend to derive means, ranges, and standard deviations for each measure.
A team
of instructors in each department will examine the results, looking at outcome
in relation to goals, and, for courses taught by more than one instructor,
comparing across courses. We intend to
develop methods for data to be cross-correlated with level of preparation and
course grade for each student using an analysis of variance. We do not currently have this ability. If improvement is needed, additional critical
thinking exercises will be incorporated in lectures, labs, and/or projects,
along with additional training of TAs.
TABLE 1: Courses assessed
DEPT
|
Course #
|
|
|
|
|
||
|
Biology |
Biol 1107K |
Biol 1108K |
Biol 2107K |
Biol 2108K |
|
||
|
Chemistry |
Chem1152K |
Chem1211K |
Chem1212K |
|
|
||
|
Geography |
Geog 1112K |
Geog 1113K |
|
|
|
||
|
Geology |
Geol 1121K |
Geol 1122K |
|
|
|
||
|
Physics |
Astr 1010K |
Astr 1020K |
Phys 1111K |
Phys 1112K |
|
||
6. Goals for the future
We need to reduce the level of confusion on the charge of our subcommittee, and to develop a way to compare across departments so that we can produce a single statistical measure of how well overall GSU trains its students to think critically.
Keys to
success are reducing the workload for faculty assessment efforts and allowing
flexibility in allowed assessment tools . Although all 5 disciplines fall under the
category of Natural Sciences, the disciplines are quite different in the way
they are taught and in the tools that are nationally available, and it is
important to consider this in planning efforts.
Assessment of Critical Thinking in Biology
Courses Assessed: Biol 1107K, 1108K, 2107K, 2108K
Assessment Tools:
The three faculty involved over
the summer were asked to
•employ how and why questions
during lectures and labs,
•give information and ask students
to use it to make a knowledge-based prediction,
•give students an observation and
ask them to develop a testable hypothesis about it,
•ask students to put "2 and 2
together" and combine information in new ways,
•ask students about what sort of
evidence was necessary for a particular scientific discovery to be accepted,
•ask students to draw conclusions
from a graph or set of data
The assessment tools used were:
•Entrance survey to assess prior knowledge of concepts and scientific method and of CT skills.
•Lab exercises that involved CT (data not gathered)
•Quizzes with embedded CT questions
•Exams with embedded CT questions
•Exit survey to assess developed knowledge of concepts and scientific method and of CT skills.
1107K Five critical thinking questions were administered at the end of the final exam during the Su’04 session.
1108K The test was not administered to the Biol 1108K class during summer session.
2107K A Critical Thinking quiz was given at the beginning and end of the course, and CT questions were embedded within exams. In addition, the exam included a question about graphing
2108K A different set of CT questions were embedded within exams and quizzes.
Summary of Results:
BIOL 1107 The results were (from 76 Biol 1107K students):
|
|
100% (5/5) |
80% (4/5) |
60% (3/5) |
40% (2/5) |
20% (1/5) |
0% (0/5) |
|
# of students |
71 |
5 |
0 |
0 |
0 |
0 |
|
% of class |
93% |
6.5% |
0% |
0% |
0% |
0% |
BIOL 2107K The mean score on the CT Quiz was 63%. The mean score on the CT questions embedded in the final was 67%. The performance on the graphing exercise was 69% (n=43) at the beginning of the course and 87% (n=37) at the end of the course.
BIOL 2108K (3 tools used, 2 exams and 1 quiz) 1) Exam 1- 71%, n=43 2) Quiz 2- 79%, n=43 3) Exam 2- 71%, n=37
Overall Conclusions:
There was substantial difficulty in getting lecturers on board for the effort, and in developing a common assessment tool. These problems may be unique to this particular summer as two of the faculty were resigning. There was also some confusion generated by having two team members from Biology, in that the division of labor was unclear. Further, the difficulty of getting all faculty involved to meet necessitated communication by email, which was unreliable.
Faculty members in charge of the introductory classes were concerned about the time that the assessment took away from instruction. As a consequence, they tended to forget to administer the tests unless specifically reminded. Conducting assessment only at the end of courses and embedding the questions in the final exam, as suggested by Joan Carson of the Provost’s office, should alleviate these concerns.
Changes Planned for Fall 2004:
The same assessment will be used
in all sections of comparable courses.
The lines of responsibility between the Biology coordinator and the Area
D coordinator have been more clearly drawn.
We are considering an assessment tool that could be given on WebCT and
analyzed by the
Assessment of Critical Thinking in Chemistry
Courses Assessed: Chem 1152K, Chem 1211K, Chem 1212K
Assessment Tools: The California Chemistry Diagnostic Test was given to assess overall Chemistry knowledge, and 4 questions from within the total of 44 on that test were identified as testing Critical Thinking ability. The same test was given at the start and the end of the course.
Summary of Results: Performance on the critical thinking questions at the end of the courses was as follows:
Chem 1152K 38% correct
Chem 1211K 46% correct
Chem 1212K 58% correct
In all cases the students performed much better at the end of the course than at the beginning (mean improvement 16%, range 7 to 28%, n=6 measures, mean improvement on CT assessment alone 13%, range 7 to 16, n=3 measures).
Overall
Conclusions: These data showed an
increase in performance for the selected critical thinking questions in each
course. The initial critical thinking
scores are also consistent with background of the particular student
group. If one examines the chemistry
1152 entry score, we see that the score is similar to the entry scores for
chemistry 1211 the first chemistry course for science majors since high
school. The 1152 students have just
completed chemistry 1151K which is similar in many ways to high school
chemistry.
Changes Planned for Fall 2004: After careful review of our efforts, we will use a different tool for examining critical thinking in
the future. We have to reduce the amount of
labor in the process and we have a better strategy for measuring
critical thinking. Though the current data are interesting, this only represents our first effort and we
will utilize a better tool in the future.
Assessment of Critical Thinking in Geography
Courses Assessed: Geog 1112, Geog 1113
Assessment Tools: A preliminary assessment of critical thinking by students in Geography 1112 (Introduction to Weather and Climate) and Geography 1113 (Introduction to Landforms) has been made through evaluations of exam responses and group essays. Five exam questions were given to students throughout the Geography 1112 courses. These questions were meant to assess over-all thinking ability of students, rather than solely critical thinking. Student responses were assessed with the aid of a binary rubric.
In the Maymester session of Geography 1112, group essays were constructed in response to a summary of atmospheric research that appeared in a peer-reviewed journal (see Appendix). Unfortunately, student data do not exist for the Summer-session course. Students were provided with several forms of data and then asked to accept or reject the conclusions of the scientists. More importantly, students were asked to justify their decisions. Responses from each group were judged and the results were organized in a rubric. The over-all level of critical thinking – which included 1 (poor), 2 (fair), 3 (good), and 4 (excellent) – by the groups was determined by scores on the following components: interpretation of statements and graphs; evaluation of alternative points of view; the drawing of non-fallacious conclusions, and the justification of key procedures and results. Possible scores on those components were 1 (disagree), 2 (weakly agree), and 3 (strongly agree).
Student
responses for the exam questions in Geography 1113 were not made available to
the assessment group; however, data for the critical-thinking assignment were
obtained and analyzed. The assignment
involved a hypothetical situation where students assessed what types of
hydrological and landform-related changes might result from a sudden
transformation of land-cover in the upper portion of a river basin. This assignment was lacking an important
aspect of the successful Geography 1112 assignment: the landforms assignment
did not allow students to accept or reject a point of view. Nevertheless, since this assignment was more
open-ended, it did allow for some insightful analyses. The over-all level of
critical thinking (i.e. poor to excellent) was determined by scores on the
following components: interpretation of statements and graphs; and the drawing
of non-fallacious conclusions. Since
several critical-thinking components could not be assessed, an additional metric
related to data acquisition and source acknowledgement was used instead.
Summary of Results:
The following is generally true for students in both sessions (Maymester and Summer): many students provided an insufficient and vague response to Question 2; and only a few students failed to grasp the idea behind Question 3 that DmO2 might behave similarly to CO2. The Maymester class performed much better on Questions 4 and 5. It is somewhat paradoxical that a majority of Summer-session students got Question 6 incorrect, for a majority of the students got Question 1 correct. Knowing the answer to Question 6 should be a prerequisite to answering Question 1 correctly, unless the relative-humidity formula is memorized only. It is suspected that the summer-session students probably memorized that formula without thinking of the applications of the general concept. Finally, it was a rare occurrence when students would provide a sound example of critical thinking (e.g., questioning the assumptions of the question, providing several justifiable alternative responses, etc.). Students exhibited only a fair amount of “critical” thinking when attempting to answer the exam questions.
The 1112 Maymester class as a whole had a fair performance in critical thinking. For example, no group that agreed with the scientific study provided sound reasons for their agreement. Nevertheless, three of the seventeen groups earned excellent scores by scoring highly on all of the critical-thinking components.
Similar to the Geography 1112 students, the Geography 1113 students demonstrated fair critical-thinking skills. An interesting critical-thinking-related finding from the essays is that nearly every student/group obtained stream flow data even though the data was not necessary. The collection of unnecessary data and reporting a summary of the values is counter to critical thinking: the students apparently did not recognize that the stream flow data did not enhance their arguments. In addition, few students reported the basic conclusion that soil in the North would be severely eroded and possibly completed removed. Quite a few students could not even restate correctly the basic information provided to them in the assignment instructions. Finally, no students noted that it is difficult to answer the questions without some important additional information (i.e. role of future human activities, etc.).
Overall Conclusions:
Geography 1112 and 1113 need substantial improvement. It is proposed that each exam question be one of the following underlying questions:
Changes Planned for Fall 2004:
The Geography 1112 assignment requires slight modifications to make it a solid critical-thinking assessment tool, while the Geography 1113 assignment needs major changes, such as the infusion of controversy. The assignments need to facilitate an assessment of student responses to the aforementioned components of critical thinking.
Courses Assessed: Geol 1121, Geol 1122
Assessment Tools:
The questions and the percentage correct from a lecture and a laboratory examination are presented herein. A total of 16 CT questions was used in the first lecture exam. The critical thinking questions were designed to test the student’s abilities to make inferences from the material that was covered in class and within the text. These questions were part of the multiple choice tests given as part of GEOL 1121; the answers were graded by SCANTRON. The laboratory questions are primarily designed to test how well students could take general principles and skills and apply these principles to real specimens or maps. Critical thinking questions were taken from the two multiple choice examinations given during the Summer 2004 by all Laboratory coordinators in GEOL 1121. Laboratory examinations are uniform for all GEOL 1121/1122 sections. The minerals, rocks, maps, or other laboratory materials are given to the students and from these they are asked to make critical inferences in the form of the following questions. The percentage correct are given in parenthesis following the question.
Summary of Results:
LECTURE RESULTS (TEST 1 in percentage correct for each
question)
63) 69% 64) 69% 65) 83% 66) 70%
67) 96% 68) 80% 69) 92 70) 72%
71) 85% 72) 85% 73) 85% 74) 90%
75) 94% 76) 77% 77) 91% 78) 58%
The average percentage correct for the critical thinking portion of the exam was 81% which was identical to the average for the entire exam.
LECTURE RESULTS (TEST 2 in percentage correct for each
question)
92) 60% 93) 45% 94) ND 95) 97%
96) 66% 97) 79% 98) 70%
99) 52% 100) 58% 101) 86% 102) 81%
103) 83% 104) 40% 105) 66% 106) 93%
107) 77% 108) 58% 109) 81% 110) 64%
111) 66% 112) 73% 113) 73% 114) 62%
115) 97% 116) 66%
The average score on the critical thinking portion of the exam was 70% and the average score on the total exam was 69%.
LAB RESULTS
1121K The mean score on 52 CT questions of Lab Exam 1 was 75 ± 16%. On Lab Exam 2 the mean score on 19 CT questions was 89 ± 6.8%.
1122K The mean score on 11 CT questions of
Lab Exam 1 was 80 ± 14%.
Overall Conclusions:
These very preliminary results indicate that the students did no better or worse on the critical thinking portion of the exam than on the exam as a whole.
Changes Planned for Fall 2004:
Dr. Rose has instructed the
Geology Department on what we have to do for fall semester. We have left the form of critical thinking
evaluation open to individual faculty members; however, Dr. Rose went over the
multiple choice option in detail (in that this is how GEOL 1121/1122 is usually
tested). The faculty members and Ph.D.
students that have been assigned to teach GEOL 1121/1122 for the Fall Semester
will submit reports to Dr. Rose.
in Physics and
Astronomy
Courses assessed:
Summer 2004:
Phys 1111K (2 lectures, 6 sections
total) Introduction to Physics I
Phys 1112K (2 lectures, 5 sections
total) Introduction to Physics II
Astr 1010
(2 lectures, 4 sections total) Astronomy of the Solar
System
Assessment Tools:
Assessment was performed in both lectures and labs and also varied by instructor. The assessment tools, data, analysis, and conclusions from each instructor are available.
Astronomy
1010 Lecture: Two hourly exams and a
final exam were given. Of the 150
questions appearing on these exams, we identified those questions that
challenged a student to think critically.
For a sample of 30 students, these questions were graded separately and
the results were compared to the course letter grade. These questions required the student to use
different combinations of the critical thinking skills, such as selecting
appropriate data, analyzing that data, formulating hypotheses, and making predictions
based on the data and hypotheses. In
general, they could not be answered simply by memorizing definitions of terms
or simple facts.
Physics Lab
sections:
Performance on quizzes was measured at the beginning and the end of the
course.
Physics 1112K Lecture: The first
assessment tool used was a standardized conceptual test given on the first day
of class. This diagnostic test (Force
Concept Inventory) covers material the students have had in a previous course
in the sequence and emphasizes their understanding of concepts. In addition, since it asks them to come to
conclusions about how these concepts relate to particular situations, these
questions require application of critical thinking skills. Additional assessment was done using essay type
questions on the second, third, and final exams.
Results:
Astr
1010 Lecture
Course Grade Critical
Thinking Score Standard
Deviation Number of Students
A 83% 10% 13
B 59% 12% 9
C 49% 15% 8
Total Sample 67% 19% 40
Averages
for students receiving the same course grade:
Course Grade Laboratory
Tutorials Standard Deviation Number of Students
A 97% 2.5% 13
B 93% 4.2% 9
C 92% 2.9% 8
Total Sample 95% 4.0% 40
Fifty three students were tracked
from Physics 1111 lab and thirty three from Physics 1112 lab. Their
average scores for the critical thinking questions in the second and third
quizzes were compared to their average scores from questions in the tenth and
eleventh quizzes. Answers were scored on a zero to three scale.
|
|
Phys1111K |
Phys1112K |
||
|
Quiz 2 and 3 |
1.36 out of 3.00 |
45% |
1.35 out of 3.00 |
45% |
|
Quiz 10 and 11 |
1.74 out of 3.00 |
58% |
1.26 out of 3.00 |
42% |
|
Change |
+0.38 |
+13% |
-0.09 |
-3% |
The 1111 students showed an
average gain of .38 points with a standard deviation of 1.14 while the 1112
students showed an average loss of .09 points with a standard deviation of
1.24. Neither is statistically significant.
Phys 1111K and 1112K lecture
sections
The pre-test was taken from the 3
major exams consisting of 2 questions each, giving a total of 6 questions. The post-test was taken for the final exam
consisting of 6 questions from the final exam.
Physics 1111 TTH class
Gain= 33% (Gain = (posttest average% - pretest average%)/(100-pretest
average%)
Physics 1111 MW class
Gain= 33% (Gain = (posttest average% - pretest average%)/(100-pretest
average%)
Physics 1112K class #1
Gain = 28 %, (Gain = (posttest average% -
pretest average%)/(100-pretest average%)
Physics 1112K class #2
No significant gain was seen across the term.
Physics 1112K class #2 Assessment Data and
Analysis
|
|
Tool 1 |
Tool 2 |
Tool 3 |
Tool 4 |
Tool 5 |
Tool 6 |
Course Points |
|
Possible |
30 |
27 |
25 |
6 |
25 |
27 |
1100 |
|
Average |
10.1 |
9.3 |
7.1 |
1.1 |
4.9 |
17.8 |
722 |
|
Percent |
33.7% |
34.4% |
28.4% |
18.3% |
19.6% |
65.9% |
65.6% |
|
|
|
|
|
|
|
|
|
|
Students Receiving |
|
|
|
|
|
|
|
|
A |
13 |
15.3 |
12 |
1.9 |
8.3 |
25.4 |
927 |
|
|
43.3% |
56.7% |
48.0% |
31.5% |
33.2% |
94.1% |
84.3% |
|
|
|
|
|
|
|
|
|
|
B |
9.6 |
10.9 |
7 |
2 |
6.4 |
21.2 |
752 |
|
|
32.0% |
40.4% |
28.0% |
33.3% |
25.6% |
78.5% |
68.4% |
|
|
|
|
|
|
|
|
|
|
C |
9.6 |
6.1 |
5.4 |
0.1 |
2.2 |
12.1 |
636 |
|
|
32.0% |
22.6% |
21.6% |
1.7% |
8.8% |
44.8% |
57.8% |
Only a small overall increase in
scores in critical thinking questions was observed over the course of the term
(Tools 5 & 6 from final exam compared with Tools 2-4 from earlier
exams). This result is not statistically
significant.
Overall Conclusions:
Astronomy
It is clear that the questions requiring critical thinking are more challenging than other test questions, and that the critical thinking abilities of the students are correlated with overall success in the course. There is a wide range of individual critical thinking scores, even among students receiving the same course grade. The course grade includes not only other test questions, but also components based on lab work and various group participation activities.
There is still a correlation between the critical thinking skills demonstrated in these laboratory exercises and the course grade, although the numerical scores are both higher and more tightly clustered than the scores achieved with the tool used in the lecture portion of the course. One could point to several possible conclusions:
Students were more successful at developing critical thinking skills in an interactive setting, or 2) the grading reflected the skills of the strongest student in each group, or 3) the instruments used in the lab and lecture portions were measuring different skills, or some combination of the above.
Aspects of Astronomy 1010 that stress critical thinking, in both the lecture and laboratory portions of the course, have been identified. Students’ critical thinking abilities generally develop during the course to meet our expectations, and these abilities directly influence the course grade. Not surprisingly, there are some students who fail to meet our expectations but pass the course due to hard work and faithful completion of other assignments.
Physics
Clearly a high correlation was
seen between performance on critical thinking assessment tools and total course
points and therefore the grade received in the Physics 1112K class #2 course. In addition,
a bigger disparity was seen between performance of A, B, and C students in critical
thinking assessment tools than in total course points. Large improvements in critical thinking
skills were generally not observed in the laboratory sections. Physics
has developed several very specific plans on how they might make changes in
both lectures and labs to explicitly teach and assess more fully critical
thinking skills. These are detailed in
the individual reports attached.
Changes Planned for Fall 2004:
Astronomy
Because of the compressed nature of the summer session and limited faculty involvement, the pilot program was restricted to a sample of Astronomy 1010 students. Based on what we have learned, plans for an expanded and revised program in the coming full semester are highlighted below.
· Existing questions which measure critical thinking skills will also be identified in Astronomy 1020.
· New questions for Astronomy 1010 and 1020, potentially useful for administration at the beginning and end of the semester, will be tested. A sample of such questions for Astronomy 1010 is included in Appendix B.
· The assessment of critical thinking will be extended to a broader sample of students in multiple sections.
· The tutorial exercises will be continued as part of the laboratory work in both Astronomy 1010 and 1020, but with more emphasis on measuring retained individual achievement after the group activity.
Physics
Fall 2004 Courses to be
assessed:
Astr1010, Astr1020, Phys1111K, Phys1112K, Phys2211K, Phys2212K
Assessment in these courses will
follow the same general strategy as used in the summer courses, including
assessment in lab and lecture. However several significant changes will
be implemented to improve on the assessment performed over Summer
2004, e.g. development of better pre- and post- questions for Physics and
Astronomy classes, and broader sample sizes used in all courses.
Changes to be made to improve
the teaching of critical thinking skills:
* Improved training of TAs in
teaching critical thinking.
* Additions to lab and lecture
descriptions to make expectations regarding critical thinking more explicit.
* Use of more context-rich and
graphical questions in laboratory quizzes and reports.
* Adoption of a new textbook in
Phys2211K (and for Phys2212K beginning in Spring 2005)
with emphasis on analysis, critical thinking, and graphical
representation.
Developing a better critical
thinking teaching program in the freshman labs:
* The lab manual introduction now
has a clearer explanation of our expectations of the successful student,
including a part about critical thinking.
* More critical thinking questions
are being added to the weekly quizzes.
* There is more emphasis on graph
construction and analysis, application of physics principles to everyday life,
and the way seemingly different concepts are related.
* TA's are now expected to set
aside some time at the start of each period to go over a problem from the
preceding week's quiz with the class. A "Stop and Think"
section has been added to each experiment with problems or ideas the students
can discuss among themselves or with their instructor.
* TA training will have more
emphasis on CT. A professor from the philosophy department is going to
give a presentation on teaching critical thinking.