Annual Student Learning Outcomes Assessment Report
Unit Name: Department of Economics
Degree Programs: BA, BS, MA, Ph.D.
Prepared by: James Alm, Date: September 6, 2005
Academic Year
2004-2005
Number of graduates from
each degree program this academic year
MA-Econ: 18; MA-Econ Policy Track: 1; Ph.D.: 5
Number of students
currently enrolled in each program major
MA-Econ: 28; MA-Econ Policy Track: 11; Ph.D.: 73
|
III. Changes to Procedures or Curriculum Based on Assessment |
|
A.
Outcomes
Assessed
All learning outcomes specified in the Department of Economics Learning Outcomes Assessment Plans are in the process of being assessed; see http://education.gsu.edu/ctl/outcomes/AYSPS/Economics_Assessment_Plan.htm.
B.
Elements of
Assessment
To measure the success of Economics major in the undergraduate program in learning core economic concepts, the Department of Economics developed two “Tracking Exams”, one for Principles of Microeconomics and one for Principles of Macroeconomics. Each exam is comprised of 20 multiple choice questions that cover the core concepts taught in the two principles courses. The Tracking Exams are administered each fall and spring semester in a selection of 3000/4000 level courses. At the end of the 20 questions, the student is asked whether or not they are majoring in Economics, and the student is presented with a list of all undergraduate economics courses and is asked to indicate which courses they have taken. The same exams will be given over time so that the Department can determine changes in learning outcomes. Students are not allowed to take a copy of the exam with them, and are not given the answers to the exam at any point. The two Tracking Exams were developed and first administered in Fall 2004.
In the MA-Econ program, two elements of assessment (the exit examination and the essay) were implemented. The third element of assessment (the alumni survey) will be implemented in Fall 2005. Evaluation forms with questions evaluating the particular elements along several categories were filled out by faculty grading the exam or essay.
In the MA-Econ: Policy Track program, one of three elements of assessment was implemented (the exit examination). The second and the third elements of assessment (the portfolio and the alumni survey) are not yet complete. The portfolio element is a new requirement for the program, and will be implemented when the first students that are bound by this new requirement graduate. The alumni survey will be implemented in Fall 2005. For the exit examination, evaluations forms as described above were filled out by the faculty grading the examination.
In the Ph.D. program, three of four elements of assessment have been implemented (the comprehensive examination, the field examination, and the dissertation). The last element of assessment (the alumni survey), but will be implemented in Fall 2005. For each of the three elements of assessment, questionnaires were designed to evaluate each element along a number of dimensions. The questionnaires were filled out by the faculty evaluators of the particular examination or dissertation.
C. Data Collected
For undergraduate majors, the Microeconomics Tracking Exam (MicroTE) was administered to 113 students in four upper-level economics courses. The Macroeconomics Tracking Exam (MacroTE) was administered to 76 students in two upper-level courses. Both have been administered again in Spring 2005, with some modifications, as discussed below.
D. Data Analysis
For undergraduate majors, performance on the two Tracking Exams was determined by examining pass rates on each of the questions. In addition, regression analysis will be used to determine which core concepts are best reinforced in the upper level courses (i.e., to relate the performance of students on the exams to the number of upper level courses, as well as to specific upper-level courses).
Results from Fall 2004 indicated that on four questions more than 90% of students answered the question correctly. While the Department was pleased that students had mastered the concepts in these questions, there was concern about the exam’s ability to detect improvements in learning outcomes over time. These four questions were modified and made more difficult, although the same concepts were tested in each question. The MicroTE was re-administered in Spring 2005 to 80 students in two upper-division courses. Overall, 58% of the questions were answered correctly. On 7 of 20 questions, less than 50% of the students answered the question correctly. In general, students performed best (e.g., more than 50% answered the questions correctly) on questions that were definitional in nature or were simple applications, and they performed worse on questions that required more complex applications. The MacroTE was not thought to need adjustments initially. Results from the Fall 2004 MacroTE indicated that on average 60% of the questions were answered correctly. The minimum score was 30% correct, and the maximum score was 95% correct. Only one question was answered correctly by more than 90% of students. The macro results were harder to interpret, as no general patterns emerged among individual questions relating to specific learning outcomes. Unlike the MicroTE, students sometimes did not answer a definitional question correctly, but then students answered an application question on that same topic correctly; sometimes the opposite pattern was the case. It was the consensus of the faculty involved that the MacroTE needed to be modified to better discriminate concepts that the students have mastered.
For the MA-Econ program, there were 21 students who took the Macroeconomics exam during the
academic year 2004-2005, with 8 Fail and 13 Pass grades, for a pass rate of
62%. Recall that each exam was graded on
the eight factors. A factor analysis
revealed the weakest factor was Application with a mean score of 2.48, while
the strongest factor was Communication with a mean score of 3.52. There were 13 students who took the
Microeconomics exam during the academic year 2004-2005, with 3 Fail and 10 Pass
grades (a 77% pass rate). A
factor analysis revealed that the weakest part was Mathematics with a mean
score of 2.31, while the strongest parts were Analytics and Applications with a
mean score of 3.08. There were 21 MA-Econ students who took the
Econometrics exam during the academic year 2004-2005, with 6 Fail and 15 Pass
grades (a71% pass rate). A factor
analysis revealed the weakest factor was Application with a mean score of 2.9,
while the strongest factor was Definition with a mean score of 3.48.
For the MA-Econ:
Policy Track program, there were 2 students who took the Macroecnomics exam
in April 2005 with 1 Fail and 1 Pass grades.
A factor analysis showed that the weakest factor was Definition with a
mean score of 2, and the strongest factor was Communication with a mean score
of 3.5. No student took the
Microeconomics exam during the academic year 2004-2005. One
MA-Econ: Policy Track student took the Econometrics exam in April 2005 with a
Fail grade. The strongest factor for
this student was Definition with a score of 4, and the weakest factors were
Analytics and Applications with scores of 2.
For the Ph.D. Program, 11 out of 14 students taking the Macroeconomics Comprehensive Exam
received an overall grade of Pass, and 3 students received a grade of
Fail. The mean grade across all
questions was 3.37 out of 5 for passing exams and 2.35 out of 5 for failing exams. Regarding specific questions in the
evaluation forms, all the following categories received an average grade
between 3 (Low Pass) and 4 (Pass): Definitional, Analytics, Mathematics,
Critical Judgment, and Communication. The category Application received the
lowest grade 1.9. Regarding the field
examination in Public Finance, 3 out of 6 students passed the exam. The 3
students that failed will attempt the exam a second time in January 2006 as per
program regulations. The mean grade
across all questions was 3.33 out of 5 for passing exams and 2.04 out of 5 for
failing exams. The following categories
received an overall grade of at least 3.16: Definition, Applications,
Mathematics, and Computation. The remaining categories received grades as
follows: Analytics (2.33), Critical Judgment (1.83), and Communication
(2.4). Regarding the dissertations, all
6 students who attempted a final defense passed. The average in 6 of 9
categories was 4.0 or higher (the maximum possible is 5); the average in the
remaining 3 categories was higher than 3.3.
II. Achievement of Departmental Student Learning Objectives
Results from Fall 2004 indicated that the MicroTE needed modifications. On four questions, more than 90% of students answered the question correctly. While the Department was pleased that students had mastered the concepts in these questions, there was concern about the exam’s ability to detect improvements in learning outcomes over time. These four questions were modified and made more difficult, although the same concepts were tested in each question. The MicroTE was re-administered in Spring 2005 to 80 students in two upper-division courses. Overall, 58% of the questions were answered correctly. On seven of twenty questions, less than 50% of the students answered the question correctly. In general, students performed best (e.g., more than 50% answered the questions correctly) on questions that were definitional in nature or were very simple applications, and they performed worse on questions that required more complex applications.
The MacroTE was not thought to need adjustments initially. Results from the Fall 2004 MacroTE indicated that on average 60% of the questions were answered correctly. The minimum score was 30% correct, and the maximum score was 95% correct. Only one question was answered correctly by more than 90% of students. The macro results were harder to interpret, as no general patterns emerged among individual questions relating to specific learning outcomes. Unlike the MicroTE, students sometimes did not answer a definitional question correctly, but then students answered an application question on that same topic correctly; sometimes the opposite pattern was the case. It was the consensus of the faculty involved that the MacroTE needed to be modified to better discriminate concepts that the students have mastered.
In the MA-Economics program, the exit examination element of assessment evaluated the first learning outcome of the program (“To learn and grasp basic analytical skills of microeconomics, macroeconomics, and econometrics”). The data analysis discussed above showed that this learning outcome was achieved with passing grades by a majority of the students who took the exams (62% for macroeconomics, 77% for microeconomics, and 71% for Econometrics). Due to insufficient data, the other three learning outcomes cannot be assessed for the academic year 2004-2005. The weakest area in macroeconomics and econometrics is Application, while for microeconomics Mathematics is the weakest area. The strongest areas are Definition (in econometrics), Communication (in macroeconomics), and Analytics and Application (in microeconomics). The weaker areas will be addressed in section III below.
In the MA-Economics, Policy Track program, the exit examination evaluates the first learning outcome of the program (“To be able to learn and to grasp basic tools for economics analysis”). The exit exam is a new requirement for this program. The data analysis shows that the students in this program, who traditionally have a weaker background in mathematics and statistics, have difficulty in achieving this learning outcome; the pass rate for Macroeconomics was 50% and the one student who took the Econometrics section did not pass. Again, due to insufficient data, the other three learning outcomes cannot be assessed for the academic year 2004-2005.
In the Ph.D. program, the desired outcomes are generally being met quite well. In terms of
learning outcome #1 (“To achieve a high level of competence understanding and
using the analytical skills of microeconomics and macroeconomics”), the
Macroeconomics Comprehensive Exam shows that 11 of 14 students passed this exam
during the academic year. In addition,
the overall average by category is between 3 (Low Pass) and 4 (Pass) for all
categories but one. Hence, students are achieving mastery of macroeconomics in
terms of Analytics, Definition, Mathematics, Critical Judgment, and
Communication. The only area of concern is Application, which will be addressed
below. Learning outcome #3 (“To
demonstrate mastery of the issues, theories and latest advances in one of the
sub-fields of economics offered by the program”) is also generally being
satisfied. Three out of 6 students passed the Public Finance field exam. The
students who did not pass in this first attempt can re-take the exam a second
time. There have not been any students who did not pass the field exam by the
second examination in recent years. Students did quite well in the Definition,
Applications, Mathematics, and Computation categories of the Public Finance
field exam, although some attention needs to be given to improving in the
Analytics, Critical Judgment, and Communication categories. The faculty in the
public finance field committee believe that some exam answers were somewhat
mechanical regarding analytics and critical judgment. They have suggested a way
to improve this as detailed below.
Learning outcome #2 (“To achieve a high level of competence
understanding the most recent theoretical and quantitative methods in
economics”) and learning outcome #4 (“To demonstrate ability to conduct
independent and original basic and applied research in economics”) are clearly
being met, as indicated by the evaluation of dissertations. Concerning the
question of whether the dissertation will result in publication in a peer
reviewed academic journal, faculty evaluators “strongly agree” (the highest
rating) for 4 out of 6 dissertations and “agree” for one other dissertation.
Only one dissertation was rated as neutral as regards this question. Since publication is one of the strongest
indicators of the quality of research, this speaks well of the faculty’s
opinion of these dissertations.
Questions regarding comprehension of the literature, ability to convey
technical material and research questions, and command of econometric skills,
data collection, measurement, and computation all received average grades of
4.0 or higher (out of a possible 5).
Contribution to the literature, economic analysis, and theoretical
skills received 3.33 or higher. In
summary, these dissertation scores indicate that learning outcomes #2 and #4
are being met very satisfactorily.
III. Changes
to Procedures or Curriculum Based on Assessment
A. Undergraduate
Programs
As discussed above, the MicroTE was modified in response to the first set of results. The MacroTE is currently under revision in response to faculty concerns about some specific questions included. The Department is considering the effectiveness of this assessment instrument in the absence of incentives for students to do well, by exploring how to provide appropriate incentives for the exams.
After evaluating the Spring MicroTE and the Fall Macro TE, the Department is sharing the results with the faculty, and emphasizing that students need more practice applying the core economic concepts in the upper level courses (for microeconomics) and that instructors in macroeconomics must have a consistent approach that emphasizes the core macro concepts rather than definitions and stylized relationships. These areas of emphasis are being suggested for both the introductory and upper-level courses. Additionally, for those teaching upper-level courses, it is recommended that faculty revisit the core concepts and give students ample opportunity to apply the concepts in new settings, rather than assuming the students have a firm grasp on the core concepts as a result of an earlier principles course.
The goals for next year are to improve the results so that the average score on the tracking exams is 60% or better and also to improve on questions that relate to applications of core concepts.
B. Graduate Programs
For the MA-Econ and MA-Econ: Policy Track programs, the macroeconomics and econometrics courses need to increase the student’s exposure to applications since this was the weakest area. More application-type questions need to be placed in homework and on course examinations for practice. In microeconomics, the area that needs renewed emphasis is mathematics. These changes are being made for the upcoming academic year.
For the Ph.D.
program, given the low average score in the Application category of the
comprehensive exam in macroeconomics, the faculty teaching those courses will
place renewed emphasis on this dimension during coursework. Specifically, application-type questions will
be prominently featured in homework and in every course examination so as to
provide the students more training in how to approach such questions.
Regarding the
mastery of a sub-field as measured by the field exam, scores in the Analytics
and Critical Judgment can be raised by increased exposure in class to a variety
of economic models that require creative ways of solving them, avoiding a
“one-size-fits-all” mechanical approach.
In addition, these elements will be more prominently featured in
homework and exams during coursework so that students get more practice trying
to implement a variety of solution approaches.
Overall, while the evaluation questionnaires for all
graduate programs have been very useful, these can be improved to yield still
more useful information on comprehensive and field exams and on theses and
dissertations. The graduate committee
will add several fill-in questions where the evaluating faculty can discuss the
strengths and weaknesses of an exam/thesis/dissertation in his or her own
words. The present questionnaire may be
too rigid as it only asks the evaluator to evaluate on specific dimensions
rating from highest to lowest. Also, the alumni survey will be implemented in
Fall 2005, which is expected to yield information on the effectiveness of
training received after students have joined the workforce.
IV.Changes in Department’s Assessment Goals
There are no changes in assessment goals planned at this
time.