DEPARTMENT OF POLITICAL SCIENCE

Georgia State University

Annual Student Learning Outcomes Assessment Report, 2004-2005

I. Assessment Procedures	III. Changes to Procedures or Curriculum Based on Assessment
II. Achievement of Departmental Objectives for Students A. Undergraduate--	IV. Changes in Department’s Assessment Goals

ASSESSTMENT OF LEARNING OUTCOMES: UNDERGRADUATE PROGRAM

I. Assessment Procedures

A. Outcomes assessed

The Political Science undergraduate program serves multiple audiences, most notably departmental majors and those students who take political science courses to fulfill university requirements or out of general interest. Student learning outcomes include:

1. Substantive knowledge

First learning outcome: Students should demonstrate understanding of the structures and processes of American and Georgian government commensurate with the performance of citizenship duties and the stability of an effective civil society. Specifically, students should have a fundamental knowledge of constitutionalism, federalism, separation of powers, civil liberties, and the electoral process.

Second learning outcome: Students should demonstrate recognition of the universality of politics in human experience, an appreciation of political issues from a global perspective, and an appreciation of global issues from a political perspective.

2. Analytical skills

First learning outcome: Students should demonstrate an understanding of the difference between normative and descriptive explanations of political behavior.

Second learning outcome: Students should demonstrate an ability to assess evidence using principles of logical analysis and be able to apply that evidence when making conclusions. This outcome includes the ability to recognize appropriate supporting evidence as well as assessing contrary evidence.

3. Communication skills

Learning outcome: Students should demonstrate an ability to write a paper or make an oral presentation with a clear thesis statement or question, support this statement or address this question in a logical manner, and draw logical conclusions from findings. In doing so, students should demonstrate organizational skills of presentation without distracting grammatical errors. In such communication, students should demonstrate the analytical skills in outcome #2 above. Students also should be able to demonstrate an ability to support their findings by citing relevant authorities. Students should demonstrate a nuanced understanding of plagiarism when writing their own papers and must not use the ideas of others without citation.

4. Methodological Skills:

First learning outcome: Students should demonstrate basic knowledge of the use of social statistics.

Second learning outcome: Students should demonstrate an ability to understand data reported in various forms.

Third learning outcome: Students should demonstrate an ability to conduct research using traditional and new technological resources.

Fourth learning outcome: Students should demonstrate an understanding of the scientific method, including the formulation of hypotheses and the role of independent, control and dependent variables.

B. Elements of assessment

The Director of Undergraduate Studies (DUGS) and the Undergraduate Program Committee will implement the assessment process outlined below on a yearly basis and provide a report of the findings, plus any recommendations, to the Chair of the Department by the end of spring semester and to the Faculty at the beginning of fall semester. Regarding the specific learning outcomes, the procedures are as follows:

1. Substantive knowledge assessment

Concerning the first learning outcome (American and Georgia government), students should be able to pass exams involving these concepts. The Department will review syllabi and exams from PolS 1101 classes (Introduction to American Government) to ensure that students are being successfully taught these concepts in this required class that emphasizes these concepts. It also will collect data involving the overall pass rate for these classes.

Concerning the second learning outcome (the universality of politics in human experience and appreciation of political issues from a global perspective), students should be able to pass exams demonstrating the political nature of global issues. The Department will review syllabi and exams from PolS 2401 (Global Issues), a required class in which this outcome is addressed, to ensure that the outcome is being addressed successfully. It also will collect data involving the overall pass rate for these classes.

2. Analytical skills assessment

The assessment of this goal is the same for both learning outcomes listed above (an understanding of the difference between normative and descriptive explanations of political behavior, and an ability to assess evidence using principles of logical analysis and be able to apply that evidence when making conclusions).

Here the department has instituted compulsory writing assignments in all upper division courses and has encouraged communication assignments in the lower division. In order to assess the achievement of students in regard to the outcomes of sections 2 and 3 of our undergraduate program, the Department will require students to present at the time of their senior audit their best paper written in a political science class. Such papers will be used by the Department to assess its success in achieving these outcomes.

3. Communication skills

The assessment of this learning outcome is conducted in the same manner as goal #2 above, involving an evaluation of class writing assignments.

4. Methodological skills

The assessment of this learning outcome involves evaluating syllabi and exams from PolS 3800 (Introduction to Political Research), a course required of all political science majors.

C. Data collected

Three sets of data were collected from sample sections of the three core courses mentioned above (PolS 1101, 2401, and 3800) for the current assessment (2004-05 academic year): syllabi and exams, course papers, and grades. The DUGS has reviewed the content of the syllabi, exams, and papers in preparing the analysis presented below; instructors of the core courses were also asked to provide recent grades for their courses:

PolS 1101 data

Fall 2004 sections:

CRN 81860 Average 76% Passing 95%

CRN 81862 Average 80% Passing 100%

CRN 83414 Average 75% Passing 94%

Spring 2005 sections:

CRN 12121 Average 72% Passing 97%

CRN 12123 Average 74% Passing 93%

CRN 12131 Average 73% Passing 94%

PolS 2401 data

Fall 2004 section:

CRN 81874 Average 84% Passing 99%

Spring 2005 sections:

CRN 12135 Average 87% Passing 98%

CRN 12137 Average 81% Passing 96%

PolS 3800 data

Fall 2004 sections:

AM section: Average 74.6% Passing 91%

PM section: Average 82.3% Passing 97%

Spring 2005 sections:

AM section: Average 68% Passing 94%

PM section: Average 69% Passing 89%

PolS 3800 is required of all political science majors while PolS 1101 and PolS 2401 have a majority of non-majors. Since political science majors do not get credit for grades lower than a “C”, the actual pass rate for PolS 3800 is slightly lower when one includes “D” grades with the “F” grades:

Fall 2004:

AM section: 85% passing with a “C” or better

PM section: 80% passing with a “C” or better

Spring 2005:

AM section: 82% passing with a “C” or better

PM section: 76% passing with a “C” or better

To supplement the Department’s own self-evaluation with the views of our own students, we have collected Fall 2004 average scores from selected questions on all student evaluation surveys from the two social science core curriculum courses in political science: PolS 1101 and PolS 2401. On the key questions from the survey, the average scores were as follows:

Total N Q1 Q2 Q3 Q4 Q5[1]

PolS 1001

< 70 students/section 206 4.57 4.65 4.48 4.36 4.38

> 70 students/section 1402 4.42 4.46 4.31 4.10 4.13

PolS 2401

< 70 students/section 270 4.51 4.52 4.42 4.42 4.37

> 70 students/section 399 4.53 4.59 4.28 4.39 4.28

D. Data Analysis

Both syllabi and exams from multiple sections of PolS 1101 and PolS 2401 directly and consistently address the two learning outcomes falling under the heading of substantive knowledge. Syllabi and exams for PolS 1101 focus on issues related to the first substantive knowledge learning outcome: the practice of governmental institutions in the U.S. and Georgia, highlighting fundamental concepts such as constitutionalism, federalism, the separation of powers, civil liberties, and the electoral process. Syllabi and exams for PolS 2401 directly address the second substantive knowledge learning outcome: the universality of politics in human experience, an appreciation of political issues from a global perspective, and an appreciation of global issues from a political perspective. In doing so the course covers a very wide range of “current events” issues from a political science perspective, including security, environment, energy, human rights, immigration, gender, development, and many others depending on the instructor of record. It also typically includes other disciplinary perspectives when analyzing these issues, such as economics, history, anthropology, area studies, sociology, and so on. We should note as well that the majority of our political science majors choose a special concentration in either pre-law or international studies; such students receive even more instruction on these aspects (American/Georgian institutions and universality of politics) of the substantive knowledge learning outcome as they complete their programs. Finally, all political science courses beyond those analyzed in this assessment frequently include such topics as part of their specialized course content.

Papers collected by the DUGS during senior audits were used to provide assessment material for two learning outcomes: analytical skills and communication skills. Here there is a substantial degree of variation in quality as related to these skills. This is however more likely due to factors related to the collection process rather than any inherent skills of our students. In particular, four challenges emerged:

1) This was a new requirement and not all graduating political science majors were able to submit papers in the past year (34 papers were received in the past academic year out of 107 graduates; a response rate of about 30%);

2) For those students who did comply, by their own admission some could present only whatever paper they had at hand rather than a paper representing their “best” work;

3) The types of assignments clearly varied across the papers. Some were more substantial research papers; others were closer to opinion or review papers. The length varied widely as well depending on the nature of the assignment;

4) Not all papers received had a grade on them, making it difficult for the DUGS to evaluate them without more knowledge of the assignment requirements and grading criteria of the instructors for whom the papers were written.

Therefore, while the majority of these papers (typically the students’ “best papers” from their political science courses) indeed conformed to the analytical and communication skills noted above (i.e., an understanding of the difference between normative and descriptive explanations of political behavior; an ability to assess evidence using principles of logical analysis and be able to apply that evidence when making conclusions; and an ability to write a paper with a clear thesis statement or question, support this statement or address this question in a logical manner, and draw logical conclusions from findings), others simply were not substantial enough to make a clear judgment on either the analytical or communication skills learning objectives.

Regarding exam grades, the average grades for PolS 1101, PolS 2401, and PolS 3800 are exactly that: average (i.e., right in the middle to high “C” range overall). As the department has not yet set any standards regarding the desired grade distributions for the core courses, nor do we have a “benchmark” against which to measure these results on a time-series basis, we can only conclude that the course grades seem to conform to a normal distribution curve. This is true of both PolS 1101 and PolS 2401, which have hundreds of students (and therefore a larger and more representative sample), as well as PolS 3800, which typically has only a few dozen students and only political science majors. And all cases the overall passing rate is very high, above the 90^th percentile. PolS 1101 and PolS 2401 cover the substantive knowledge learning outcome noted above and we can conclude that this objective is being met based on exam scores and syllabi/exam content. PolS 3800 also covers the fourth learning outcome noted above: methodological skills, and this objective is being met as well, though with a demonstrably lower success rate if one includes “D” grades with “F” grades as “D” grades are not considered passing grades for our majors. PolS 3800 however covers material that is often more difficult for students to master in a single course and we expect its grade distribution is similar to the research methods courses required in other departments, such as Sociology and Economics. Research papers for PolS 3800 were not generally evaluated as part of this assessment and the Department may want to consider making that a component of future assessments (see below).

Finally, looking at the students’ own evaluation scores as a “control” for the two core political science courses (PolS 1101 and PolS 2401) taken by a large number of GSU students (mostly non-majors), we are pleased with a very high success rate across the most important questions on the survey. For all such questions in both PolS 1101 and PolS 2401 the average scores for Fall 2004 were over 4.0 on the 5 point scale (5 being the best). We also should note that these results were achieved in both large and small sections of the courses surveyed, suggesting that our instructors are equally effective in reaching their students whether in small or very large (200+ students) sections.

II. Achievement of Departmental Objectives for Students

A. Undergraduate program

Review of syllabi and exam content reflects a high degree of conformity with the substantive learning outcomes noted above. PolS 1101 and PolS 2401 are well-established courses in the department; they are always taught in multiple (and often large) sections each semester by junior and senior faculty as well as advanced graduate students. Each course is further supported by the appointment of a faculty coordinator to help ensure greater consistency among the various sections and instructors in terms of substantive content, textbook choices, and learning outcomes. In addition, these two courses are generally required of all GSU students; therefore we have a very large number of non-majors in each one. These courses also require a significant level of department resources to meet the needs of hundreds of such students each semester. However, despite the large of number of students taught, and the likely lack of interest in these courses among the majority of those students (i.e., non-majors), the department achieved a very high and fairly consistent success rate across both courses: over 9 out of 10 students pass these courses and the average grades conform to a normal distribution curve.

Our analysis of the student writing samples shows a far more uneven range of quality due in part to reasons noted above. For students who presented their best political science papers as graded by their instructors in various upper-level political science classes, the papers generally conform to the two learning outcomes noted above: analytical skills and communication skills. Regarding analytical skills, the best papers tend to distinguish between normative and descriptive explanations of political behavior (particularly papers from courses related to American public law, a major component of our undergraduate program); they also make use of appropriate sources of evidence when analyzing the issue at hand and making conclusions. On the communications skills learning outcome, the best papers tend to be well-organized, have a clear thesis statement or argument, and are relatively free of spelling or grammatical errors. Students who turned in actual research papers (as opposed to essays or opinion pieces) often showed an understanding of the learning outcomes associated with the methodological skills noted above. Finally, while these papers were not used part of our assessment of methodological skills, it is interesting to note that only 2 out of 34 papers received (about 6%) made any use whatsoever of statistical analysis as taught in PolS 3800. In fact, only a handful of papers demonstrated any familiarity with the scientific method as defined by our methodological skills learning outcome, particularly in terms of formulating hypotheses and manipulating variables. There might be a variety of reasons why this is so; however, if statistics-based methodological skills are an important part of the political science curriculum then the Department may want to either require the use of such methods across a wider range of upper-level courses and/or pay more attention to the teaching of qualitative methods as most of the papers relied upon historical or legal/institutional evidence to make their arguments rather than statistical data.

For students who were unable to turn in their best work, however, the quality of the papers was far more erratic regarding all three learning outcomes (analytical, communication, and methodological skills). Some papers were more polemical rather than analytical; others lacked relevant sources of evidence or failed to make a clear argument/thesis, while still others suffered from multiple spelling or grammatical mistakes. Obviously it is difficult to assess our students’ best work if they fail to make such work available to the assessors and instead turn in whatever paper they have at hand, even if it is not from a political science course. We also should note that the department has had a few instances of plagiarism on written assignments; our instructors have been asked to include clearer definitions of, and penalties for, plagiarism on their syllabi and to share information regarding such incidents with each other in case such students attempt plagiarism in another political science course.

III. Changes to Procedures or Curriculum Based on Assessment

A. Undergraduate program

Regarding changes to assessment procedures, we first see little need to make dramatic changes to our assessment of the substantive content of the core courses, except perhaps to include a wider range of course sections in the data used for assessment. PolS 1101 is based on a fairly well-established common body of knowledge and uses well-established textbooks and exam questions. PolS 2401 allows for more flexible content on the part of each instructor and also is especially susceptible to the use of outdated examples or issues. However, all instructors of PolS 2401 are aware of the need to keep the topic fresh with each cohort of new students while attempting to create a new common body of knowledge on this emerging sub-field. Beyond the substantive knowledge assessment, both PolS 1101 and PolS 2401 instructors might be encouraged to make greater use of WebCT resources to improve the written communication skills of undergraduates, especially since these courses are taught in large sections where it can be difficult for instructors to implement extensive written assignments. Oral communications skills could be improved in these classes through the use of in-class debates and presentations, as some instructors of PolS 2401 have implemented. The Department also might consider making use of Writing Across the Curriculum support to provide teaching assistants for instructors who choose to make their courses more writing-intensive. Finally, since both PolS 1101 and PolS 2401 often involve the use of graduate student instructors, the Department might consider implementing a more systematic training and mentoring program for its graduate instructors. The “Preparing Future Faculty program” of the American Political Science Association might be a good place to start, and the Department could appoint a faculty member to implement such a program.

In terms of grades on exams, the department will first need to determine how to interpret these data in the absence of any clear benchmarks. Specifically, should the department attempt to maintain the current success rate even in the face of higher enrollments and greater demands on faculty resources, or should it in fact attempt to improve the success rate of student performance in the core courses? Since over 90% of PolS 1101 and PolS 2401 students pass these courses, the only way to improve performance significantly is to work with students to raise their average scores from the “C” range to the “B” range. This, of course, defeats the basic meaning of “average” and would likely result in some degree of grade inflation. The Department may want to consider a comparative analysis of its grade distributions in these core courses with those of other required social science courses before setting or changing its grading standards.

In terms of using student papers as part of the assessment process, the Department at a minimum should insist that students save and submit their best research-intensive work (i.e., substantial research papers with a demonstrably high grade) from political science courses, ideally papers that involve independent research and analysis. One possibility is to require students to present their papers from PolS 3800, the research course required of all majors. While this would ensure consistency of the type of papers submitted (i.e., research papers with a statistical focus), it might reduce consistency on other dimensions. For example, some students might not have done their best work in PolS 3800 and the assessment would suffer accordingly. PolS 3800 also makes intensive use of quantitative methods (statistics) while some students might feel more comfortable using qualitative methods. If the sample of papers received over the past year is any guide, our political science majors are very interested in issues concerning public law, and the Department may want to consider ways to improve their legal writing skills. In addition, students take PolS 3800 at different stages of their careers and might need several opportunities to practice the methodological skills taught in that course before their “best” work begins to appear. Still, the Department needs to find a way to ensure greater consistency in the types of papers submitted by students for the purposes of general assessment. While the Department will continue to collect and monitor the various data sources noted above, it will need to think about standardizing the collection and evaluation of student writing samples, whether in the form of research papers or otherwise. Once the appropriate types of papers are chosen for assessment the Department should attempt to devise a basic evaluation form or scale to further track student performance on three learning outcomes: analytical skills, communication skills, and methodological skills. Possibly the Undergraduate Committee could work together with the larger Department to devise and implement such an assessment during the next phase of this process.

Regarding changes to the undergraduate political science curriculum, the Department believes it should not make any permanent changes at this time based on this first tentative assessment of learning outcomes. Once we have clearer benchmarks and time-series data beyond a couple of years, the Department can then consider possible revisions to the curriculum to improve our success in achieving the learning outcomes. We do believe, however, that both PolS 1101 and PolS 2401 are essential tools in educating GSU students to be active and aware American and global citizens. Multiple sources of data from our instructors and students make it clear that we are achieving our learning outcomes in these courses and the Department will continue to support the maintenance of PolS 1101 and PolS 2401 in the core curriculum. In addition, we should note that the revenues currently generated by enrollments in political science courses far outweigh the expenditures consumed by the Department, according to a recent analysis of cross-subsidies released by the University. The Department’s revenue/expense ratio is 4.2, which is well over the required “break-even” point of 2.6 and is in fact the second-highest ratio of all revenue-generating departments in the University (only Physical Therapy is slightly higher, at 4.3). Since the Department is therefore subsidizing many other departments in the University, it sees no reason to expand its course offerings or revise its basic curriculum in the absence of a more equitable distribution of resources.

IV. Changes in Department’s Assessment Goals

As the assessment process has not been completely vetted by the Department, at this point we might only suggest to expand the assessment goals to include substantive knowledge from other popular core courses in the political science major, such as PolS 3200 (Comparative Politics), PolS 3400 (International Politics), PolS 3140 (Judicial Process and Courts), PolS 4130 (American Constitutional Law), and so on. The Department also might want to consider ways to evaluate non-course learning experiences and related data in the overall undergraduate program instruction assessment process, such as internships, directed study projects, Model UN/Model Arab League, Moot Court, study abroad programs, and similar programs. These are very popular with our political science majors and the Department has enjoyed considerable success in terms of competitions with other institutions. As such experiences may play a significant role in our training of undergraduate students for future careers in law, diplomacy, or international business the Department should begin thinking of ways to incorporate measures of participation and success.

ASSESSMENT OF LEARNING OUTCOMES IN POLITICAL SCIENCE

Report on the Graduate Program, 2004-05

I. Assessment Procedures

Learning Outcomes and Assessment Methods

According to the Department’s Faculty Guidelines, the Graduate Program should be assessed according to how well it meets the following learning outcomes:

A. Ph.D. Program

- Demonstrate familiarity with the breadth and diversity of models, approaches, and intellectual traditions within that student’s major sub-field of expertise.

- Demonstrate competency in at least a second substantive area of political science.

- Demonstrate a high level of competency in research skills appropriate to his or her research endeavors and a familiarity with a broad range of methodologies, including quantitative and qualitative approaches.

- Demonstrate a full understanding of the research enterprise. This includes the ability to critique others’ work and an ability to be a contributing scholar by producing original research.

- Demonstrate an ability to teach courses in his or her primary field and sub-fields of the discipline.

B. M.A. Program

- Demonstrate research skills commensurate with their area of specialization.

- Demonstrate knowledge of the research literature in their area of specialization.

- Demonstrate ability to write a professional research report or thesis in their area of specialization indicating ability to formulate research questions, to synthesize such questions with appropriate literature, to utilize appropriate research methods to answer the question(s), and to analyze data so as to answer the question(s) and raise additional questions.

The methods for assessing achievement of these goals are described as:

A. Doctoral Program

1. The members of each doctoral dissertation committee will individually provide to the DGS a written assessment stating the degree to which the dissertation and its defense indicate success in achievement of the program’s stated learning outcomes.

2. Based on the program’s learning outcomes, the lead reader for each field or sub-field doctoral comprehensive committee shall write an assessment of the degree to which the answers provided by the students indicate success in achievement of the outcomes.

3. Utilizing syllabi and data from student evaluations of graduate students teaching courses, the DGS shall assess the competence of the doctoral graduate students in teaching courses.

B. M.A. Program

1. The members of each M.A. thesis committee or of non-thesis paper will individually assess the achievement of the student in terms of the program’s stated learning outcomes.

II. Assessment

Ph.D. Program

1. Dissertations. Three students completed the requirements for the PhD in Political Science during the 2004-05 academic year. Two completed the dissertation in Fall 2004, and one in Spring 2005. Two of the dissertations were written in the fields of Comparative/International Politics, and the third was written in the field of American Politics.

Following successful oral defense of each dissertation, the Director of Graduate Studies requested completion of a “PhD Learning Outcomes Assessment” form by each member of the respective dissertation committees (see Appendix 1). Summary data are provided below:

Average Scale Score (1=”Very Low Degree” to 5=”Very High Degree”)

To what degree does the dissertation demonstrate familiarity with the breadth and diversity of models, approaches, and intellectual traditions within the student’s major field of expertise? To what degree does the dissertation demonstrate a high level of competency in research skills appropriate to student’s research endeavors and a familiarity with a broad range of methodologies, including quantitative and qualitative approaches? To what degree does the dissertation demonstrate a full understanding of the research enterprise? This includes the ability to critique others’ work and an ability to be a contributing scholar by producing original research. To what degree does the dissertation demonstrate the student’s competency in a second substantive field of political science outside his/her primary field? To what degree does the dissertation demonstrate the student’s ability and preparedness to teach in his/her primary field and sub-fields of the discipline?

4.3 4.3 4.1 3.0 4.2

Open-ended comments from committee members provide additional evidence of achievement:

Comments on PhD 1

- “brings an impressive array of evidence about a less-studied phenomenon. He effectively frames that phenomenon and [has] made an important contribution to that area.”

- “An excellent dissertation which merits publication. [He] took the literature and research, expanded on it, and made new and interesting contributions to it. He was focused and strongly self-motivated; a developing scholar with growing research skills and competence.”

- “…dissertation demonstrated a strong grasp of the learning goals important to American politics research. It advances an important body of literature and does so with methodological sophistication.”

Comments on PhD 2

- “The dissertation makes a substantive contribution to our knowledge…More generally, perhaps we should have worked more with [him] to improve his English pronunciation so that we could have given him more teaching experience as a graduate student.”

- “In the end, this turned out to be a sold dissertation that contributed to the state of knowledge…One major weakness was difficulty in writing a literature review that situated the work within the literature. Strengths were that a variety of angles were used to find supporting evidence for the argument.”

- “Very detailed and comprehensive; path-breaking work…But could be stronger theoretically and more forceful with the argument.”

Comments on PhD 3

- “The dissertation met the departmental learning goals for the PhD program to a substantial extent.”

- “Important expansion of our understanding of the bases for international cooperation in global environmental agreements. Contribution to the field’s base of knowledge about international agreements in general is more limited.”

- “[Dissertation] demonstrated a very high level of knowledge about the empirical topic and a very high degree of methodological skill in terms of formulating and testing hypotheses through advanced statistical analysis. Specifically, [the student] seemed to have some difficulty in identifying the real puzzle(s) behind the topic (both theoretical and empirical), and then framing the entire project around those puzzles. There is an art to framing such puzzles in a creative and intriguing fashion, and [the student] could benefit from more guidance on that dimension.”

2. Comprehensive Examinations. Two students took written comprehensive examinations in December 2004. Both students failed to pass any of their three examinations (Comparative Politics, International Politics, African Politics, and Political Theory). Two students were re-examined in May 2005, with the results of those exams still pending.

Select comments from lead readers on those exams:

- “[student] fails to explore ideas in the depth and detail required of a comprehensive examination”

- “[student] should pay particular attention to answering the questions directly as posed”

- “surprised that the response to the mandatory question constituted little more than 1 ½ pages”

- “in its entirety the answer cites only four sources”

- “by not attaching dates with the authors cited the response does not provide a sense of the evolution of thinking within the field about this question”

- “[answer] suggests a less than adequate familiarity with the theoretical literature on foreign policy”

- “suspect that he could have done a better job if he had simply provided longer answers to the questions”

- “of the three answers, none answered the question completely, and two of them did not answer the question at all directly”

- “[student] evidences adequate familiarity with general issues in the study of politics and the broad contours of the study of international politics, but not of the relevant scholarly literatures in the study of international politics”

3. Teaching Abilities of Doctoral Students During 2004-05 doctoral students taught 34 course sections. Syllabi from each section in POLS 1101 American Government and POLS 2401 Global Issues were collected and assessed for conformity to departmental and university guidelines. The majority of the syllabi demonstrated such conformity, but several failed to provide adequate instructions regarding the last day to drop the course and receive a W, the class attendance policy, penalties for academic dishonesty, and guidelines for students with disabilities.

The teaching abilities of doctoral students are enhanced through their enrollment in EPY 9000 prior to teaching their own course section. Efforts were also made during the year, especially by the POLS 2401 Coordinator, to monitor teaching effectiveness by twice visiting each graduate instructor’s class session.

Student evaluations of graduate instructors constitute another indicator of teaching ability. Below is a summary of scores from evaluations conducted in Fall 2004. Scores for Spring 2005 are not yet available.

Course	Semester	Date	#1 #10	#2 #11	#3 #12	#4 #13	#5 #14	#6 #15	#7 #16	#8 #17	#9
POLS 1101	Fall	2004	4.6 4.5	4.5 4.7	4.5 4.7	4.6 4.3	4.5 4.3	4.3 4.2	4.4 4.4	4.5 4.3	4.6
POLS 2401	Fall	2004	4.6 4.6	4.6 4.7	4.6 4.7	4.7 4.5	4.6 4.5	4.4 4.4	4.4 4.6	4.5 4.5	4.7
POLS 3145	Fall	2004	4.7 4.7	4.7 4.8	4.8 4.8	4.8 4.6	4.8 4.6	4.6 4.5	4.7 4.7	4.6 4.6	4.8
POLS 4130	Fall	2004	4.6 4.6	4.6 4.6	4.5 4.7	4.8 4.4	4.8 4.5	4.6 4.3	4.5 4.6	4.6 4.5	4.9
POLS 4131	Fall	2004	4.8 4.7	5.0 4.7	4.8 3.9	4.3 4.8	4.5 4.8	4.8 4.7	4.8 5.0	4.7 4.4	4.5

Student Evaluation Questions

1. Explained the goals of this course clearly.

2. Explained the grading system clearly.

3. Gave assignments related to the goals of the course.

4. Followed the plan for the course as established in the syllabus.

5. Was well prepared.

6. Spoke in a way that communicated the subject in an understandable manner.

7. Responded constructively and thoughtfully to questions and comments.

8. Used class time effectively.

9. Had designated office and student appointment hours and was available to students during these times.

10. Assigned grades fairly.

11. Returned test results and evaluations of my work in a reasonable period of time (typically, 7-10 days or less is considered a reasonable College benchmark)

12. Met the class according to the published Schedule of Classes.

13. Stimulated my thinking and gave me new insights into the subject.

14. Related well to students.

15. Motivated me to learn.

16. Assigned readings (including the text(s)) that contributed to what I learned.

17. Considering both the limitations and possibilities of the subject matter and course, how would you rate the overall teaching effectiveness of this instructor?

Sample of open-ended student comments:

Evidence of teaching effectiveness

- “…is an excellent [instructor] who above all else responded well and thoughtfully to students’ questions.”

- “Excellent teacher but requires actual studying and hard work.”

- “The instructor was very prepared every class period. She made our learning process interesting and made the material easy to understand. In my whole four years in college, she is by far the best teacher I’ve had.”

- “an excellent professor that relates well to all ages and ethnic backgrounds.”

- “Despite having to teach so many people in such a large room, the instructor kept the class an interactive one.”

- “…took extra efforts to be fair and present both sides of an issue.”

Weaknesses

- “The class was interesting, though sometimes the instructor seemed unprepared.”

- “Tests are pretty simple…She’s very easy to please and I recommend her!!!!!”

- “…did not communicate well what she expected out of the class…was not clear about what she wanted in papers.”

- “did not allow for challenging discussions”

- “…did not focus on the subject material and spent the majority of class time talking about current events which did not necessarily pertain to class”

- “The only problem I encountered is that I felt as if I wasn’t being motivated to attend class on a regular basis.”

- “…sometimes student commentary and debate was not controlled properly”

- “I find it inappropriate for an instructor in a POLS course to voice his/her political opinions, especially in a society as polarized as ours around political parties.”

As is always the case in interpreting student evaluations, it is necessary to also provide data on grade distribution. Below is the distribution of grades by graduate instructors in Fall 2004.

Distribution of Grades in Courses Taught by Doctoral Students, Fall 2004

	A	B	C	D	F	AVG
POLS 1101	178	219	118	44	45	2.7
POLS 2401	134	72	35	17	17	3.0
POLS 3145	7	19	9	2	5	2.5
POLS 4130	10	24	7	1	2	3.0
POLS 4131	9	19	8	0	1	2.9

MA Program

Twelve students completed the requirements for the MA in Political Science during the 2004-05 academic year. Five completed the thesis or non-thesis requirements in Fall 2004, and seven did so in Spring 2005. Seven of the MA projects were written in the field of American Politics, four were in Comparative/International Politics, and one was in Political Theory.

Following approval of each thesis or non-thesis paper, the Director of Graduate Studies requested completion of a “MA Learning Outcomes Assessment” form by each member of the respective faculty evaluation committees (see Appendix 2). Summary data are provided below:

Average Scale Score (1=”Very Low Degree” to 5=”Very High Degree”)

To what degree does the thesis/non-thesis paper demonstrate research skills commensurate with the student’s area of specialization? To what degree does the thesis/non-thesis paper demonstrate knowledge of the research literature in the student’s area of specialization? To what degree does the thesis/non-thesis paper demonstrate the ability to write a professional research report in the student’s area of specialization? a. ability to formulate research questions b. ability to locate those questions within the appropriate literature c. ability to utilize appropriate research methods to answer the question(s) d. ability to analyze data to answer the question(s) e. ability to raise additional questions based on interpretation of research findings

3.7 3.9 4.1 3.9 3.7 3.6 3.8

Open-ended comments from committee members provide additional evidence of achievement:

- “Best non-thesis paper I’ve read; better than most theses.”

- “Thesis was well done and likely exceeded the department’s learning goals for MA students.”

- “This paper was adequate and likely met the department’s goals for learning outcomes for a non-thesis research project. However, the paper did not exceed expectations.”

- “…has written a quality non-thesis MA paper….has shown mastery of the literature and contributed important material to the sub-field.”

- “…the paper did meet the standard of being original, generally well written, and it was on a generally important and interesting topic…[student] just didn’t really ‘pull it off’ as far as I was concerned.”

- “…paper was rushed and could have benefited from one more revision…results could have been presented more clearly and forcefully.”

- “This thesis was thought provoking since it took a perspective counter to much of the published research in the field. It was well defended and rooted in research literature…It did not involve data analysis or quantitative methods, but it was well presented and enjoyable.”

- “Well written and stimulating—if [student] is the caliber of our grad students we are in good shape.”

- “[student] did not follow my advice on assessing the huge literature on the topic or testing hypotheses.”

- “The paper met the learning goals to a minimally satisfactory extent.”

- “[This thesis] might make a publishable research note.”

- “…thesis was well done. It shows great promise as a project that could be published in a peer-reviewed journal. The project also holds promise for a doctoral dissertation.”

- “…paper was weakly organized and alternative explanations were not pursued. It really needed one more revision.”

III. Analysis and Interpretation

The data collected in 2004-05 suggest a number of strengths and weaknesses in the Department’s efforts to meet its stated learning goals.

Doctoral Program. Although the number of doctoral dissertations completed during the time period is not as high as we might wish, the quality of each appears significant. On four of five standard learning outcome indicators, faculty committee members judged the achievement of learning outcomes to be greater than 4.0 on a 5-point scale. Dissertation projects effectively demonstrate familiarity with intellectual traditions in the students’ major fields, a high degree of competency in research methodologies, a clear capacity to make original scholarly contributions, and the preparedness to teach in the field. We must note, however, that on a fifth indicator—competency in a second substantive field—the scores were markedly lower. It is unclear if this score represents a substantive problem, or if it is simply more difficult to demonstrate achievement of this learning outcome in a doctoral dissertation that may, by definition, be focused narrowly on contributions to a single primary field. In reviewing its learning goals, the Department may wish to reconsider whether a dissertation in, say, American Politics should really be expected to demonstrate competency in a second substantive field of political science. Demonstration of such mastery may be most likely to occur in comprehensive written examinations, rather than in the dissertation.

Performance on written comprehensive examinations is clearly deficient, at least as measured by the performance of the two candidates who took exams in December 2004. The Director of Graduate Studies spent much time in discussion with the faculty evaluation committees to discern the root causes of such poor performance. The collective conclusion is that the problem lay not with graduate student instruction, nor with examination questions, nor with exam format. The two students simply failed to adequately prepare for the exams, including the failure to consult with relevant faculty members during the months prior to exams.

The demonstrated teaching abilities of doctoral students are impressive. Evidence suggests that our doctoral students are indeed able to teach courses in their primary fields as well as in more specialized sub-fields of the discipline. The scores on standardized student evaluations range between 4.3 and 4.6 on the measure of overall instructor effectiveness. Student feedback does raise a few concerns about pedagogy and class management, some of which are likely to be allayed with instructor experience. One note from the distribution of grades awarded by graduate instructors: the comparatively high number of A’s given in POLS 2401 may be some cause for discussion.

Other indicators:

Achievement of learning goals further evidenced by:

- 14 graduate student research papers accepted and presented at professional conferences

- Two successful applications for GSU Dissertation Grants

- $45,000 in external research grant funds received by one doctoral student

- 2004-05 PhD recipients hired to positions at: Kennesaw State University, Government of Mauritius, and postdoctoral fellowship in Chicago

- ABD hired to tenure-track faculty position at Vanderbilt University

Masters Program. Evidence on learning outcomes at the MA level suggests variable success. On the 5-point scale evaluating success on seven dimensions, the average score is 3.8. We should set a goal of at least 4.0 for all stated learning outcomes. MA graduates seem best able to effectively formulate research questions but are somewhat less adept at analyzing data and integrating their findings with the extant literature. Faculty feedback on MA projects indicates that some studies are indeed outstanding, while some (indeed too many) only minimally meet the target goals. There is also faculty concern that the non-thesis option for MA students too frequently results in papers that are of only minimally acceptable quality.

III. Modifications Based on Assessment Results

Training graduate students is both a collective enterprise and an individual process of mentoring. As a collective body, there are some steps that the Department can take to strengthen its success in meeting learning outcomes goals. Among these recommendations are:

The Graduate Committee should meet to discuss whether and how doctoral students can demonstrate mastery of a second major field in the dissertation. This current expectation has not been satisfactorily met in recent dissertations. The Graduate Committee should make a recommendation to the full faculty on either modifying the goal or strategies to more effective achievement of the existing goal.

All teaching faculty should redouble their efforts to train graduate students in research design. This tasks starts, but clearly does not end, with POLS 8800. Maximizing feedback on the design of research projects over the course of a semester—rather than after the submission of the final research paper at the end of the semester—may result in more effective mastery of the fundamental elements of the research process. Iterative writing and project design are to be prioritized.

The Director of Graduate Studies has already taken steps to improve student performance on doctoral written comprehensive examinations. To increase clarity about expectations, the DGS wrote and distributed to all doctoral students a four-page guide to preparing for the exams. The DGS has also allowed exam-takers greater access to old comprehensive examinations also as a means of clarifying expectations.

The DGS and coordinators for POLS 1101 and 2401 should meet with all TAs and graduate instructors prior to the start of each academic semester to reinforce expectations regarding grading, policies on plagiarism, professional ethics and conduct, and the fundamentals of writing syllabi.

The DGS and Graduate Committee should meet to discuss the quality of MA non-thesis papers. The primary task should be to clarify student and faculty expectations about the paper. At present, Departmental policy reads:

“The non-thesis research paper should be a seminar paper, extended and enhanced to meet the standards of a conference paper presentation. These standards include a statement of the problem, questions or hypotheses raised by the problem, and either a major review and critique of the literature addressed to the problem and relevant questions, or original research on that problem and relevant questions. Conference papers are normally 20 to 40 pages in length and should address an important problem, reflect knowledge of the relevant literature as it pertains to the questions, and provide preliminary answers based on competent research to the questions raised by the problem.”

Overall, there needs to be greater and more systematic discussion among the members of the Graduate Committee and among the full faculty about learning outcomes for the MA and PhD programs. Variable standards will most likely result in variable performance, so consensus and clarity about what constitutes desirable learning outcomes is important.

APPENDIX 1. PhD Learning Outcomes Assessment

Faculty Member Name: ______________________________

Doctoral Student Name: ______________________________

Based on the learning outcomes specified for the doctoral program, the members of each doctoral dissertation committee are asked to individually provide this written assessment stating the degree to which the dissertation and its defense indicate success in achievement of these outcomes. The assessment should be returned to the Director of Graduate Studies within a week of the oral defense.

Please circle the number that best describes achievement of each learning goal:

Very Little Degree Very High Degree

To what degree does the dissertation

demonstrate familiarity with the breadth

and diversity of models, approaches, and

intellectual traditions within the student’s

major field of expertise? 1……….2……….3……….4……….5

To what degree does the dissertation

demonstrate a high level of competency in

research skills appropriate to student’s research

endeavors and a familiarity with a broad range

of methodologies, including quantitative and

qualitative approaches? 1……….2……….3……….4……….5

To what degree does the dissertation

demonstrate a full understanding of the research

enterprise? This includes the ability to critique others’

work and an ability to be a contributing scholar by

producing original research. 1……….2……….3……….4……….5

To what degree does the dissertation demonstrate

the student’s competency in a second substantive field

of political science outside his/her primary field? 1……….2……….3……….4……….5

To what degree does the dissertation demonstrate

the student’s ability and preparedness to teach in

his/her primary field and sub-fields of the discipline? 1……….2……….3……….4……….5

COMMENTS: [Please provide an overall, descriptive assessment of how well the dissertation met the department’s learning goals for the PhD program. In providing these comments, you may wish to compare this dissertation with others completed in the department with which you have worked or have knowledge.]

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

APPENDIX 2. M.A. Learning Outcomes Assessment

Faculty Member Name: ______________________________

Masters Student Name: ______________________________

Based on the learning outcomes specified for the M.A program, the members of each thesis or non-thesis are asked to individually assess the student’s achievement of those learning goals. The assessment should be returned to the Director of Graduate Studies within a week of the defense of the thesis or committee approval of the non-thesis paper.

Please circle the number that best describes achievement of each learning goal:

Very Little Degree Very High Degree

To what degree does the thesis/non-thesis

paper demonstrate research skills commensurate

with the student’s area of specialization? 1……….2……….3……….4……….5

To what degree does the thesis/non-thesis paper

demonstrate knowledge of the research literature

in the student’s area of specialization? 1……….2……….3……….4……….5

To what degree does the thesis/non-thesis paper

demonstrate the ability to write a professional

research report in the student’s area of specialization?

a. ability to formulate research questions 1……….2……….3……….4……….5

b. ability to locate those questions within

the appropriate literature 1……….2……….3……….4……….5

c. ability to utilize appropriate research

methods to answer the question(s) 1……….2……….3……….4……….5

d. ability to analyze data to answer the

question(s) 1……….2……….3……….4……….5

e. ability to raise additional questions based

on interpretation of research findings 1……….2……….3……….4……….5

COMMENTS: [Please provide an overall, descriptive assessment of how well the thesis/non-thesis paper met the department’s learning goals for the M.A. program. In providing these comments, you may wish to compare this research project with others completed in the department with which you have worked or have knowledge.]

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

____________________________________________________________________________________________

___________________________________________________________________________________

ss in our assessment goals and processes.

---

[1]Q1: Course goals were described clearly.

Q2: Instructor was well prepared.

Q3: Instructor communicated in an understandable manner.

Q4: Instructor stimulated student thinking.

Q5: Overall instructor effectiveness.