Department of Psychology
Learning Outcomes Assessment Report
Academic Year 2004-2005
Undergraduate Program Graduate Programs Assessment Data
|
III. Changes to Procedures or Curriculum Based on Assessment |
|
|
II. Achievement
of Departmental Objectives for Students A. Undergraduate--
|
I.
Assessment Procedures -
Undergraduate
A. Outcomes
Assessed.
The Department of Psychology assessed all learning outcomes listed in Table 1 except C2: Students articulate how psychology can further social understanding and public policy and E3: Students analyze phenomena at multiple levels of analysis including the biological, individual, family, community, and society. Table 2 illustrates courses in which outcomes were assessed.
Table 1
Department of Psychology Undergraduate Learning Outcomes
|
A. Theory and Content:
Demonstrate familiarity with major concepts, theoretical perspectives,
empirical findings, and historical trends |
|
1. Students learn the historical development of the discipline, its contemporary context (including
social and political contexts, organizational and self-governance), and
interaction with other disciplines. |
|
2. Students learn key psychological theories and concepts (e.g.
biological, psychological, and social bases of affect, behavior, and
cognition) and the nature and scope of supporting data. |
|
B. Research Methods:
Understand and apply basic research methods including research design, data
analysis, and interpretation. |
|
1. Students develop testable hypotheses, differentiate research
designs and/or statistics, evaluate aptness of research conclusions, and
generalize them appropriately. |
|
2. Students design and conduct quantitative or qualitative research
studies in laboratory or field settings. |
|
3. Students adhere to ethical guidelines for collection, storage, and
use of data from human or non-human participants. |
|
4. Students use print and electronic library resources
effectively and appropriately. |
|
C. Application: Understand
and apply psychological principles in personal, social, and organizational
matters. |
|
1. Students identify psychology’s major applications in
laboratory and field settings (e.g. clinical, industry, education). |
|
2. Students articulate how psychology can further social
understanding and public policy. |
|
D. Communication and
Collaboration skills: Communicate and work in groups effectively |
|
1. Students demonstrate effective written communication skills and
use discipline specific writing conventions and formats. |
|
2. Students demonstrate effective oral communication skills. |
|
3. Students work effectively within groups or teams. |
|
E. Critical thinking
skills: Respect and use critical and creative thinking, skeptical inquiry and
the scientific approach. |
|
1. Students use research data to formulate or evaluate new
research questions, using reason and persuasion in a logical argument. |
|
2. Students summarize and evaluate a body of research including primary
literature, and can compare psychology’s methods with other disciplines’
methods. |
|
3. Students analyze phenomena at multiple levels of analysis
including the biological, individual, family, community, and societal. |
|
F. Personal development:
Shows insight into one’s own and others’ behavior and mental processes and
apply effective strategies for self-management and self-improvement. |
|
1. Students apply psychology to personal and professional
development. |
|
2. Students are aware of their thoughts, feelings, and
behaviors. |
|
3. Students define personal and professional integrity. |
|
G. Information and
Technological Literacy: Demonstrate information competence and the ability to
use computers and other technology for many purposes |
|
1. Students demonstrate competent, ethical, and responsible use
of information in academic work. |
|
2. Students apply software in research reports (e.g statistical) |
|
3. Students master computer basics such as Internet navigation,
document and spreadsheet generation. |
|
4. Students assess web-based sources of information, popular
presentations of psychological research, as well as pseudoscience. |
|
H. Values in Psychology:
Weigh evidence, tolerate ambiguity, act ethically, and reflect other values
underpinning psychology |
|
1. Students understand the need to behave ethically in personal
and professional domains, and appreciate the need to tolerate ambiguity. |
|
2. Students demonstrate skepticism and intellectual curiosity, attunement
to scientific evidence, civic responsibility, and respect for human
diversity. |
|
I. Sociocultural
and International Awareness |
|
1. Students respect individual differences. |
|
2. Students define diversity and its role in psychological theory and
research. |
|
3. Students consider and explain the role of cultural, racial, ethnic
and economic factors, privilege, and discrimination, in affect, behavior, and
cognition. |
|
J. Career Planning and Development:
Emerge from the major with realistic ideas about how to use psychological
knowledge, skills, and values in various occupations, and in graduate or
occupational schools. |
|
1. Students apply psychological principles to career
decision-making. |
|
2. Students identify and pursue realistic career paths. |
|
3. Students identify realistic graduate education pathways. |
|
4. Students take practical career steps. |
|
5. Students value lifelong learning and ongoing professional
development. |
Table 2
Learning Outcomes Assessed in Psychology Major Curriculum
|
Learning Outcomes |
COURSE |
|
A1-A2 |
1101 |
|
B1-B4
& E1-E2 |
3010 |
|
B1-B4, C1, D, E1-E2, &
G |
3030 |
|
D2, F1-F3, & H1 |
3110 |
|
B, D, E, G, H, & J |
4760 |
|
A, C, I, & J |
4770 |
B. Elements
of Assessment / Data Collection / Data Analysis.
PSYC 1101 is a broad
survey of the major topics in psychology, including research methodology, the
biological and social factors that influence behavior, development, learning,
memory, personality, and psychological disorders. The course can be used to fulfill part of the
Social Science core requirement (Area E) for the
Sample & Procedure
Instructors administered a 50-question, pencil-and-paper mastery test to all students in all sections at the end of the Fall 2004 and Spring 2005 terms. All of the questions were multiple-choice with four or five options. Questions focused on basic psychological theories and phenomena, with a particular emphasis on the application of these concepts to real-life situations and current events.
Instructors were given the
option of administering the mastery test by itself, during the last week of
classes or as part of the final exam. In the latter case, the 50 mastery test
questions were always presented as a block before or after any additional
questions the instructor included in the final exam. Each semester, two instructors chose to give the mastery test before the final exam; the remainder
(six for Fall 2004 and five for Spring 2005)
incorporated the mastery questions in to their final exam. With one exception in Fall
2004, the questions were presented in the same order and prior to any other
section-specific questions.
At the end of
each semester, instructors submitted the original exam answer sheets for
independent analysis by the course coordinator.
Prior to analysis, these sheets were coded by section using a letter of
the alphabet that was unrelated to the instructor’s name or the official
computer registration number. The sheets
were processed by the
Analyses & Results
Students’ scores (0=wrong; 1=correct) were summed across the multiple choice questions, and their percentage of correct answers was calculated.
For the Fall 2004 term, data were collected for 781 students. The median percent correct for all 50 questions was 75%. The percentage correct ranged from a low of 31% to a high of 99%. Sixty percent of the students scored 70% correct or higher on the mastery test as a whole.
For the Spring 2005 term, data were collected for 513 students. The median percent correct for all 50 questions was 72%. The percentage correct ranged from a low of 34% to a high of 98%. Slightly more than one-half of the sample (51%) scored 70% correct or higher on the test overall.
PSYC 3010 provides an introduction to descriptive and inferential statistics as applied to psychological research questions and is required for all Psychology majors. Four sections are offered each semester (Fall and Spring), with typical enrollments between 48 and 75 students. Topics include measures of central tendency and variability, an introduction to probability and hypothesis testing, the t test for independent and related samples, analysis of variance including repeated measures, correlation and regression, and chi-square tests of independence. Concepts are introduced and discussed in class through lecture and discussion and then are applied through exercises in laboratory and homework assignments. Students who successfully complete this course should be able to understand and apply basic research statistics including data analysis and interpretation.
Sample & Procedure
Instructors were asked to randomly sample students from their sections for a projected sample size of N = 150 across all sections offered during the academic year . Data were collected on a set of standardized multiple choice questions on key concepts and computations assessed throughout the semester. Additionally, instructors rated students’ written interpretation of key statistics (i.e., t-test, ANOVA, correlation) and students’ use of APA style in the write-up. These open-ended questions were rated on a scale from 1 to 5. A score of 1 on interpretation means that the student completely misunderstood the statistical procedure. A score of 1 on APA format means that the student made more than 5 errors in the write-up. A score of 5 on interpretation means that the study interpreted the statistical procedure and results completely correctly, and a score of 5 on APA format means no errors were made in the write-up. Finally, students were administered a 10-question pre-test on the first day of the semester, and they completed an identical post-test on the last day of the semester.
Analyses and Results
Multiple choice. Valid multiple choice data were collected on 124 students. Students’ scores (0=wrong; 1=correct) were summed across the multiple choice questions, and their percentage of correct answers was calculated. The median percent correct was 74%, and the percentage correct ranged from a low of 26% to a high of 100%.
Open-ended questions. Valid data were collected on 103 students. Students’ ratings were averaged across statistical procedure for interpretation and APA format. On the 1-5 scale, the median interpretation score was 2.67 (M = 2.62, SD = .89), with a range from 1 to 4. The median APA score was also 2.67 (M = 2.50, SD = .77), with a range from a low of 1 to a high of 4.
Pre- and post-test. A total of 83 students had valid pre- and post-test data. The median pre-test score was 3 out of 10 correct, with a range from 0 correct to 4 out of 10 correct. The median post-test score was 6 out of 10 correct, with a range from 2 to 10. A paired sample t-test was computed to examine whether students got significantly more questions correct on the post-test than on the pre-test. The results indicated that they did perform better on the post-test (M = 6.45, SD = 1.70) than on the pre-test (M = 2.39, SD = 1.25), t(82) = 18.65, p < .001.
PSYC 3030 Principles and Methods of Psychological Investigation: Outcomes B1-B4, C1, D, E1-E2 & G
PSYC 3030 is a
small, writing intensive course that meets a major requirement for the
Department of Psychology. The focus is
to provide students with a detailed knowledge of the variety of research
methods that psychologists use, the range of decisions that psychologists must
make when planning, conducting, and presenting research, and the factors that
must be considered when making these decisions.
The course has a lecture and laboratory.
Labs involve the design of studies, collection of data, and analysis of
data. As a part of both the lecture and
lab, students develop familiarity with the style of the American Psychological
Association and practice writing in that style.
Typical enrollment for the course is approximately 130 students in the Fall (five sections) and 160 students (six sections) in the
Spring.
Sample & Procedure
For this
assessment, all students in all sections of the course were required to
complete two specific writing assignments.
The first was a 3-4 page essay on research ethics, including an
assessment of the compliance of a particular study with the APA code of ethics;
this was submitted during the third week of the course. The second assignment was an original,
full-length research paper, based on data they themselves collected and
analyzed over the course of the semester; this was submitted during the last
week of the course. Both assignments required students to use the current
version of APA style and incorporate information from one or more peer-reviewed
journal articles. Students were told to
put forth their best effort with regard to grammar, spelling, paragraph
structure, and argument structure.
Students submitted electronic versions of both assignments to their instructors, who then forwarded these to the course coordinator each semester. The coordinator then randomly selected 20 of the students who had completed both assignments and assessed these assignments in terms of the quality of writing overall and the use of APA style. Overall writing quality was defined as the organization of the essay and the author’s ability to “clearly distinguish between fact and opinion,” “support factual statements with observable evidence,” “motivate opinions with well-reasoned arguments,” and “use wording that is clear and concise.” A score from 1-5 was given for each of these writing elements, for a total possible overall writing score of 25. The use of APA style was defined in terms of the author’s ability to appropriately paraphrase the ideas of others, without plagiarizing, provide all necessary citations and references in the correct format, present ideas in an objective manner, and follow APA style guidelines for the presentation of numbers and the use of abbreviations. A score from 1-5 was given for each of these five APA elements, for a total possible APA style score of 25. The total of the overall and APA scores was used as the final assessment score for an assignment.
For each semester, the course coordinator compared the assessment scores for the ethics essays (pretest) with those for the research papers (posttest). Differences in the scores were intended to reflect the development of the students’ writing skills over the term.
Analyses & Results
For the Fall 2004 term, the assessment scores for the ethics papers had a median value of 34, with a range of 25 to 47. The scores for the research papers had a median of 42.5 and a range from 32.5 to 47.5. A paired sample t-test showed that the ethics paper scores (M = 34.9, SD = 7.1) were significantly lower than the research paper scores (M = 41.0, SD = 6.6), t(19) = -4.2, p < .05.
For the Spring 2005 term, the median assessment score for the ethics papers was 36.25, with a range of 25 to 46. The scores for the research papers had a median of 42.5 and a range from 25 to 48. Again, a paired sample t-test revealed the assessment scores for the ethics papers (M = 34.9, SD = 7.4) to be significantly lower than those for the research papers (M = 38.8, SD = 7.2), t(19) = -4.2, p < .05.
PSYC 3110 (Psychology of Interpersonal Behavior) was originally developed within the Department of Psychology at GSU to facilitate undergraduate students’ psychological understanding and application of interpersonal skills. Accordingly, the evaluation strategy used to assess learning outcomes in this course focused on students’ acquisition of 11 different communication skills. These skills, which are systematically taught in the course via didactic and experiential methods, fall into two domains: listening and talking. Listening skills include attending, acknowledging, inviting, summarizing, and asking open-ended questions. Talking skills include speaking for ones’ self, describing sensory data, expressing thoughts, sharing feelings, disclosing wants, and stating actions. Approximately ten sections, with maximum enrollments of 26 students, are offered each semester (Fall and Spring).
Students enrolled in ten of 21 sections of PSYC 3110 during Fall 2004 and Spring 2005 completed a pre- and post-test self-report questionnaire. These sections were taught by six different instructors. Valid data were not available from all of the students enrolled in PSYC 3110 in the 2004-2005 academic year because either instructors failed to administer the questionnaires or students did not follow directions. An N of 150 was available to analyze the self-report data.
Three instructors responsible for a total of four sections were asked to complete a behavioral evaluation. Data from all four sections were available. After dropping the behavioral rating forms with incomplete data, an N of 64 was included in this phase of the evaluation.
The pre-and post-test self-report measure, “Gauge My Progress,” was drawn from the Core Communication (Miller & Miller, 1998) workbook used in the course. Consisting of questions pertaining to the 11 communication skills listed earlier, it asks the respondent to rate on a 5-point scale his or her typical and desired behaviors vis-à-vis the different skills. Improvement in communication skills is indicated by an increasingly smaller differential between ratings of typical and desired behaviors. The pre- and post-test measures were administered on the first and last days of class, respectively.
Developed for purposes of this evaluation, the behavioral measure assessed students’ maximal performance in an interpersonal role-play situation in which they worked in groups of three rotating through the roles of listener, talker, and evaluator. The role-plays involved relational conflicts of an ethical nature. The students were evaluated on their ability to display each of the five listening and six talking skills at least once during the role-plays. To reach criterion in listening and talking, they had to demonstrate at least 80% of the listening skills and 80% of the talking skills. The behavioral measure was administered during the last week of classes.
Data Analysis
Average typical-desired rating differences pre- and post-test across the eight sections were 41.71 (SD = 22.06) and 37.94 (SD = 22.64), respectively. This difference was significant, t (149) = 2.15, p < .05.
Combining the results of the four sections that completed the behavioral portion of the evaluation, 72.20% of the students exhibited criterion level performance for listening skills and 81.80% for talking skills.
The goal of research practicum is to expose undergraduates to the mechanics of psychological research in a professional setting, thereby helping students develop realistic career goals. Thus, the evaluation strategy used to assess learning outcomes in this course assessed both professional development as well as research related learning outcomes.
Evaluations were completed for 49 students enrolled in PSYC 4760 during the fall and spring semesters. Data were missing for 8 students.
Procedures
Faculty completed evaluations of students’ performance and returned the evaluation to the practicum coordinator. The evaluation consisted of 18 questions and a space for comments. Faculty used a likert scale (1 = poor to 5 = excellent) to respond to all questions. Five questions assess outcome H (values in psychology, i.e. professional development outcomes). The average score across these questions was calculated. Ten questions assessed outcomes B, D, E, & G (research relevant outcomes) and as above, average scores were calculated. The remaining 3 questions were not specific to the evaluation plan. To assess career planning and development (outcome J), students wrote a brief self-assessment and listed five experiences that described their practicum. This list was used to evaluate whether the experiences helped the student refine career goals.
The mean score for professional development items was 4.84 (SD=.40). These scores ranged from a minimum of 2.60 to a maximum of 5.0. Only 2% or 1 student received a score below 4 (good).
The mean score for research outcomes was 4.58 (SD=.56). These scores ranged from a minimum of 3.0 to a maximum of 5.0. Six students (18%) received a score below 4 (good).
Eighty-six percent of students enrolled provided a list of 5 experiences that characterized their practicum. Thirty-five percent specifically mentioned that their practicum experiences either confirmed or refined their career goals. Only one student rejected a psychology related career goal as a result of the practicum. Fifty percent mentioned at least one specific skill they acquired and an equal amount mentioned gaining at least one new important insight into the process of research or academic life.
The goal of applied practicum is to expose undergraduates to a professional setting in psychology or an allied field, to facilitate the development of realistic career goals. Thus, the evaluation strategy used to assess learning outcomes in this course assessed both professional development as well as learning outcomes related to the application of psychological theory and content.
Forty-six students were enrolled in PSYC 4770 during the fall and spring semesters. Evaluations regarding outcome H were completed for 43 and evaluations for remaining outcomes were completed for 41 students.
Procedures
Field supervisors completed evaluations of students’ performance and returned the evaluation to the practicum coordinator. The evaluation consisted of 13 questions and a space for comments. Faculty used a likert scale (1 = poor to 5 = excellent) to respond to all questions. Five questions assess outcome H (values in psychology, i.e. professional development outcomes). The average score across these questions was calculated. Five questions assessed outcomes A, C, & I (application relevant outcomes) and as above, average scores were calculated. The remaining 3 questions were not specific to the evaluation plan. To assess career planning and development (outcome J), students wrote a brief self-assessment and listed five experiences that described their practicum. This list was used to evaluate whether the experiences helped the student refine career goals.
The mean score for professional development items was 4.73 (SD=.48). These scores ranged from a minimum of 3.20 to a maximum of 5.0. Only 9% or 4 students received a score below 4 (good).
The mean score for application outcomes was 4.64 (SD=.54). These scores ranged from a minimum of 3.2 to a maximum of 5.0. Five students (12%) received a score below 4 (good).
Ninety-four percent of students enrolled provided a list of 5 experiences that characterized their practicum. Fifty-three percent specifically mentioned that their practicum experiences either confirmed or refined their career goals. Only three students rejected a psychology related career goal as a result of the practicum. Thirty-two percent mentioned at least one specific skill they acquired and over 90% mentioned gaining at least one new important insight into the professional and personal requirements for applied work.
Assessment
Procedures - Graduate
A. Outcomes Assessed.
In March, 2005, the Psychology Department faculty agreed on a set of 10 learning goals for graduate students. The goals were derived from guidelines for graduate training developed by the American Psychological Association, and were revised and expanded by the faculty to better suit the aims of our graduate programs. Outcomes were assessed to address each of these goals. Specifically, learning outcomes were assessed in the following domains: 1. expertise in theory and content, 2. expertise in research methods, 3. application of psychological principles in professional activities, 4. communication and collaboration skills, 5. critical thinking skills, 6. personal development, 7. information and technology literacy, 8. ethics and values, 9. sociocultural awareness, and 10. career planning and development. See Attachment A for a description of each of the learning goals and representative skills.
B. Elements of Assessment.
For each learning goal, a set of from 4 to 12 assessment strategies was specified. Across the various outcomes, the elements of assessment include student performance (grades) in course work relevant to each goal, student progress and performance on master’s theses and doctoral dissertations, faculty evaluations of general exams completed usually in the third year of graduate study, yearly faculty reviews of each student as summarized in faculty ratings (clinical program only) and yearly evaluation letters, semester evaluations of practicum performance by practicum supervisors and, for graduate students who taught courses, teaching portfolios and student course evaluations.
C. Data Collection.
Course grades are available on-line through university records. The assistant director of graduate studies maintains a database to track the completion of major milestones, including proposal and defense of the masters thesis and dissertation, and completion of the general exam and the outcome (pass/fail). The database is updated on an ongoing basis as milestones are achieved. The assistant director also keeps records on any student placed on academic probation by the college for poor grades. Yearly faculty reviews of each student are completed during the fall semester. For students enrolled in the clinical program (approximately 50% of all graduate students), ratings are solicited from all faculty members for every student the faculty member has supervised during the year. The ratings are completed by course instructors, supervisors of graduate assistantships, teaching supervisors, graduate advisors who direct masters theses and dissertations, thesis and dissertation committee members, practicum supervisors, and general exam committee members. The ratings are made in any of four content areas, including academic performance, research-scholarship, clinical skills, and professionalism. The ratings are given on a six-point scale from 1= poor to 6 = exceptional. Faculty members also make written comments about the student. For students in programs other than clinical, evaluative comments are gathered from faculty members in program-area faculty meetings. In all programs, these data are summarized in a yearly evaluation letter kept in the student files, with a copy distributed to the student. Clinical students who are enrolled in practicum also received written evaluations by the supervisor each semester, which are kept on file by the clinical program and reviewed by the faculty are part of the yearly review process. Graduate students who teach courses enroll in a teaching practicum seminar (PSYC 9960A/B), for which they submit teaching portfolios that are evaluated by the course instructor. Also, student course evaluations are obtained for all courses.
D. Data
Analysis.
The data were compiled and summarized by the assistant director and director of graduate studies, into a chart listing the assessment methods, criteria, and outcomes for each learning goal. This chart is Attachment B. The criterion for success for course grades is a grade of B or better, or S. For the completion of masters theses, dissertations, and general exams, the percent of students who passed was calculated. In addition, at the beginning of the fall term, the graduate program committee reviews the progress of all graduate students to identify students who are not meeting progress guidelines for completing these milestones as stipulated in the graduate program handbook. These students are placed on probation and required to develop a plan for the completion of the milestone in a timely manner. The number of students placed on probation by the committee was used as a second outcome criterion for these milestones. The yearly evaluation ratings were examined to determine the percent of students who obtained ratings of at least 4 = very good for each of the four domains evaluated. The yearly faculty review evaluation letters were reviewed by the graduate program director to identify students who were seen as having significant problems in relations to each of the learning outcomes. This review also identified the number of students who were commended for giving presentations at professional conferences and for publishing articles or book chapters during the prior year. The graduate program director also reviewed all teaching course evaluations to identify students with significant problems.
II.
Achievement of Departmental Objectives for
Students
Our goal was for students to answer 70% of the mastery questions correctly. Between 51-60% of students met this goal, depending on the semester they took the course.
The findings are mixed. Students approached but did not quite meet the goal of 75% correct on the multiple choice questions and the post-test questions. However, students’ improvement from pre-test to post-test was double the department’s goal of 15%.
PSYC 3030 Principles and Methods of Psychological Investigation: Outcomes B1-B4, C1, D, E1-E2 & G
The goal was to demonstrate a significant improvement in formal written communication in a discipline-specific style over the semester. This goal was met for both the Fall 2004 and Spring 2005 terms.
PSYC 3110 Psychology of Interpersonal
Behavior: Outcomes D2, F1-F3 & H1
The results of this evaluation of the effectiveness of PSYC 3110 are mixed. On the one hand, the pre- and post-test self-report data suggest that by the end of the course, students judge that their communication skills have significantly improved. On the other hand, the behavioral data imply that at the end of the course, 20-30% of students do not demonstrate criterion level interpersonal skills. Approximately 80% reached criterion for talking skills, and fewer (approximately 70%) for listening skills. Although the majority of the students performed at criterion in these domains, it was anticipated that 90% or more of the students would have done so. Of note is that one of the instructors whose section was included in the behavioral evaluation reported that he or she devoted little class time to the communication exercises. This was at variance with the curriculum plan for PSYC 3110. Predictably, his or her section performed poorly. When data from this section were dropped from the analyses, the percentages of students obtaining criterion for listening and talking skills increased to 78.60% and 88.40%, respectively.
The objectives are that
all students will show criterion performance or expected progress in all
learning outcome domains. The data
indicate that the objectives were generally achieved in all domains. With one
or two exceptions, all students passed all courses, the vast majority received
satisfactory ratings for academic performance in course relevant to all 10
learning outcomes, over 75% completed theses, dissertations, and general exams
on time, and all who completed the milestones received passing evaluations. The
outcomes are particularly favorable in theory and content (Domain 1), research
methods (Domain 2), critical thinking skills (Domain 5), information and
technology literacy (Domain 7), values in psychology (Domain 8), and sociocultural and international awareness (Domain 9). The
outcomes for application (Domain 3) indicate that 15% of the clinical students
obtained at least one rating below “very good” for clinical skills. However,
since these evaluations are conducted as part of ongoing monitoring during the
course of skill development rather that as final evaluations of ultimate
success, it should be expected that fewer than 100% of the students would reach
criterion. Similarly, regarding personal
development (Domain 6), the fact that 18% of the clinical students obtained at
least one rating below criterion may be acceptable for skills that are expected
to show continuous development during the course of graduate training. Outcomes
for communication and collaboration skills (Domain 4), and career planning and
development (Domain 10), are difficult to evaluate because of ambiguity in
interpreting the data regarding professional presentations and publications. It
is likely that the data are incomplete because faculty may have neglected to
report these data in the yearly evaluation letters. Also, since a target level
of involvement was not specified in our goals, it is unclear whether
involvement in presentations by two-thirds of the graduate students, and
publication by nearly half during one year is a satisfactory outcome.
III.
Changes to Procedures or Curriculum Based on Assessment
A. Undergraduate
Although a majority of PSYC 1101 students are able to answer 70% or more of the mastery test questions correctly at the end of the term, 40-49% of our students are still not realizing this goal. For the upcoming year, mastery test questions that 50% or fewer got right will be reviewed for possible revision or replacement. Particular attention will be given to insuring the quality and clarity of questions involving application of concepts in novel situations; these seem to present the greatest challenge for students in this course. Instructors for PSYC 1101 in 2005-06 will be provided with these results. The course supervisor will work with a team of instructors to develop and disseminate strategies for increasing students’ ability to apply and generalize key concepts.
Based on the findings, slight changes will be made to the curriculum and procedures. For the few multiple choice questions on which fewer than 50% of students received a correct score, reworded questions assessing those constructs will be written, and instructors will be alerted of students’ difficulty with those constructs. Additionally, the post-test questions will be integrated into the in-class and final exams rather than being administered separately on the last day of class. Anecdotal evidence suggests that many students did not put much effort into the post-test because it did not contribute to their course grade. Finally, instructors will be asked to include students’ course grades in their data files so that results can be stratified based on whether students passed (a grade of at least C) the course.
PSYC 3030 Principles and Methods of Psychological Investigation: Outcomes B1-B4, C1, D, E1-E2 & G
The structure and result of the current assessment is satisfactory. However, in light of departmental intentions to increase the size of each section to 50-75 students, some modifications to the assessment strategy are likely to be necessary. During 2005-2006, an effort will be made to increase the number of students whose work is assessed and to distribute the responsibility for this assessment across multiple individuals. In order to increase the number of students’ whose work is assessed by at least two-fold, fewer elements of each assignment will be assessed and the rating scale will be compressed from 1-5 to 0-2.
PSYC 3110 Psychology of Interpersonal
Behavior: Outcomes D2, F1-F3 & H1
Although the inclusion of both self-report and behavioral ratings of interpersonal competence are strengths of this assessment, interpretations of the findings are only suggestive in the absence of baseline behavioral ratings to allow for assessment of change over the course of the semester, and/or control of such potentially confounding variables as time and attention. Further evaluation of outcomes for PSYC 3110 should incorporate additional measures of these types of data.
The curriculum should be reviewed to identify strategies to place greater emphasis on the acquisition of interpersonal skills, particularly in the listening domain.
1) Encourage research supervisors to use evaluation instrument to conduct mid-semester evaluations as a way of proving feedback to students regarding strengths and weaknesses and setting individualized goals.
3) Ask a specific question about whether the practicum experience assisted the student in confirming or refining career goals.
4) Improve compliance with evaluation by faculty to 100%
5) Improve compliance with self-assessment by students to 100%
1) Encourage field supervisors to use evaluation instrument to conduct mid-semester evaluations as a way of providing feedback to students regarding strengths and weaknesses and setting individualized goals.
3) Ask a specific question about whether the practicum experience assisted the student in confirming or refining career goals.
4) Improve compliance with evaluation by field supervisors to 100%
4) Improve compliance with self-assessment by students to 100%
B. Graduate
One change to the assessment procedure will be the development and use of a set of performance rating scales to be completed by faculty committee members to rate students’ attainment of learning outcomes with regard to theses, dissertations, and general exams. The rating scales will ask the faculty to evaluate the student in each of the relevant domains of learning outcomes. In order to collect more accurate data on the number of professional presentations and publications by graduate students, we will require students to turn in a report of their professional accomplishments in a standardized electronic format at the end of the next academic year. Other changes to the assessment procedures may be developed after the review of this first year of data by the graduate program committee during the fall 2005 semester. Changes in curriculum are instigated by the faculty in an ongoing fashion based on a variety of considerations. Since this is the first year of for compiling data on learning outcomes, there have been no curriculum changes based specifically on this assessment. During the 2005-06 academic year the data will be shared with the faculty in departmental faculty meetings, program area faculty meetings, and graduate program committee meetings, for possible use in making future curriculum changes.
IV.
Changes to Department’s Assessment Goals
A. Undergraduate
No change in goals is anticipated.
Criteria for learning will be revised to focus more on change over the semester than on absolute percentage correct.
PSYC 3030 Principles and Methods of Psychological Investigation: Outcomes B1-B4, C1, D, E1-E2 & G
No
change in goals is anticipated.
PSYC 3110 Psychology of Interpersonal
Behavior: Outcomes D2, F1-F3 & H
In the future, the evaluation design should assess PSYC 3110 students’ acquisition of not only applied skills, but also key concepts and empirical findings in interpersonal psychology.
B. Graduate
Because these data from the first assessment cycle have not yet been processed and discussed by the faculty, there are not yet any changes in the department’s assessment goals. Appropriate changes may be identified in the coming academic through review of these first-year data. One topic for discussion will be departmental expectations for presentations and publications for students in our graduate program.
Attachment A
Graduate Learning
Outcomes
Goal 1 - Theory and
Content: Develop expertise with major
concepts, theoretical perspectives, empirical findings, and historical trends
in the field of Psychology, the program area, and the research specialty area.
Representative Skills 1:
Assessment 1:
Field of Psychology
1. Performance in Psych 8500, History of Psychology.
2. Ratings and written evaluations by departmental faculty during yearly review.
Program Area:
1. Performance in required & elective courses in program area.
2. General Exam – Written; scored by committee of faculty.
3. General Exam – Oral Defense; conducted by committee of faculty.
4. Ratings and written evaluations by program faculty during yearly review.
5. Ratings and written evaluations by practicum supervisors completed each semester.
Research Special Area:
1. Masters Thesis - Proposal; evaluated by faculty committee and defended orally in committee meeting.
2. Masters Thesis – Final Document; evaluated by faculty committee and defended orally in committee meeting.
3. Dissertation - Proposal; evaluated by faculty committee and defended orally in committee meeting.
4. Dissertation – Final Document; evaluated by faculty committee and defended orally in committee meeting.
5. Ratings and written evaluations by departmental faculty during yearly review.
Goal 2 – Research Methods:
Understand and apply research methods including research design, data analysis,
and interpretation.
Representative Skills 2:
1. Be competent in a variety of research techniques relevant to area of specialization (e.g., observation, action-research, survey methods).
2. Strives to produce optimal research and measurement strategies for reducing challenges to validity of conclusions.
3. Interprets behavior at appropriate level of complexity that reflects contextual factors.
4. Establishes a research focus that identifies and builds on primary interests in behavior.
5. Uses statistical reasoning routinely for evaluating research, develops fair and appropriate applications of statistics and other analytical methods that are consistent with data.
6. Implements and demands high standards for adherence to ethical standards and to research methods to minimize complications of bias and confounds.
Assessment 2:
1. Performance in Psych 8410-8420, Psychological Research Statistics I & II.
2. Performance in methods courses required by program area.
3. Ratings and written evaluations by departmental faculty during yearly review.
4. General Exam – Written; scored by committee of faculty.
5. General Exam – Oral Defense; conducted by committee of faculty.
6. Masters Thesis - Proposal; evaluated by faculty committee and defended orally in committee meeting.
7. Masters Thesis – Final Document; evaluated by faculty committee and defended orally in committee meeting.
8. Dissertation - Proposal; evaluated by faculty committee and defended orally in committee meeting.
9. Dissertation – Final Document; evaluated by faculty committee and defended orally in committee meeting.
Goal 3 – Application:
Apply psychological principles in professional activities.
Representative Skills 3:
1. Develops unique applications of theory and research methods.
2. Evaluates relevant content from broader range of available resources; reflects how context influences actions, shows refined and flexible use of published research.
3. Uses psychological concepts fluently to explain and predict behavior.
4. Creates compelling arguments with attention to subtle meaning of content; anticipates and defends against criticism, adapts arguments for wide range of audiences.
5. Engages in ethical professional behavior.
Assessment 3:
1. Ratings and written evaluations by departmental faculty during yearly review.
2. Course evaluations and teaching portfolios
3. Faculty evaluations of teaching.
4. Supervisor evaluations of practicum performance.
Goal 4 –
Communication and Collaboration Skills: Communicates and works in groups
effectively.
Representative Skills 4:
1. Creates compelling arguments with attention to subtle meaning of content; anticipates and defends against criticism; adapts arguments for wide range of audiences.
2. Uses APA format expertly with minimal errors; demonstrates sophisticated conventional oral and written expression.
3. Independently completes sophisticated group projects that require collaboration over time.
4. Systematically plans project completion strategy, including back-up plans to overcome likely obstacles.
5. Convenes colleagues to improve quality of projects and programs.
6. Conscientiously seeks expression of broad opinions and productive conflict resolution.
7. Actively promotes and enjoys group creativity.
Assessment 4:
1. Yearly student report of conference presentations and publications.
2. Ratings and written evaluations by departmental faculty during yearly review.
3. General Exam – Written; scored by committee of faculty.
4. General Exam – Oral Defense; conducted by committee of faculty.
5. Masters Thesis - Proposal; evaluated by faculty committee and defended orally in committee meeting.
6. Masters Thesis – Final Document; evaluated by faculty committee and defended orally in committee meeting.
7. Dissertation - Proposal; evaluated by faculty committee and defended orally in committee meeting.
8. Dissertation – Final Document; evaluated by faculty committee and defended orally in committee meeting.
9. Course evaluations and teaching portfolios.
10. Faculty evaluations of teaching
11. Supervisor evaluations of practicum performance.
Goal 5 – Critical
Thinking Skills: Respect and use critical and creative thinking, skeptical
inquiry, and the scientific approach.
Representative Skills 5:
1. Practices scientific method and accepts its limitations.
2. Acknowledges and guards against bias or subjective influences.
3. Seeks the most precise explanation.
4. Uses skepticism consistently as an evaluative tool.
5. Enjoys complexity during search for clarifying behavioral explanations.
Assessment 5:
1. Ratings and written evaluations by departmental faculty during yearly review.
2. General Exam – Written; scored by committee of faculty.
3. General Exam – Oral Defense; conducted by committee of faculty.
4. Masters Thesis - Proposal; evaluated by faculty committee and defended orally in committee meeting.
5. Masters Thesis – Final Document; evaluated by faculty committee and defended orally in committee meeting.
6. Dissertation - Proposal; evaluated by faculty committee and defended orally in committee meeting.
7. Dissertation – Final Document; evaluated by faculty committee and defended orally in committee meeting.
8. Course evaluations and teaching portfolios
9. Faculty evaluations of teaching.
10. Supervisor evaluations of practicum performance.
Goal 6 – Personal
Development: Shows insight into one’s own and other’s behavior and mental processes
and applies effective strategies for self-management and self-improvement.
Representative Skills 6:
1. Uses self assessments to establish goals.
2. Formulates back-up plans to anticipate and overcome obstacles.
Assessment 6:
1. Ratings and written evaluations by departmental faculty during yearly review.
2. Course evaluations and teaching portfolios
3. Faculty evaluations of teaching.
4. Supervisor evaluations of practicum performance.
Goal 7 – Information
and Technology Literacy: Acquire skills in accessing and disseminating
information with the use of computer technology.
Assessment 7:
1. Ratings and written evaluations by departmental faculty during yearly review.
2. Masters Thesis - Proposal; evaluated by faculty committee and defended orally in committee meeting.
3. Masters Thesis – Final Document; evaluated by faculty committee and defended orally in committee meeting.
4. Dissertation - Proposal; evaluated by faculty committee and defended orally in committee meeting.
5. Dissertation – Final Document; evaluated by faculty committee and defended orally in committee meeting.
Goal 8 – Values in
Psychology: Weigh evidence, tolerate ambiguity, act ethically, and reflect
other values underpinning psychology.
Representative Skills 8:
1. Monitors ethical practices in area of specialization.
2. Routinely evaluates research, practice, and teaching from an ethical standpoint as an ongoing professional responsibility.
3. Advocates for the best ethical practices to protect the public and improve the discipline.
Assessment 8:
1. Performance in Psyc 8490, Scientific and Professional Ethics in Psychology.
2. Ratings and written evaluations by departmental faculty during yearly review.
3. General Exam – Written; scored by committee of faculty.
4. General Exam – Oral Defense; conducted by committee of faculty.
5. Masters Thesis - Proposal; evaluated by faculty committee and defended orally in committee meeting.
6. Masters Thesis – Final Document; evaluated by faculty committee and defended orally in committee meeting.
7. Dissertation - Proposal; evaluated by faculty committee and defended orally in committee meeting.
8. Dissertation – Final Document; evaluated by faculty committee and defended orally in committee meeting.
9. Course evaluations and teaching portfolios
10. Faculty evaluations of teaching.
11. Supervisor evaluations of practicum performance.
Goal 9 – Sociocultural and International Awareness
Assessment 9:
1. Performance in Psyc 8050 or 8060, Issues of Human Diversity in Psychology.
2. Ratings and written evaluations by departmental faculty during yearly review, especially relevant to experiences in applied research, intervention, and consultation with special populations.
3. Performance in required and elective courses with emphasis on socio-demographic, cross-cultural, and contextual influences on human behavior.
4. General Exam – Written; scored by committee of faculty.
5. General Exam – Oral Defense; conducted by committee of faculty.
6. Supervisor evaluations of practicum performance in settings that provide experience with diverse American or global population groups.
Goal 10 – Career
Planning and Development: Emerge from graduate school with credentials and
plans for career path.
Representative Skills 10:
1. Attends and participates in conferences relevant to area of specialization.
2. Collaborates on publications relevant to area of specialization.
3. Gain teaching experience and credentials through GLA and GTA responsibilities.
4. Gain research experience and credential through GRA activities.
Assessment 10:
1. Presentations at regional, national, international conferences.
2. Number of publications in peer reviewed journals, invited chapters.
3. Course evaluations and teaching portfolios.
4. Faculty evaluations of teaching
5. Ratings and written evaluations by departmental faculty during yearly review.