Annual Student Learning Outcomes Assessment Report
Unit Name: Applied Linguistics & ESL, MA program in Applied Linguistics
Prepared by Sara Cushing Weigle Date July 21, 2005
Academic Year: 2004-2005
|
III. Changes to Procedures or Curriculum Based on Assessment |
|
|
II. Achievement
of Departmental Objectives for Students B. Graduate |
The following outcomes for the MA program, approved by the AL/ESL faculty in 2004, were assessed:
B. Elements of assessment
Our long-term assessment plan is to collect data regularly from the following seven sources:
In this first annual cycle of assessment, we were able to collect data from five of these sources (syllabi, portfolio, teaching performance videotapes, survey of current student perceptions, and survey of alumni perceptions); these are noted in bold face. A progress report on data that we were unable to collect in AY 2004/5 is found in Appendix A.
C. Data collected
The following data were collected from the data sources
outlined above.
· Completion of 90 hours of classroom-based experience, with reflective essays documenting this experience every semester
· Completion of at least 2 professional development activities every semester, with reflective essays documenting this experience every semester
· Number of publications and/or conference presentations completed by each student.
Complete data on 24 students who graduated between Summer 2004 and Spring 2005 were available.
D. Data Analysis
This section of the report presents the results of data analysis by data source.
1. Course syllabi.
The goal set for this year was that syllabi for all required courses would contain explicit statements of learning outcomes and how those outcomes would be measured by course requirements and activities, and that there would be 100% coverage of outcomes A-F across the required courses.
In the 2004/5 academic 50% of syllabi for required courses (5/10) contained appropriate learning outcomes.
à
GOAL NOT MET.
Across those syllabi that contained learning outcomes, however, there was 100% coverage of outcomes A-F.
à
GOAL MET.
2. Student portfolios
a. The goal for AY 2004/5 was that 100% of graduating students would complete their CBE and PD requirements. Advisors reported that for all students for whom they had data completed these requirements.
à
GOAL MET.
b. The goal for AY 2004/5 was that 70% of graduating students would have at least one conference presentation or publication before graduation. Resumes were available for 16 students who graduated between Summer 2004 and Spring 2005. Of these student, 9 presented at conferences and/or published papers, for a total of 56%.
à
GOAL NOT MET.
3: Teaching performance and videotapes
Students in the practicum course were evaluated on teaching effectiveness and oral communication. The criterion for success in this area was that 90% of student lessons would meet the minimum standards agreed to by department faculty.
Eighteen students attended the practicum course during fall and spring semesters. Of these, 100% met the minimum standards.
à
GOAL EXCEEDED.
4 and 5: Survey of current student and alumni
perceptions
Because these two surveys are parallel, the results of the survey are discussed together. A survey was developed and results tallied in the department. In our original assessment plan, the stated criterion for success was as follows: “90% of students who respond to survey will report that they received ‘excellent’ or ‘good’ preparation in Outcomes A-G.” However, on the advice of the Office of Institutional Research, the survey options were modified so that students reported their level of confidence in outcomes A-F as a result of the program, rather than the degree of their preparation. The main results of the surveys are found in Table 1.
As the table shows, the 90% criterion of success was only reached in 2 out of 8 outcomes for alumni, and none for current students. However, it should be noted that approximately half of the current students responding to the survey were in their first or second semester of the program, so it is perhaps unsurprising that the current students reported lower levels of confidence in many areas of the survey.
à
GOAL PARTIALLY MET.
Table 1. Percentages of students rating their confidence 4 or 5 on a scale of 1-5
|
|
Current students |
Alumni |
|
73% |
94% |
|
73% |
83% |
|
77% |
83% |
|
16% |
89% |
|
80% |
100% |
|
48% |
77% |
|
64% |
75% |
Bold face = at least 90%
II. Achievement of Departmental Objectives for Students
Table 2 summarizes the results of data collection and analysis for AY 2004/5. As the table shows, all of the objectives were reached on some criteria but not on others. Two outcomes were particularly strong; these were (A) Applies the basic principles of ESL/EFL learning and teaching methodology and (E) Communicates effectively in both written and oral language in English. The two outcomes that were the weakest were (F) Uses technology effectively in research and teaching and (G) Conducts and participates in professional development activities.
Table 2. Summary of outcomes
assessment
|
Outcomes |
Syllabi required courses |
CBE |
PD |
Resume |
Teaching Performance videotapes |
Survey of current students |
Survey of alumni perceptions |
|
A. Applies the principles of
ESL/EFL language teaching methodology and learning |
YES |
YES |
|
|
YES |
NO |
YES |
|
B.Demonstrates knowledge of the linguistic systems of English
phonology, grammar, and discourse |
YES |
|
|
|
|
NO |
PARTIAL |
|
C. Uses cultural knowledge in second language learning and teaching |
YES |
|
|
|
|
NO |
PARTIAL |
|
D. Analyzes and critiques theory and practice of L2 teaching and
learning |
YES |
YES |
|
|
|
NO |
PARTIAL |
|
E.Communicates effectively in both written and oral language in
English |
YES |
|
|
|
YES |
PARTIAL |
YES |
|
F. Uses technology in research and teaching |
YES |
|
|
|
|
NO |
NO |
|
G. Conducts and participates in professional development activities |
|
|
YES |
PARTIAL |
|
NO |
NO |
KEY: YES = met objectives as originally stated in departmental plan
PARTIAL = came close to meeting objectives
NO = did not meet objectives
III. Changes
to Procedures or Curriculum Based on Assessment
Based on the results of the assessment, one simple change in procedures has already been made: the Chair has met with all faculty to reiterate the importance of including learning outcomes on their course syllabi. In addition, a number of changes are being considered for the program. A subcommittee of faculty will be meeting to discuss the survey results in more detail and recommend changes to the program based on these results, concentrating particularly on outcomes F and G. One possible change will be to provide training to faculty on incorporating technology into their courses; another possibility is to offer a course on technology in language teaching. A third possibility is to revisit the original assessment goals and determine whether our criteria for success were too stringent; see below.
IV. Changes in Department’s Assessment Goals
A number of changes in the department’s goals are anticipated for AY 2005/6. As discussed above, the criteria for success in the survey results will need to be reworded to match the actual wording of the survey. We will also be taking a closer look at the results from this year’s assessment to determine whether our goals are realistic: it seems clear, for example, that we cannot expect current students, especially those at the beginning of their programs, to express confidence in their abilities to do many of the things we are asking them to do when they have not yet completed their required coursework. It may also be overly ambitious to expect 70% of graduating students to present at conferences or publish their work. Finally, we should have more information on the achievement of our departmental goals when we have put in place the last pieces of our assessment procedures in the coming year (see Appendix A).
Appendix A. Planned assessment procedures
There are three assessment procedures that we have not yet implemented. In each case we have provided the original wording from our assessment plan and an indication of the current status of each procedure.
Data source 2: Portfolio (MA paper, CBE, PD essays,
resume)
Data source 2A: students’ master’s papers.
Evaluation procedure: In addition to evaluating the content of the paper (as is currently the practice), the two faculty members who read each students’ master’s paper will explicitly evaluate the paper for critical analysis and the relationship of theory to practice (Outcome D) and clarity of expression, organization, grammar, use of language, and appropriate formatting and documentation. (Outcome E). A rubric has been developed for this purpose.
Criterion: 90% of student papers will be rated “excellent” or “good” in their application of critical thinking (Outcome D) and in their written communication (Outcome E).
Status: A rubric was developed and pilot tested during 2004/5. Faculty will begin using this rubric with all student MA papers during 2005/6.
Data Source 3: Oral presentation of student’s MA paper
Evaluation procedure: During their final semester, students will make a formal oral presentation of their master’s paper. Two faculty members will rate the presentation for clarity, organization, effective use of visual aids, and overall presentation (e.g., volume, body language). (Outcome E). A rubric has been developed for this purpose.
Criterion: 90% of students will meet the minimum standards.
Status: This is a new requirement for students who began the program in Fall 2004. Students who started the program in Fall 2004 will be graduating in Fall 2005 at the earliest. Thus we will be able to collect this data beginning in the 2005/6 academic year.
Data Source 5: Survey of
employer perceptions
Evaluation procedure: Every three years, a representative sample of employers of recent graduates will be surveyed by mail. Survey results will be tallied by a staff or faculty member.
Criterion: 90% of employers who responded to survey will rate graduates of our program “excellent” or “good” in the understanding and application of the basic principles of ESL/EFL language teaching methodology and learning (Outcome A); in their knowledge of the linguistic systems of English (Outcome B); their understanding of intercultural issues (Outcome C); their oral and written communication (Outcome E); their use of technology (Outcome F); and their commitment to professional growth (Outcome G).
Status: A survey has been developed for this purpose; however, we have not yet been able to compile a list of recent graduates and their current employers to sample for the survey. We hope to collect these data in the 2005/6 school year.