Summary Report on GSU Summer 04 Assessment of

Oral Communication Competency In the University Core Curriculum

 

 

09/08/04 Draft

 

I.  Overview of Oral Communication Assessment Strategies Pertinent to the GSU Core Curriculum

 

Georgia State University’s General Education Goals, approved by the Undergraduate Council (1/30/04) and the University Senate (2/13/04) commit the university to developing oral communication competencies in its undergraduates.  Goal I.2 says students should “communicate effectively using appropriate oral or signed conventions and formats.”

 

Although oral communication proficiency is often built into course requirements by individual faculty and the university has committed to assure oral communication opportunities in each of its undergraduate majors, the core curriculum makes no explicit place for the targeted enhancement of such skills apart from SPCH 1000 (Fundamentals of Human Communication), an elective option in Area B.  Along with PHIL 2410 and other Perspectives options, all of which are two-credit hour classes, the area is organized to foster communication and critical thinking competencies.  Because a significant percentage of GSU undergraduates take SPCH 1000, it provides a natural starting point from which to begin assessment data collection.

 

With respect to the quality of oral communication instruction provided by Georgia State University, two points bear repeating, both made in the earlier spring report on this issue:

 

FIRST:  There is indirect evidence that GSU students are achieving oral communication competencies over the length of their undergraduate programs of study at a pace comparable to peer urban and doctoral institutions.  My spring report cited NSSE data, which reports student satisfaction with learning outcomes on many measures including some directly pertinent to oral communication skill acquisition and development.   As I reported there:  "In the 03 dataset, both GSU first years and seniors responded that they “made a class presentation” at a rate comparable with other NSSE Research I (e.g., GSU senior = 2.60; Research I senior = 2.61) and NSSE Urban peer institutions (GSU senior = 2.60; Urban = 2.75).  On the measure, “GSU contributed to your speaking clearly and effectively,” overall reported scores rise from first to senior year (generally speaking an exception to the overall data, where numbers from frosh to senior often decline in line with the large infusion of transfer students who join GSU mid-degree program), 2.65 to 2.75.  At the first year level, the GSU mean exceeds the NSSE Research I mean (2.50) and is on a par with the NSSE Urban mean (2.69).  The senior GSU mean (2.75) lags Research I and Urban means (2.83 and 2.85) but not by much.  One can read such data as showing that while much room for progress remains, the GSU experience is roughly comparable in the oral communication area to that offered by peer institutions, at least with respect to student self-reports."

 

Given such data, our efforts in Summer 04 were mainly designed to begin the collection of benchmark data of sufficient specificity to be useful in reviewing our core curricular instruction.

 

Second:   Our data collection efforts were shaped by a widely shared agreement reflected in the speech communication literature that oral communication competency can be very widely defined.   As I put it in our spring interim report, “'Oral communication competency,' as typically understood, involves the inculcation of a wide range of interconnecting skills.  Skilled communicators must be mindful of the ethical and evaluative standards of their culture, must adapt their messages in a context of multicultural difference, and are increasingly required to possess technological competencies.  Because these skills are often the central focus of disciplinary training in the major, most work relating communication competency to general education requirements focuses on the development of speaking and listening skills."  The specific competencies identified by NCA researchers include all of the following (each adapted from Quianthy and available online at www.natcom.org).

 

 Expected Student Outcomes for Speaking and Listening

 

 Because this full a range of competencies is too diverse to have been adequately assessed in a summer pilot project, we focused on measures specific to the experience of preparing and delivering a public speech (a project already built into the assignment structure for SPCH 1000).  These measures include determinations of student competency in such performance areas as:  successfully gaining the audience’s attention, justifying a topic, establishing credibility, logical presentation style, successful use of supporting materials, and delivery measures such as those relating to eye contact, natural gesturing, and strong vocal projection. 

 

II.  Summer 04 Assessment Efforts | Summary of Findings

 

We collected data by use of three instruments, all of which are in common use.  Two are focused surveys which ask students to report their level of performance apprehension.  Both measures have been in wide use for decades and by now have accumulated considerable evidence of reliability.  Because these measures take little class time to administer, they can establish some benchmark pre- and post-test indication of apprehension and its reduction over the course of a term.   The third instrument is a rubric also in common national use, The Competent Speaker, which requires the instructor to provide a detailed assessment of a single public speaking event.  The combination of PRCA-24, WTC, and CS instruments is a common one for national oral competency efforts.[1]

 

What distinguishes these measures?  The PRCA-24 is designed to measure behavioral trait apprehension, which is to say, a person's level of nervousness or apprehension when confronted with communication situations.  PRCA-24 is thus useful in assessing a student's level of stage fright, actual or anticipated anxiety, or apprehensiveness.   The Willingness to Communicate Scale aims to measure a person's shyness or reticence.

 

PRCA-24 | Measuring Trait Apprehensiveness

 

The Personal Report of Communication Apprehension has been in widespread use for more than a quarter-century, and was developed and refined by James McCroskey, a professor of communication and educational psychology at West Virginia University.  The suffix -24 refers to the fact the instrument is a 24-item survey.  The PRCA-24 seeks to measure perceived apprehension in four communication contexts:  public speaking, speaking in small groups, speaking up in meetings, and speaking in face-to-face dyads.  Six items correspond to each context.  While these four settings do not exhaust the possibilities (communication also occurs in intercultural, supervisor/subordinate, interviewing, and other situations), they are typical of communication competencies required in public and professional life.

 

Benchmarking data for the PRCA-24 has been collected from several large-size sample populations.  A national study of college students enrolled in pharmacy programs (more than 10,000) produced a mean score of 65.2.  Data collected from West Virginia University students (the number exceeded 12,000) produced a mean score in the same neighborhood – 65.6 (McCroskey, Fayer, and Richmond, 1985).   Given the scoring matrix for the subcategories, one would expect scores in each area to hover at roughly 18.0, and this has also been confirmed by the large-sample research.

 

Valid surveys were received from 61 GSU undergraduates enrolled in summer sections of SPCH 1000 in an administered pre-test, and from 85 students in an end of term post-test.[2]  The overall numbers were consistent both with national benchmarks, and with the experiences reported in similar first-year communication survey courses elsewhere.  Pre-test means yielded an overall mean score of 63.3279, which is slightly higher than the benchmark averages; post-test averages (recalling the post-test was administered after all students had prepared for a major public address) dropped to 61.3412, below the national norm.  These mean scores confirm both the short-term pedagogical value of even limited public speaking experiences in alleviating self-reported apprehension, and also illustrate that even providing a rather narrowly construed opportunity for communication (the formal speech) can have spillover benefits in reducing apprehension in other contexts.  Not surprising, the sub-score for reported apprehension in public settings dropped the most – this was, after all, a main objective of the public speaking activity – from a pre-test mean of 19.5410 to a post-test mean of 18.7529 (the sub-score range for all categories is 6-30).  But reported reticence in all categories dropped as well – the group score dropped from 14.4426 to 14.0824 and dyad reticence dropped from 13.7213 (pre-test) to 13.6471 (post-test), for instance.  As one would expect, the dyad score showed the least improvement, since as presently constructed the SPCH 1000 course includes no activities directly relating to interpersonal communicative competency.

 

The GSU summer data also confirmed another widely reported finding, which relates to the broad benefits of communication for all student groups.  That is, in reviewing the distribution of self-report scores pre- and post-test, the evidence is fairly compelling that the benefits of immersion in a communication-intensive environment was just as beneficial for students reporting extreme early reticence as for students more initially comfortable with communication events.  Although the sample sizes at the extremes are small, the number of students reporting exceptional reticence in group activities dropped roughly by half (from 5 to 2).  And the distribution of self-report scores relating to public speaking, which in the pre-test showed a higher than expected number of "moderate anxiety" scores, had leveled by the time of the post-test to a distribution much more closely resembling normal.  Exceptionally confident communicators, as one might expect, showed little movement to higher confidence numbers (when viewed in the aggregate), but students in the wide middle showed strong overall movement higher in self-reported comfort.

 

Of course a single-shot exposure such as that enabled by SPCH 1000 cannot be over-credited.  The amount of reported progress was modest even in the best cases, and to some extent scores might be expected to drift higher simple as students become more comfortable in their particular classroom and with the instructor.  Obviously such effects could be minimized by assessment designs which seek to measure competency at similar points in the sample term.

 

Willingness to Communicate Scale | Measuring Reticence

 

Also developed by McCroskey, the Willingness to Communicate Scale is a 20-item, probability estimate scale.  Eight of the items are filler material and the remaining 12 produce the overall score.  As McCroskey has described it, WTC "yields a total score, three subscores based on types of receivers (strangers, acquaintances, friends), and four subscores based on types of communication contexts (public, meeting, group, dyad)."  The purpose of the scale was to produce direct measures of a person's predisposition to approach or avoid communicative contact.

 

A 1641-student sample assembled at West Virginia University (McCroskey, 1992) provides some basic benchmarking data which might be useful for the sake of comparison.  That data showed, perhaps not surprisingly, that respondents are most uncomfortable communicating in a public speaking context and with strangers and most willing to communicate in small groups of friends, and in interpersonal dyads.  For example, the mean score for public settings (on a self-report scale of 1-100, where 100 reflects an exceptional willingness to communicate) is 65.2 (SD = 15.1, reliability = .92), whereas the mean score for conversation with a friend was 84.7 (SD = 14, reliability = .76).  Obviously these numbers are shaped to some extent by cultural differences, and while the data show that participants from Sweden, Australia, Micronesia, Finland, and Estonia roughly rank communication apprehension experiences the same, the mean scores can vary considerably.  The cross-cultural data showed, for example, that Australians showed considerably higher reticence than Americans.

 

For GSU students (58 valid pre-tests, 83 valid post-tests), the willingness to communicate numbers increased across the board.  The overall number increased from a mean of 201.3103 to 208.0602 (the SD decreased from 57.29547 to 50.96844).  The public communication sub-score rose from a mean of 59.5287 to 62.9157 (these numbers fall on a 1-100 scale); meeting scores rose from 63.3908 to 68.2088; stranger scores went from 41.3017 to 45.4518.  Interesting, the self-reported willingness to communicate in dyadic encounters actually dropped, from a pre-test mean of 88.9713 to 86.9197.

 

A word about reliability, which varied in this administration of the survey.  In this context a reliability alpha exceeding .80 would be preferable; for some subscores reliability totals fell short.  In the pre-test, for instance, reliability totals were within acceptable for public (.7706), meeting (.7642), stranger (.9163), and acquaintance (.8096) categories, but much lower for the small group (.4669) and dyad (.5879) categories.  Reliability numbers were higher across the board for the post-test and closer to nationally reported means, though in some cases still lower than one would prefer.  There are several potential explanations for this.  Perhaps students failed to take the pre-test seriously.  And the instrument itself may subvert reliability given small samples, since each sub-score aggregates only three scores.  Finally, the number of students taking the pre-tests were lower, and this may have had an effect as well.

 

Consistent with the PRCA-24 results, these instruments also revealed slight but nearly across-the-board increases in students' willingness to communicate, though the same caveats apply (small increase, possible distortion from the improved comfort level of having participated in the same class and with the same survey cohort).

 

The Competent Speaker | Measuring Performance

 

The major measure used to assess performative competency is an adapted version of the Competent Speaker, a university-level assessment instrument designed by the NCA, which has been adapted to also serve as the speech evaluation form in SPCH 1000.  The Competent Speaker instrument was developed by S. P. Morreale, M. R. Moore, et al., and was published in 1993.  The instrument was validity tested in several ways.  For instance, a panel of eleven speech communication educators was involved in the final editing, and negative correlations have been reported between this instrument and the six public speaking-related items on the Personal Report of Communication Apprehension (which is to say, more competent public speakers also have lower apprehension, a finding consistent with communication research and common sense).  A reviewer for NCA who independently analyzed available oral competency instruments rated this one highly, although work remains to develop items that better assess “thinking, reasoning, and evidence as substantial components of speech preparation and presentation” (Ellen Hay, Large Scale Assessment of Oral Communication:  K-12 and Higher Education, 2E, NCA, p. 52).  Hay calls attention to the usefulness of this instrument for “institutions that offer multiple sections of the public speaking course because it allows for standardized evaluation of the speeches.”

 

The Competent Speaker is standardized and has been cited for its high levels of inter-coder reliability.  Among speech communication professionals use of the instrument generated an Ebel's coefficient of .92, for graduate teaching assistants a Cronbach's alpha of .76 has been reported, and for community college speech instructors a Cronbach alpha was reported at .84.  Georgia State University course instructors were trained in the use of the instrument in a standard orientation session, and training materials accompanying the instrument were provided to all course instructors.  Our goal was to combine the standard national instrument for oral communication with the course specific speech evaluation form, which was accomplished fairly easily since the criteria for effective speech making tend to dovetail with the Competent Speaker measures.

 

The results obtained by the limited summer use of the revised instrument are reported below.[3]  Relatively high rates of oral communication competency are reported across the board, which is not terribly surprising given the intensive effort which goes into these single speeches.

 

 

 

As one can readily see, the scores range relatively high.  On a three-point scale (where 3 = outstanding, 2 = satisfactory, and 1= unsatisfactory), these scores would translate into roughly B+/A- grades.  Although that may seem high, such results are consistent with the course history, where speech grades tend to pull up the overall grade mean and test scores tend to pull it down.  And of course the single major speech is an activity into which considerable instructional time is allocated. 

 

One can also observe from the scores reported above that they range in perhaps predictable ways.  The scores most susceptible to short-term improvement (topic selection, use of a thesis statement, vocal variety, and gesturing behavior) tend to produce the highest grade assessments, while behaviors likely to be more resistant to short-term instruction (the ability to produce sophisticated research backing for claims made, vocabulary diversity, and grammatical competency) tend to remain relatively lower.

 

 

III.  Recommendations for Expanded Full-Scale Assessment Activity in the 04-05 Academic Year

 

Use of the three assessment instruments should, for the time being, be continued, and on an expanded scale.  Over time, and as more extensive data are collected, it may prove unnecessary to continue collecting information twice relating to communication apprehension.  The marginal benefit of collecting pre- and post-data on apprehension will likely recede as well.  In the meantime it might be advisable to link survey responses student-by-student, so that apprehension and speaking performance can be specifically compared and tracked.  Because the apprehension instruments can be so quickly accomplished in the classroom, and because the reduction of speech anxiety is already and often a normal topic during early class sessions, it appeared that use of the two instruments was not burdensome on instructional staff.  The two instruments (Willingness to Communicate and the PRCA-24) do provide distinguishably useful information.  An expanded dataset will help provide more nuanced information about the aptitude and competency of GSU core curriculum students.

 

Meanwhile, use of the Competent Speaker instrument (as modified to serve as a grading/feedback sheet) should also be continued and expanded, although the form should be revised to reflect usability feedback received from instructors.  In several cases the sub-categories were judged by instructors inappropriate to the overall evaluation categories, and their specific feedback would be easy to take into consideration.  Meanwhile, the overall form seemed easy for instructors to use, and also seemed to provide high quality feedback to students -- summer instructor evaluation data is in line with historical SPCH 1000 numbers, and so far as we know no students complained about the feedback they received on the public speaking assignment.  It would be helpful to explore the extent to which the Competent Speaker instrument adds information beyond that provided by the assigned grade.

 

Use of the instruments should be explicitly built into the annual instructor orientation sessions provided by the department of communication.  Although the assessment forms are relatively straightforward, consistency of use in a core curriculum course would be desirable, and some aspects of judging speech performances can be difficult.  The NCA provided criteria information should remain in use.

 

Communication faculty affiliated with university assessment efforts should look for additional instruments able to assess oral communication competency which can be used in more varied settings.  While this summer pilot project and projected 04-05 academic year plans envision continuing reliance on the Competent Speaker, an instrument that works well for providing feedback for a formal speech, the instrument works less well when it comes to assessing oral communication skills when defined more broadly.  Consider the standard methods of inculcating oral communication competency in an advanced seminar:  such courses are more likely to rely on the give-and-take of Socratic dialogue, or short reports, or panel discussions, than they are to assign major formal speeches.  If data are to be compared down the road, contrasting first- and senior-year experiences, some more general communication instruments should probably be tried in the core curriculum that will translate to the broader experiences of the junior and senior year instruction provided by major degree programs.  Otherwise, an "apples vs. oranges" problem will emerge, where detailed core curriculum-generated data relating to the delivery of a formal public address will not be usefully comparable to communication as assessed in less formal academic contexts.

 

Other courses in the core curriculum should be identified as able to enhance oral communication skills.  Although SPCH 1000 does obviously connect to the enhancement of oral communication competency, it should be noted that this is not its major purpose, and the structural limitations (section size, staffing, and limited contact hours) would make it difficult to envision SPCH 1000 functioning as a major or exclusive vehicle for the enhancement of performance skills.  An earlier report also noted the challenge of student self-selection:  because SPCH 1000 is one option in a menu, the students most apprehensive of public presentation (and thus in most need of attention and training) are the most likely to opt out of SPCH 1000 in favor of other classes where their fears are less likely to be activated.  In addition, because SPCH 1000 only involves students in one public speaking event, even well planned data collection, predicated on commonly used assessment instruments, will not likely tell us very much about whether the GSU core produces or strengthens oral communication competency.  Identifying a broader range of core classes which contain a meaningful oral communication component will help minimize these risks and assist in producing more a more useful institutional profile.

 

The University should consider implementing an initiative that would train faculty in assessing and providing feedback to oral communication skills in their classes.  One possibility would be use of a variant on widely used Writing-Across-the-Curriculum centers, something like a Communication-Across-the-Curriculum (CXC) initiative, which would enable the university to provide some centralized resources to all faculty who teach writing and speaking without incurring major additional costs relating to new faculty hiring and class size reductions.  The case for such an initiative was powerfully reinforced by our summer 04 experience.  Although many departments had identified courses they run in the core curriculum as containing an oral communication component, we found on examining their course syllabi that even where such activities are scheduled, many instructors do not provide systematic feedback relating to such oral performance.  Our impression is that oral communication activities are already structured into a wide range of course experiences, regardless of subject matter; a relatively easy fix would be to provide faculty with resources enabling them to knowledgably give feedback on activities they are already requiring, or to equip them to add such activities in an integrated manner to their existing instruction.

 

Longer term attention might also be given to evaluating oral communication in a more detailed and meaningful way.  Whether that would require core curriculum modification or not (e.g., Area B modification, explicit course modification to assure oral communication training is reinforced across the areas, etc.), remains to be seen.  But given the number of students who move through the SPCH 1000 class, it may be advisable down the road to supplement assessment by use of randomly sampled videotaping and external assessment measures of oral communication performance.  In addition, it might be advisable to add the use of other instruments which are more closely focused on other aspects of oral performance, including assessment of listening skills.  Here Georgia State University might benefit from a fuller examination of oral communication assessment at peer institutions.  Appalachian State University (NC) has piloted an effort to have students deliver presentations during first year orientation and senior year exit exams; these speeches are videotaped (and become part of a student portfolio) and samples are assessed by faculty.  While such an effort is resource intensive, it also provides by far the richest information for use in improving instruction relating to oral communication practices.

 

IV.  Fall 04 Postscript

 

Implementation of these recommendations, where possible, have been undertaken in the fall 04 term.  All sections of SPCH 1000 are administering these three instruments.  All instructors were provided a detailed one day orientation to their use and institutional purpose.  All departmental faculty have been briefed on these efforts.  The speech feedback form has been modified to reflect summer input, and we have modified the data collection process to hopefully enable case-by-case tracking (to see, in other words, whether a particular student's improvement on the apprehension measures correlates to performance in the speaking activity). 

 

 

 

References

 

Bassett, R.E., Whittington, N., and Staton-Spicer, A.  “The Basics in Speaking and Listening for High School Graduates:  What Should be Assessed?”  Communication Education 27 (1978): 293-303.

 

Christ, W.G. (eds.).  Assessing Communication Education:  A Handbook for Media, Speech, and Theater Educators (Hillsdale, N.J.:  Lawrence Erlbaum, 1994).

 

Jones, E.A.  Essential Skills in Writing, Speech and Listening, and Critical Thinking for College Graduates:  Perspectives of Faculty, Employers, and Policymakers (University Park, Penn.:  National Center for Postsecondary Teaching, Learning, and Assessment, 1994).

 

–––.  National Assessment of College Student Learning:  Identifying College Graduates’ Essential Skills in Writing, Speech and Listening, and Critical Thinking; Final Project Report [NCES Publication No. 95-001] (Washington, D.C.: U.S. Department of Education, Office of Educational Research and Improvement.

 

McCroskey, J.C.  "Reliability and Validity of the Willingness to Communicate Scale."  Communication Quarterly 40 (1992): 16-25.

 

McCroskey, J.C., Fayer, J., and Richmond, V.P.  "Don't Speak to Me in English:  Communication Apprehension in Puerto Rico."  Communication Quarterly 33 (1985): 185-192.

 

Morreale, S. and Brooks, M. (eds.). NCA Summer Conference Proceedings and Prepared Remarks:  Assessing College Student Competency in Speech Communication (Annandale, Va.: National Communication Association, 1994).

 

Quianthy, R.L.  Communication is Life:  Essential College Sophomore Speaking and Listening Competencies (Annandale, Va.:  National Communication Association, 1990).

 

Westphal-Johnson, N., and Fitzpatrick, M.A.  “The Role of Communication and Writing Intensive Courses in General Education:  A Five Year Case Study of the University of Wisconsin-Madison.”  JGE:  The Journal of General Education 51.2 (2002): 73-102.