Summary Report on GSU Fall 04 & Spring 05 Assessment of

Summary Report on GSU Fall 04 & Spring 05 Assessment of

Oral Communication Competency in the University Core Curriculum=

Draft prepared by: Dr. Davin Allen Grindstaff, Speech = 1000 Course Director

September 26, 2005

I. Oral Communication Assessment: Rationale and Method= s

II. Assessment Activity in the 2004-2005 Academic Year=

In the fall 2004 and spring 2005 semesters, all Speech= 1000 sections administered three measures assessing students' oral communication competen= cy (see details below). These assessment measures, used for identical purposes in summer 2004, yielded mo= re extensive data and thus enabled broader claims regarding student aptitude.<= span style=3D'mso-spacerun:yes'> As recommended in the summer 2004 assessment report, all Speech 1000 instructors were oriented to the three assessment measures at the beginning of the fall 2004 term and the measures were universally administered across sections for fall 2004 and spring 2005 semesters. Additionally, instructors discussed communication apprehension issues with students with = more regularity, emphasizing the importance of these measures to University-wide learning objectives. To impro= ve record-keeping, the Communication Department also created data storage space for assessment materials. Rev= iewing all Communication curricula, the Undergraduate Committee identified promoti= ng and teaching oral competency as a central aim. The Communication Department has c= ontinued its efforts to strategize communication across curriculum, including meeting with faculty in other Area B courses to confer about communication issues.<= br clear=3Dall style=3D'page-break-before:always'> III. Fall 04: Assessment Data Summary

PRCA-24

= &nb= sp; = &nb= sp; = Table 1: Descriptive Statistics

	N	Minimum	Maximum	Mean	Std. Deviation
pre ca group=	269	6.00	30.00	14.8587	5.04851
pre ca meet<= /o:p>	269	6.00	30.00	15.6877	5.15353
pre ca dyad<= /o:p>	269	6.00	30.00	13.8290	4.60537
pre ca public	269	6.00	30.00	18.4201	5.66922
pre ca all	269	24.00	111.00	62.7955	16.95844
post ca group	236	6.00	30.00	15.5890	5.87716
post ca meet=	269	6.00	30.00	15.6877	5.15353
post ca dyad=	236	6.00	30.00	15.3559	5.66591
post ca public	236	6.00	30.00	18.2161	5.61608
post ca all<= /o:p>	236	32.00	107.00	64.5975	14.93087

The Personal Report of Communication Apprehension (see Appendices), used widely in the field of communication studies due to its internal validity and reliability, measures students’ perceived apprehension across diverse communication contexts, including: small group discussions, meetings, interpersonal dyads and public speaking. Although the survey generates data regarding apprehension in all areas, the public speaking context remains the most relevant to our current assessment goals.

Although PRCA-24 surveys were administered in all Spee= ch 1000 sections in the fall 2004 term, only ten sections were selected for da= ta reporting purposes.[2] In these ten sections, 269 pre-test surveys were administered in the fall 2004 semester and 236 post-test surve= ys were collected. As indicated = in Table 1 (above), the overall mean score showed an slight increase in apprehension (from 62.8 in the pre-test to 64.6 in the post-test). This counter-intuitive result migh= t be explained in several ways. In= the first instance, research methods likely played a role. The use of a relatively small samp= le (approximately 30% of all Speech 1000 students) might have compromised the overall reliability of the measure. In the second instance, course curriculum might not be up to the tas= k of reducing apprehension across all types of situations. While Speech 1000 students are introduced to course materials covering group and interpersonal communicati= on, for example, they are not given practical training in these areas thus pote= ntially creating the perception of lacking communication competency in those settings. Notice that the pub= lic speaking apprehension scores dropped from 18.4 to 18.2, which might support this conclusion.

Willingness to Communicate

= &nb= sp; = &nb= sp; = Table 2: Descriptive Statistics

	N	Minimum	Maximum	Mean	Std. Deviation
pre wc public	231	10.00	141.67	63.8947	20.34421
pre wc meet<= /o:p>	232	5.00	100.00	62.7701	19.89213
pre wc group=	232	20.00	173.33	71.9138	17.49971
pre wc dyad<= /o:p>	232	28.33	100.00	69.4914	17.32945
pre wc stranger	231	.00	100.00	36.4351	25.46800
pre wc acquaint	232	.00	100.00	73.9332	21.54356
pre wc friend	232	47.50	155.00	90.5700	11.78527
pre wc total=	231	77.50	300.00	200.7955	46.02574
post wc public	220	3.33	100.00	65.5348	20.49854
post wc meet=	220	8.33	100.00	65.3227	19.62079
post wc group	219	16.67	100.00	72.1050	17.58473
post wc dyad=	220	20.00	100.00	68.8273	18.12896
post wc stranger<= o:p>	219	.00	100.00	41.7534	25.77796
post wc acquaint<= o:p>	220	11.25	100.00	73.9500	20.86992
post wc friend	220	26.25	100.00	88.0659	15.98537
post wc total	219	74.25	300.00	204.3379	48.19018

The Willingness to Communicate Scale (see Appendices),= also used universally among Speech Communication departments, indicates students’ predisposition to engaging in or avoiding communication contact. Like the PRCA-24 sur= vey, WTC measures students’ responses in different communication settings = and once again the public communication setting is most relevant to assessing o= ral communication competencies in the Speech 1000 curriculum.

The sampling methods were identical to those used in t= he PRCA-24 measure, thus producing 232 pre-tests and 220 post-tests.[3] As shown in Table 2 (above), the m= ean score indicated a small overall increase in students’ willingness to communicate in the various situations, from 200.8 pre-test to 204.3 post-te= st. The public communication score and communication with a stranger score both increased (from 63.9 to 65.5 and f= rom 36.4 to 41.8 respectively) and logic suggests that the public speaking assignment in Speech 1000 is partly (but not wholly) responsible for these results.

When combined with the PRCA-24 scores from the fall 20= 04 term, this assessment data suggests two conclusions. First, exhaustive data collection (enabled by the recently added online component to Speech 1000) will greatly improve the validity and reliability of these measures in the 2005-2006 Academic Year. Second, the pu= blic speech assignment in Speech 1000, while not the perfect vehicle for improvi= ng oral competency by itself, provides GSU undergraduate students with an opportunity to develop those skills and achieve the stated learning outcomes for the core curriculum.

The Competent = Speaker

Table 3: Me= an Scores for 8 Performance Categories (n =3D 468)

1. Chooses and narrows a = topic appropriately for the audience and occasion.	2.67
2. Communicates the thesis/specific purpose in a manner appropriate for audience and occasion= .	2.39
3. Provides appropriate supporting material based on the audience and occasion.	2.51
4. Uses an organizational= pattern appropriate to the topic, audience, occasion and purpose.	2.49
5. Uses language that is appropriate to the audience, occasion and purpose.	2.78
6. Uses vocal variety in = rate, pitch, and intensity to heighten and maintain interest.	2.67
7. Uses pronunciation, gr= ammar, and articulation appropriate to the designated audience.	2.79
8. Uses physical behavior= s that support the verbal message.	2.50
Mean Summative Score of Competencies	2.60

468 Competent Speaker evaluations were completed acros= s all Speech 1000 sections.[4] The eight performance competencies measured correspond to a universal grading form used in all Speech 1000 sections during the fall 2004 term. Each performance competency was measured on a three-point scale, wit= h 1 =3D unsatisfactory, 2 =3D satisfactory, and 3 =3D outstanding. The Competent Speaker scores range higher than one might intuitively expect due to the way in which the speech assignment operates within the Speech 1000 curriculum. Students are only provided a single opportunity to hone their public speaking skills; and instructors’ evaluations often reflect an acknowledgement of the assignment’s limi= ted ability to improve the performance competencies targeted. Generally speaking, however, we mi= ght conclude that the assignment does improve oral competency among GSU undergr= aduates and reduce speech apprehension to some degree. Collecting more exhaustive data and examining more diverse relationships between variables (see Recommendations= for 2005-2006 Assessment) should yield a more nuanced assessment of the Speech = 1000 course and its development of oral competency skills.
IV. Spring 05: Assessment Data Summary

PRCA-24

= &nb= sp; = &nb= sp; = Table 4: Descriptive Statistics

	N	Minimum	Maximum	Mean	Std. Deviation
precagroup	290	6.00	30.00	13.8586	5.09603
precameet	290	6.00	30.00	14.8172	5.23209
precadyad	290	6.00	27.00	13.4207	4.81516
precapublic<= /o:p>	290	6.00	30.00	17.5862	5.64839
precaall	290	24.00	114.00	55.5172	18.25563
postcagroup<= /o:p>	237	6.00	30.00	13.9494	4.87965
postcameet	238	6.00	30.00	14.5000	5.02042
postcadyad	238	6.00	30.00	13.2605	4.59951
postcapublic=	238	6.00	30.00	17.2227	5.59204
postcaall	237	24.00	115.00	58.8354	17.13587

In spring 2005, ten sections of Speech 1000 were selec= ted for data interpretation using the PRCA-24 measure, producing 290 pre-tests = and 238 post-tests. As indicated = in Table 4 (above), the overall mean score showed an slight increase in apprehension (from 55.5 in the pre-test to 58.8 in the post-test). Once again, method limitations and= the Speech 1000 curriculum might explain this result (see PRCA-24 results for f= all 2004 above). Likewise, spring= 2005 sections demonstrated a slight decrease in apprehension for the public communication scale, dropping from 17.6 (pre-test) to 17.2 (post-test).

Willingness to Communicate

= &nb= sp; = &nb= sp; = Table 5: Descriptive Statistics

	N	Minimum	Maximum	Mean	Std. Deviation
prewcpublic<= /o:p>	275	.00	100.00	61.3527	21.28482
prewcmeet	275	.00	100.00	61.7079	19.46700
prewcgroup	275	1.67	220.00	69.3200	20.28172
prewcdyad	275	2.00	100.00	69.1782	17.70205
prewcstranger	275	.00	161.25	36.8445	25.00398
prewcacquaint	275	1.25	100.00	70.7627	22.35702
prewcfriend<= /o:p>	275	.00	100.00	88.5618	14.41612
prewctotal	275	5.25	310.00	196.1691	49.26643
powcpublic	245	.00	100.00	63.3088	21.83535
powcmeet	245	11.67	100.00	64.2639	20.16305
powcgroup	245	11.67	218.33	72.8830	20.33987
powcdyad	245	10.00	220.00	71.2367	19.99490
powcstranger=	245	.00	152.50	41.1367	25.74182
powcacquaint=	245	17.50	100.00	73.7959	20.58171
powcfriend	245	15.00	210.00	88.8367	16.86635
powctotal	245	37.75	340.00	203.7694	50.84550

In spring 2005, 275 pre-tests and 245 post-tests of th= e WTC measure were calculated and interpreted.

As shown in Table 5 (above), the mean score indicated a small overall increase in students’ willingness to communicate in the various situations, from 196.2 pre-test to 203.8 post-test. The public communication score and communication with a stranger score both increased (from 61.4 to 63.3 and f= rom 36.8 to 41.1 respectively) and logic suggests that the public speaking assignment in Speech 1000 is partly (but not wholly) responsible for these results.
The Competent Speaker

Table 6: Me= an Scores for 8 Performance Categories (n =3D 256)

1. Chooses and narrows a = topic appropriately for the audience and occasion.	2.78
2. Communicates the thesis/specific purpose in a manner appropriate for audience and occasion= .	2.54
3. Provides appropriate supporting material based on the audience and occasion.	2.42
4. Uses an organizational= pattern appropriate to the topic, audience, occasion and purpose.	2.60
5. Uses language that is appropriate to the audience, occasion and purpose.	2.96
6. Uses vocal variety in = rate, pitch, and intensity to heighten and maintain interest.	2.74
7. Uses pronunciation, gr= ammar, and articulation appropriate to the designated audience.	2.95
8. Uses physical behavior= s that support the verbal message.	2.65
Mean Summative Score of Competencies	2.71

256 Competent Speaker evaluations were collected and recorded in spring 2005, yielding results similar to the fall 2004 semester (see analysis and conclusions above).

V. Recommendations for Expanded Assessment Activity in= the ‘05-‘06 Academic Year

Online Assessm= ent

The PRCA-24 and WTC measures have been built into a new Speech 1000 online component, thus enabling students to complete both pre-t= est and post-test measures efficiently and consistently across sections. Administering the PRCA-24 and WTC measures online will yield several benefits for gathering and reporting assessment data in the fall 2005 & spring 2006 terms. Online assessment will:

[5]<= ![endif]>

Communication = Across the Curriculum

An initiative has been proposed by department of communication faculty over the past year and is currently under broader review. The specific developm= ent of an initiative awaits broader major-by-major assessment, the results of which are needed to inform the specific commitments of a university-wide initiati= ve.