0人評分過此書
Rating EFL Written Performance
The study focuses on the investigation of the process during which raters of EFL written performance make their decisions. It consists of a pilot and a main study, each of which concentrates on assessment of writing. The rationale is to detect the decision-making processes that raters follow, which can be used for training raters, and with which the reliability of rating can be improved. The pilot study is based on data collected during a large-scale language proficiency assessment of two age groups from learners of English and German languages. Raters were asked to think aloud during the rating task. Data was then transcribed and analysed. The participants in the main study were novice raters, who produced verbal protocols. 37 EFL teacher trainees took part in rater training and practised think-aloud protocol production. Then, they evaluated ten compositions written by EFL learners and verbalised their thought processes. The verbal protocols served as a basis for data collection. The analysis of data resulted in the conclusion according to which more reliable and objective assessment is possible when evaluating written performance.
- cover
- title
- Copyright
- Contents
- Introduction
- Overview of the Book
- Part I An Overview of the Literature on L2 Writing and Assessment
-
Chapter 1 Writing Ability and L2 Writing Instruction
-
1.1 The Writing Skill in L1 and L2
-
1.1.1 Theoretical Models of Written Text Production
-
1.1.2 Theoretical Frameworks of Communicative Competence
-
1.2 Writing Ability in L1
-
1.3 Writing Ability in L2
-
1.3.1 The Relationship Between L1 and L2 Writing Ability
-
1.3.2 Skilled and Unskilled L2 Writers
-
1.4 L2 Writing Instruction
-
1.4.1 Controlled Composition Approach to Writing
-
1.4.2 Process-Oriented Approach to Writing
-
1.4.3 The Role of Writing in Communicative Language Teaching
-
1.5 Conclusion
-
-
Chapter 2 Assessing Language Ability
-
Introduction
-
2.1 Assessing Language Ability
-
2.1.1 History of Language Testing
-
2.1.2 Recent Developments in Language Testing Research
-
2.2 L2 Ability Assessment
-
2.2.1 Language Ability and Language Performance
-
2.2.2 Characteristics of Performance Assessment
-
2.3 Test Features
-
2.3.1 Test Purpose
-
2.3.2 Testing Methods
-
2.4 Qualities of Language Tests
-
2.4.1 Reliability
-
2.4.2 Validity
-
2.4.3 Authenticity
-
2.4.4 Impact
-
2.4.5 Washback
-
2.4.6 Practicality
-
2.5 Test Method Characteristics
-
2.6 Assessing the Four Language Skills
-
2.7 Scoring Methods
-
2.7.1 Characteristics of Rating Scales
-
2.7.2 Rater Variables
-
2.7.3 Score Interpretation
-
2.8 Test-Taker Characteristics
-
2.9 Language Test Construction
-
2.10 Conclusion
-
-
Chapter 3Assessing Writing Ability
-
Introduction
-
3.1 Written Performance Assessment
-
3.2 Alternative Ways of Writing Ability Assessment
-
3.3 Nature of Written Performance Assessment
-
3.3.1 Validity of Writing Assessment
-
3.3.2 Task Characteristics
-
3.3.3 Definition of Audience
-
3.3.4 Test Taker Characteristics
-
3.4 Conclusion
-
-
Chapter 4Rating Written Performance
-
Introduction
-
4.1 Scoring Procedures
-
4.2 Rater Variables
-
4.3 Rating as a Problem-Solving Activity
-
4.4 Frameworks of Scoring Processes
-
4.4.1 Milanovic, Saville and Shuhong’s Framework of the Scoring Process
-
4.4.2 Wolfe’s Framework of the Scoring Process
-
4.4.3 Lumley’s Framework of the Scoring Process
-
4.4.4 Cumming, Kantor and Powers’ Framework of the Scoring Process
-
4.5 Rater Stability
-
4.6 Rater Training
-
4.7 Conclusion
-
-
Chapter 5Verbal Protocol Analysis as a Research Method in Written Performance Assessment
-
Introduction
-
5.1 Research in L2 Acquisition Studies
-
5.2 Introspective Reports as Data Collection
-
5.3 Categorisation of Verbal Protocols
-
5.4 Concurrent Think-Aloud Protocols in Written Performance Assessment
-
5.5 Verbal Protocol Analysis Procedure
-
5.5.1 Data Preparation and Collection Procedures
-
5.5.2 Verbal Report Procedure Preparation
-
5.5.3 Data Transcription
-
5.5.4 Segmenting the Protocols for Coding
-
5.5.5 Developing a Coding Scheme
-
5.5.6 Analysis of the Coded Data
-
5.6 Advantages and Limitations of Verbal Protocol Analysis as Research Methodology
-
5.7 Conclusion
-
-
PART IIInvestigating Raters’ Decision-Making Processes and Awarded Scores in Rating Hungarian Efl Learners’ Compositions
-
Chapter 6A Pilot Study: Tracing Raters’ Decision-Making Processes
-
Introduction
-
6.1 Background to the Pilot Study on Assessment of Written Performance
-
6.2 Research Questions
-
6.3 Research Design
-
6.3.1 Participants: the Raters
-
6.3.2 Procedures for Data Collection
-
6.3.3 Test of Written Performance: the Task
-
6.3.4 The Assessment Scale
-
6.3.5 The Coding Scheme
-
6.4 Results and Discussion
-
6.4.1 Distribution of Comments During Rating
-
6.4.2 Comments on Rating Technicalities
-
6.4.3 General Comments on the Scripts
-
6.4.4 The Way Raters Arrived at a Score
-
6.4.5 Assessment of the Communicative Goal
-
6.4.6 Assessment of Vocabulary
-
6.4.7 Assessment of accuracy and spelling
-
6.4.8 Assessment of the Text Organisation
-
6.5 The Rating Process
-
6.6 Conclusion
-
-
Chapter 7The Main Study: Processes and Outcomes in Rating L2 English Written Performance in a Pre-Service Tefl Course
-
Introduction
-
7.1 Background to Main Study
-
7.2 Design of the Study
-
7.3 Research Questions
-
7.4 Participants
-
7.5 Data Collection Instruments
-
7.5.1 The Scripts
-
7.5.2 The Rating Scale
-
7.5.3 The Rating Task Assignment Package
-
7.6 Data Collection Procedures
-
7.6.1 Script Characteristics
-
7.6.2 Script Selection
-
7.6.3 Training Raters
-
7.6.4 The Rating Task
-
7.7 Processing Verbal Data
-
7.7.1 Transcript Segmentation
-
7.7.2 Coding Scheme Production
-
7.8 Rater Characteristics: Grouping Raters
-
7.9 Summary
-
-
Chapter 8Features of Raters’ Rating Patterns
-
Introduction
-
8.1 Raters’ Gender Distribution
-
8.2 Language of the Verbal Protocols
-
8.3 Sequencing the Scripts for Rating
-
8.4 Length of Verbal Protocols
-
8.5 Raters’ Rating Sequences
-
8.6 Raters’ Rating Patterns
-
8.7 Conclusion
-
-
Chapter 9Raters’ Rating Focus
-
Introduction
-
9.1 Raters’ Rating Foci
-
9.1.1 Management Focus: Management Strategies
-
9.1.2 Rating Focus: Rating Strategies
-
9.1.3 Reading Focus: Reading Strategies
-
9.1.4 Raters’ Own Focus: Other Comments
-
9.2 Conclusion
-
-
Chapter 10Raters’ Focus on the Four Rating Criteria
-
Introduction
-
10.1 Raters’ Focus on the Four Rating Criteria
-
10.2 Raters’ Focus When Rating Task Achievement
-
10.3 Raters’ Focus When Rating Vocabulary
-
10.4 Raters’ Focus When Rating Grammar
-
10.5 Raters’ Focus When Rating Organisation
-
10.6 Conclusion
-
-
Chapter 11Raters’ Script Interpretation on the Weakest and Top Script
-
Introduction
-
11.1 Rating Script N2: Benchmarks and Total Scores
-
11.1.1 Comments on Script N2 in the Pre-Scoring Stage
-
11.1.2 Rating Task Achievement of Script N2
-
11.1.3 Rating Vocabulary of Script N2
-
1.1.4 Rating Grammar of Script N2
-
11.1.5 Rating Organisation of Script N2
-
11.2 Ratings of Script N6: Benchmarks and Total Scores
-
11.2.1 Comments on Script N6 in the Pre-Scoring Stage
-
11.2.2 Rating Task Achievement of Script N6
-
11.2.4 Rating Grammar of Script N6
-
11.2.5 Rating Organisation of Script N6
-
11.3 Conclusion
-
-
Chapter 12Raters’ Perception of the Rating Task and Thinking Aloud
-
Introduction
-
12.1 The Feedback Sheet
-
12.1.1 Raters’ Feedback on the Language-Testing Course
-
12.1.2 Raters’ Feedback on Training for Rating Written Performance
-
12.1.3 Raters’ Feedback on the Rating Task
-
12.2 Conclusion
-
- General Conclusions
- Introduction
- Background to Written Performance Assessment
- Features of Rating Processes
- Raters’ Rating Foci
- Interpretation of the Rating Criteria
- Raters’ Script Interpretation
- Raters’ Perceptions of the Rating Task
- Placing Empirical Findings on the Rating Processes into Theoretical Frameworks
- Verbal Protocol Analysis as Data Collection Method
- Implications of the Findings
- Limitations of the Study
- Further Research
- References
-
Appendices
-
Appendix 6.1 The letter writing task in the pilot study
-
Appendix 6.2The rating scale in the pilot study
-
Appendix 6.3The coding scheme for the pilot study
-
Appendix 6.4 Number of utterances made during the rating process in the pilot study
-
Appendix 6.5 A sample from EngR1 transcript in the pilot study (translated from Hungarian)
-
Appendix 7.1 The writing task in the main study
-
Appendix 7.2 The rating scale in the main study
-
Appendix 7.3 The letter to the students in the main study
-
Appendix 7.4 The ten scripts in the main study
-
Appendix 7.5 The score sheet in the main study
-
Appendix 7.6 The feedback sheet in the main study
-
Appendix 7.7 The course description for the elective seminar course in Testing in ELT
-
Appendix 7.8 A sample verbal protocol transcript in the main study
-
Appendix 7.9 The coding scheme in the main study
-
Appendix 7.10 A sample of a coded protocol in the main study
-
Appendix 7.11 Competent raters’ total scores and rankings of the ten scripts in parentheses in the main study
-
Appendix 7.12 Proficient raters’ total scores and rankings in parentheses in the main study
-
- Index
- 出版地 ： 德國
- 語言 ： 德文
評分與評論
請登入後再留言與評分
EPUB
PDF
PDF
