Rating the quality of collaboration during networked problem solving activities

Abstract

This paper describes the adaptation and generalization of a rating scheme for judging the quality of collaboration during network-based problem solving activities. A rating scheme, which had been developed for assessing collaboration quality on several dimensions by the Freiburg team (Meier, Spada, & Rummel, 2007), was adapted to fit data gathered by the Patras team. The goal of this adaptation study was not only to come up with a useful tool for analyzing data, but also to show that the rating scheme’s dimensions, and thus its underlying theoretical model, were capable of capturing the main aspects of collaboration quality across different CSCL settings.

The collaborative activities studied in this paper involved dyads of first year computer science students. Their joint task was to build a diagrammatic representation of an algorithm that was provided in textual form. The collaborative activity took place during a typical laboratory class and lasted 45 minutes approximately. The students interacted through Synergo (Avouris, Margaritis, Komis, 2004), a network-based synchronous collaborative drawing tool. Synergo includes a shared whiteboard, in which students can build the diagrammatic representation (i.e. a flowchart) of the algorithm, and a chat tool. In addition, Synergo provides the “Synergo Analysis Tool”, which allows researchers to play back students' actions in a video-like format, using the sequential log files of events that Synergo maintains. In that way, evaluators can review the whole activity.

Adaptation of the rating scheme involved several rounds of adjusting the dimensions' definitions to the affordances of the Synergo learning environment and characteristics of the given algorithm task, applying the given rating dimensions to a sample of collaboration records from students solving the algorithm task, and identifying aspects of the collaboration that had not yet been covered by the rating. The sample for adapting the rating scheme consisted of the data sets from six student dyads who had participated in prior studies conducted by the Patras team (Voyiatzaki et al., to appear).

The final, adapted rating scheme includes seven dimensions. Each dimension defines a standard for a specific aspect of good collaboration: First of all, collaborators have to communicate successfully using chat as well as actions in the shared whiteboard. To achieve successful communication, partners have to maintain 1) collaboration flow, i.e. engage in a coherent exchange of information through and maintain a joint focus. Further, they need to 2) sustain mutual understanding, i.e. work towards common ground. Regarding work on the actual algorithm task, a very important dimension is that of 3) exchanging knowledge and giving explanations (i.e. self- and other-directed explanations). In addition, students have to engage in 4) argumentation in order to ensure a good solution and to foster their own learning progress. This dimension refers to all activities involved in double-checking the problem-solving process, avoiding mistakes, and maintaining a critical discussion. To ensure a timely and orderly solution to the given problem, students also have to coordinate their collaboration well by 5) structuring the problem solving process and time management. Finally, students have to maintain a 6) cooperative orientation (e.g., constructive handling of disagreements), and a high level of 7) individual task orientation throughout their collaboration. More detailed definitions and illustrative case studies can be found in the full paper.

Future directions of research include the use of the theoretical model underlying the dimensions of the rating scheme to teach students how to improve their collaboration, to train teachers in assessing specific strengths and weaknesses in their students’ collaboration, and to help them provide helpful feedback to their students.