Assessing the Quality of Synchronous Network Learning Activities using Machine Learning Techniques

Abstract

During network-mediated synchronous collaborative activities there is need for supporting reflection of the learners involved, by providing them with meaningful feedback on the state of group activity. In order to produce timely feedback to the partners, we need to automate processing activity data and producing meaningful measures of the quality of collaboration to be fed back to the students. This paper presents a study investigating applicability and effectiveness of machine learning techniques in the process. The objective is to use different classification algorithms for assessing quality of collaboration using a set of quantitative indices produced by the NSCL environment. Collaboration quality, however is a term that needs first to be defined using a relevant scheme.

The typical collaboration activities studied involved dyads of students following a distance learning computer science course. The dyads were asked to solve an algorithm problem and develop the solutions in the form of flowchart diagrams. They used a NSCL tool, Synergo, that provides a frame of reference, a shared drawing space through which various diagrammatic representations can be built jointly by a group of collaborating partners and a chatting tool for direct text based communication. In addition, Synergo offers a rich set of analysis tools that serve for analysis of collaboration by teachers or researchers. Synergo calculates a number of quantitative metrics of the activity. Statistics of logged actions are automatically provided. Messages and different types of actions on the shared workspace can be treated differently and this is important for the investigation of different patterns of activity, e.g. measures like the "symmetry" of collaboration or the "symmetry" of interaction are considered important indicators of balanced contribution of the partners involved.

First the activity of a number of typical dyads was segmented and rated according to an adapted collaboration rating scheme. Then for each collaboration dimension, a study was conducted in order to find out if the quantitative indices of the activity may be used for calculating with enough accuracy the corresponding dimension. Various established classification algorithms were used in order to calculate the value of the dimension. Bayes networks, decision trees, metalearing algorithms among others constituted a powerful palette of machine learning tools that produced similar results.

Findings of this study were encouraging although they differed along different dimensions. Prediction of communication dimensions was satisfactory since it was expected that good communication should be correlated with quantitative indices such as the number of messages exchanged or the symmetry of messages between participants. There was significant diversity of the prediction of dimensions of the cognitive domain. Whereas the most elementary cognitive skills as indicated in the activity were predicted with satisfactory rates, the accuracy rates of higher order elaborations were rather poor. For example, evaluation of the solution could not be predicted reliably. An interesting finding related to the cognitive dimensions was that the individual dimensions were more difficult to predict accurately than the collaborative ones. For example evaluation of a solution by one partner by himself could not be correlated to quantitative indices. On the other hand, there were better results for collaborative evaluation since joint evaluation is more bound to be reflected in the metrics produced by Synergo. Finally, dimensions related to orientation of the students toward collaboration and dedication to the task were predicted with the most success