Labelling Digital Skills: A Collaborative Approach

One of the main challenges when labelling concepts is to ensure homogeneity across the labelled data. The standard process is to adopt one definition and ask validators to manually make judgements on a set of data. Such method may lead to inconsistencies in the validated dataset, mainly due to possible bias of validators, unspoken disagreements among validators, and simple human errors - which are not to be excluded when considering a large dataset such as the ESCO classification. Similarly, a fully machine learning-driven labelling would have pitfalls, mainly related to limitations in the training dataset. As a solution, the methodology adopted to label ESCO digital skills and knowledge concepts involved a combination of the independent work of human validators and Machine Learning (ML) algorithms.

     1. Starting with the manual labelling activity, each concept is analysed based on its preferred term, non-preferred terms, and description. The labelling consists in verifying whether an ESCO concept should or should not be considered as digital, following the DigComp definition. Validators assign a score equal to 1 to digital concepts, 0 to non-digital ones. The result of the manual labelling is a list of concepts classified as digital by the experts.

     2. The ML classifier work-strand can be further divided into three parts: the creation of the training and validation datasets, the training of ML models, and the labelling activity by the best-performing ML model. The ML algorithm then assigns a probability score (between 0 and 1) to estimate the likelihood of a concept to be considered digital. The result of the algorithmic work is a list of concepts labelled as digital by the ML classifier. 

     3. Finally, the two lists of ESCO digital skills are compared, where disagreements between the two are further analysed and accepted or rejected by experts, following the same labelling rules set for the manual labelling.

The graph below plots the distribution of the skills and knowledge concepts labelled as digital as a result of the manual labelling (yellow), ML labelling (red), and final revision (blue) activities. These are distributed following the probability scores as assigned by the ML model. Differences between the red and yellow bars are an approximate indicator of the quality improvements gained by combining the two labelling methods. For example, the first set of bars from the right side shows that the number of blue (final) digital skills is higher than the number of skills labelled manually, but is lower than the number of skills detected by the model.

The combination of the two methods allowed to identify skills that represented more doubtful cases, and hence needed additional discussion and manual revision. It also helped filling gaps due to human mistakes or imperfections in the training dataset.

The Digital Skills and Knowledge Concepts: Labelling the ESCO classification report provides detailed information concerning the methodology, results, and provides an overview of the possible use-cases of the digital concepts by ESCO implementers. Digital skills and knowledge concepts can be downloaded here, while the documentation for the ESCO API is available here.

If you are an ESCO implementer and want to share your feedback, please get in touch via email at EMPL-ESCO-SECRETARIAT@ec.europa.eu or use our hashtag #ESCO_EU.