Machine Deep Learning-Algorithmen, Künstliche Intelligenz
WrightStudio / stock.adobe.com
2018-12-03 short info 284 0

Harbsafe-162: A gold standard for machine learning appliances in terminology harmonisation

VDE|DKE and the TU Braunschweig have developed Harbsafe-162, a gold standard for the evaluation of mechanical methods for text similarity determination. Harbsafe-162 is an intermediate result of the Harbsafe project of the same name. The focus lies on the harmonization of different understandings of the terms in the fields of cybersecurity, functional safety and reliability of critical IT infrastructures.

Contact

Holger Lange

Domain experts usually set standards - but not by standard procedure this time! To train and evaluate the performance of machine learning systems, human judgements are indispensable. With the aim of making terminological entries from different standards more accessible for harmonisation, the Harbsafe project – a cooperation by VDE|DKE and TU Braunschweig - aims to group semantically similar entries using techniques from computational linguistics. Especially standardisation experts can profit from such an assistance.

A solid foundation is the new gold standard Harbsafe-162 – a data set that has been developed with voluntary domain experts and serves as a yardstick for the evaluation of text similarity measures. It differs from comparable data sets by content and structure: The items belong to the realm of languages for special purposes and are complex concept-oriented terminological entries. The data set may therefore be of interest to the international research community in computational linguistics. VDE|DKE is accordingly presenting Harbsafe-162 as a gold standard for representations of concepts.

Several domain experts participated in a rating task and judged the conceptual similarity of 162 pairs of terminological entries. These have been extracted from technical standards in the areas of functional safety, cybersecurity, and dependability – domains that increasingly converge due to digitalisation. The rating scale ranges from 4 (very similar) to 0 (totally dissimilar and unrelated). Harbsafe’s inter-rater reliability is competitive with similar lexical data sets (Krippendorff’s α=0.7, Spearman‘s ρ=0.74). Harbsafe-162 includes median and average ratings of the domain experts as well as the standard deviation. A detailed discussion of Harbsafe-162 is currently in preparation.

The ongoing project Harbsafe will focus on the software solution for the assistance of standardisation experts – the Terminology Harmonisation Dashboard. Project developments will be presented at a Harbsafe workshop with standardisation experts that will be held in Frankfurt am Main on January 30th 2019.

Editorial note:

The project is funded by BMWi, the German Federal Ministry for Economic Affairs and Energy (Grants: 03TNG006A and 03TNG006B).

Downloads to Harbsafe-162

Fallback ZIP

Harbsafe-162 Dataset & FAQ

VDE|DKE & TU Braunschweig
Publishing date: 2018-11-30
ZIP: 18 KB

More information about Harbsafe-162

What is Harbsafe-162?

Harbsafe-162 is a data set for the evaluation of semantic representations, e.g. embeddings. It comprises terminological entries from international electrotechnical standards of the IEC that have been rated for similarity by linguists and domain experts.

Who owns Harbsafe-162?

Harbsafe-162 is a collaborative effort designed to answer questions in the research project Harbsafe (supported by the German Federal Ministry for Economic Affairs and Energy (BMWi), grants 03TNG006A and 03TNG006B). The terminological entries in Harbsafe-162 are part of international standards by the International Electrotechnical Commission (IEC), which have been provided to the project via the IEC’s national member, the German Commission for Electrical, Electronic & Information Technologies of DIN and VDE (DKE). The similarity ratings have been provided by international, anonymous, voluntarily participating domain experts. Special thanks go to Prof. Peter Bernard Ladkin for making the project known among cybersecurity experts. The rating task has been designed by the Harbsafe team of researchers from the DKE and Technische Universität Braunschweig.

What are the sources of the terminological entries?

In Harbsafe-162, each entry is supplied with proper reference to the source standard. Here is the complete list of references:

  • IEC 61508-4:2010 International Electrotechnical Commission (2010): IEC 61508-4:2010 Functional safety of electrical/electronic/programmable electronic safety-related systems - Part 4: Definitions and abbreviations
  • IEC 62443-2-4:2015+AMD1:2017 International Electrotechnical Commission (2015): IEC 62443-2-4:2015+AMD1:2017 CSV Consolidated version Security for industrial automation and control systems - Part 2-4: Security program requirements for IACS service providers
  • IEC TR 62443-2-3:2015 International Electrotechnical Commission (2015): IEC TR 62443-2-3:2015 Security for industrial automation and control systems - Part 2-3: Patch management in the IACS environment
  • IEC 62443-3-3:2013 International Electrotechnical Commission (2013): IEC 62443-3-3:2013 Industrial communication networks - Network and system security - Part 3-3: System security requirements and security levels
  • IEC 62443-2-1:2010 International Electrotechnical Commission (2010): IEC 62443-2-1:2010 Industrial communication networks - Network and system security - Part 2-1: Establishing an industrial automation and control system security program
  • IEC TS 62443-1-1:2009 International Electrotechnical Commission (2009): IEC TS 62443-1-1:2009 Industrial communication networks - Network and system security - Part 1-1: Terminology, concepts and models
  • IEC TR 62443-3-1:2009 International Electrotechnical Commission (2009): IEC TR 62443-3-1:2009 Industrial communication networks - Network and system security - Part 3-1: Security technologies for industrial automation and control systems
  • IEC Guide 51:2014 International Electrotechnical Commission (2014): ISO/IEC GUIDE 51:2014 Safety aspects -- Guidelines for their inclusion in standards
  • IEC Draft Guide 120 (AC/2/2017) International Electrotechnical Commission (2017): Draft Guide 120 Edition 1, Security aspects – Guidelines for their inclusion in standards (AC/2/2017)

What domain do the terminological entries belong to?

Functional safety, cybersecurity and dependability.

Are the terminological entries identical to the norm or have you made any editorial modifications?

We did some minor editorial changes but no changes in content. To support the rating task, we substituted potentially unknown acronyms for full forms, removed carriage return and line feed tokens between terms, corrected obvious spelling errors and removed cross-references to other entries that were not part of the definition.

Can I use Harbsafe-162 for reference in standardization and other areas of application?

No. You shouldn’t do this. If you want to quote terminological entries, use the appropriate standards and cite them accordingly. You can purchase the latest version at the IEC Webstore. Terminological entries can also be accessed free of charge via the IEC’s International Electrotechnical Vocabulary (IEV) at http://www.electropedia.org/ or via the German version of the IEV at https://www.dke.de/de/services/iev-woerterbuch provided by the German Commission for Electrical, Electronic & Information Technologies of DIN and VDE (DKE).

Can I use Harbsafe-162 for the evaluation of my embeddings?

Yes, but please make sure to give credits. Please use reference as follows:

  • Arndt, Susanne, Sven Müller & Dieter Schnäpp (2018): Harbsafe-162.

How is Harbsafe-162 structured?

Harbsafe-162 contains 162 pairs of terminological entries. Each row contains term, definition, source standard, number (of the entry in its source standard) of the first and the second entry (term1, nr1 etc.). In addition, each row contains the median as well as the average rating of all voluntary raters for each pair as well as the standard deviation. Pairs 152-161 do not have numbers since they are not taken from international electrotechnical standards. They serve the purpose of quality evaluation. Their glosses have been taken from the referenced online dictionaries. We modified these glosses slightly: We let each of them start with a capital letter and let them end with a full stop.

How did you make Harbsafe-162 and have you further reflected on its quality?

Our procedure and potential quality issues are as yet reflected in an unpublished manuscript:

  • Arndt, Susanne & Dieter Schnäpp (2018): Harbsafe-162 – A Domain-Specific Data Set for the Intrinsic Evaluation of Semantic Representations for Terminological Data. Unpublished Manuscript.