Chapter 5: Conclusion

In this chapter, we review our research questions and hypotheses and how our contributions address the research challenges (see Table 5.1). Furthermore, we discuss remaining challenges and future directions.

Table 5.1: Alignment between our contributions, the challenges, research questions, hypotheses, and whether the contribution is part of the rule creation, refinement, or execution.

Impact of contributions

We discuss each research question and corresponding hypothesis together with the contribution that tackles them, followed by how the contribution impacts the challenges. The first research question is

Research Question 1: How can we design visualizations that improve the cognitive effectiveness of visual representations of knowledge graph generation rules?

with corresponding hypothesis

Hypothesis 1: MapVOWL improves the cognitive effectiveness of the generation rule visual representation to generate knowledge graphs compared to using RML directly.

The evaluation conducted for MapVOWL provided evidence to accept Hypothesis 1: the use of MapVOWL is preferred over RML for representing knowledge graph generation rules. Furthermore, users would also prefer to use MapVOWL to create and edit rules (see Chapter 2). The second research question is

Research Question 2: How can we visualize the components of a knowledge graph generation process to improve its cognitive effectiveness?

with corresponding hypothesis

Hypothesis 2: The cognitive effectiveness provided by the RMLEditor's GUI improves the user's performance during the knowledge graph generation process compared to RMLx.

The evaluation conducted for the RMLEditor provided evidence to accept Hypothesis 2: the RMLEditor is preferred over RMLx for creating rules. This is due to the use of MapVOWL, which is independent of the underlying language, as pointed out by the participants of the evaluation (see Chapter 2). By tackling Research Questions 1 and 2, MapVOWL and the RMLEditor contribute to tackling the following challenges

Challenge 1: Improvement of users' understanding of the rule creation's components.

Challenge 2: Avoidance and removal of inconsistencies introduced by concepts, relationships, and semantic model.

MapVOWL improves the users' understanding of the rules without being required to understand the syntax and grammar of the used language, because users only need to understand the visual notation. This notation is translated to language-specific rules by, for example, the RMLEditor. Furthermore, it improves the users' understanding of how the data sources are related to the used concepts and relationships, and how the semantic model affects the generated knowledge graphs. The RMLEditor further aids users by dealing with the visualization-related scalability issues, the visualization of heterogeneous data sources, and the integration of transformations of the data sources in the visualizations of the rules.

The third research question is

Research Question 3: How can we score and rank rules and ontology terms for inspection to improve the manual resolution of inconsistencies?

with corresponding hypothesis

Hypothesis 3: The automatic inconsistency-driven ranking of Resglass improves, compared to a random ranking, by at least 20% the overlap with experts' manual ranking.

The evaluation conducted for our ranking, which is part of Resglass, provided evidence to accept Hypothesis 3: our ranking has 80% overlap with experts' rankings for both rules and ontology terms. Furthermore, our ranking improves the overlap with expert's with 41% for rules and 23% for ontology terms, compared to a random ranking (see Chapter 3). By tackling Research Question 3, Resglass contributes to tackling Challenge 2: it assists users with the removal of inconsistencies introduced by ontologies and semantic model. However, it does not assist in avoiding them.

The fourth research question is

Research Question 4: What are the characteristics of test cases for processors that generate knowledge graphs from heterogeneous data sources independent of languages' specifications?

With the introduction of an initial set of RML test cases

• developers can determine how conformant their RML processors are to the RML specification, and
• users can use the test cases results to select the most appropriate processor for a specific use case (see Chapter ).

By working towards Research Question 4, our RML test cases provide useful insights for the following challenge

Challenge 3: Selection of the most suitable processor for the use case at hand.

These test cases help towards determining the characteristics of test cases that are independent of the RML language, because they already list a number of differences between test cases of languages that only work with relational databases and languages that support multiple, heterogeneous data sources.

Remaining challenges and future directions

MapVOWL is preferred over RML to visualize knowledge graph generation rules. However, future research is needed to investigate how to visualize common data structures, such as lists and bags. Furthermore, the representation of these rules should not necessarily only happen via visualizations (see Chapter 2): there are users who desire a text-based approach. To this end, initial steps have been taken by introducing YARRRML, a human-readable text-based representation for knowledge graph generation rules. Future research efforts can compare YARRRML to existing languages, such as RML and SPARQL-Generate, through user evaluations that are similar to the one for MapVOWL. This allows a better understanding of the advantages and disadvantages of representations, together with what needs to be improved. Furthermore, research with respect to how collections, such as lists and bags, are represented is also of interest, because although they do not have a dedicated notation in RDF it is not straightforward to define how they are generated using the existing representations.

The RMLEditor is preferred over RMLx. However, future research is needed to avoid or clarify specific terminology and combine MapVOWL with other representations, such as YARRRML, to support users who do not (only) rely on visualizations. Furthermore, additional features can be added to the RMLEditor to improve the overall rule creation process, such as importing an ontology for which the corresponding rules are created based on the classes, properties, and datatypes in the ontology. Future research efforts can compare the RMLEditor to other rule editors through user evaluations that are similar to the one with RMLx. This allows a better understanding of how the RMLEditor compares to other editors that are different from RMLx, i.e., apply a different approach or support different features.

Resglass, including its ranking of rules and ontology terms, improves the manual resolution of inconsistencies in knowledge graphs. Future research can investigate how to further assist users in both detecting and resolving inconsistencies, such as, for example, suggesting possible resolutions based on the used and other ontologies, and rules of other use cases. Furthermore, Ressglass' ranking can also be used to drive a (semi-)automatic solution. For example, this can be done by building on our previous work where we proposed an automatic solution that resolves the inconsistencies by applying a predefined set of refinements to the rules. The ranking, which is use case-specific, can be used to make a more informed decision regarding which refinements should be applied, instead of solely relying on a predefined set, which is created independent of the use case. Besides the removal of inconsistencies, future research is needed to avoid inconsistencies as early as possible in the rule creation process. This can be done by extending the RMLEditor to guide users when selecting relevant data, ontologies, and their terms.

The initial set of RML test cases benefits both developers and users of RML processors, and highlights differences with test cases for relational database-specific languages. However, future research is needed to further investigate the characteristics of test cases that can easily be applied to other languages, that combine different data formats, and that tackle the use cases where data streams are used to generate knowledge graphs instead of static files or databases. This might and hopefully will trigger the discussion on what should be part of a knowledge graph generation language and what not, which will be reflected on what is tested by the test cases. Past research efforts showed that supporting multiple, heterogeneous data sources is a desired feature. However, to what extend is for example pre-processing of the data sources required? Pre-processing can go from fixing capitalization in sentences to deduplication of entities using additional data sources. Do we include the former and the latter? Or only one of them? Thus, which features are in the scope of a knowledge graph generation language and which are not? Furthermore, investigating other metrics, besides conformance, further improves the selection of the most suitable processor. Such metrics could include but are not limited to speed, memory consumption, usability, and extensibility.