Controlled Vocabularies

Requirements for controlled vocabularies

• Be published under an open licence.
• Be operated and/or maintained by an institution of the European Union, by a recognised standards organisation or another trusted organisation.
• Be properly documented.
• Have labels in multiple languages, ideally in all official languages of the European Union.
• Contain a relatively small number of terms (e.g. 10-25) that are general enough to enable a wide range of resources to be classified.
• Have terms that are identified by URIs with each URI resolving to documentation about the term.
• Have associated persistence and versioning policies.

Expected usage of controlled vocabularies

• The property MUST use as range values codes from {codelist}
  This expectation results in that the value space is closed under the codelist. Validation systems SHOULD produce errors. All profiles in the ecosystem MUST avoid conflicts by creating subproperties.
• The property MUST have at least one value from {codelist}
  This expectation makes the value space minimally constrained. Validation systems SHOULD produce warnings. All profiles in the ecosystem MUST adopt this interpretation in case they want to restrict the value space. Adopting the former expectation will lead to cross-profile interpretations challenges.
• The property IS RECOMMENDED to use as range values codes from {codelist}
  The value space is closed under the codelist, but other values are tolerated. Recommending means expressing a strong preference. To make this stronger direction towards harmonisation visible in the data exchanges, validation systems SHOULD produce warnings.
• The property MAY use as range values codes from {codelist}
  The value space is closed under the codelist, but other values are accepted. This expectation is more a hint to be used. As there is no obligation nor strongly suggestion expressed here, the use of other codelists is valid is all cases. Therefore validation systems MAY produce warnings but users are free to ignore them. No validation in this case is also acceptable.

The first two (and preferably also the others) SHOULD only be used for requirements that can be verified by a machine. If the validation can only be realised with the involvement of humans, then a less strong requirement (RECOMMMENDED or MAY) is used, even if the intention is to be very strict. This is to ensure that the provided SHACL representations correspond closely to all use cases possible. Stronger enforcements are left for implementations as they have control on actual data exchange.

Controlled vocabularies to be used

Properties with controlled vocabularies that MUST be used for the listed properties

Properties with AT LEAST 1 value from the controlled vocabularies

Properties with a RECOMMENDED use of controlled vocabularies

Properties with MAY use a controlled vocabularies

Other controlled vocabularies

Legal information

Providing correct and sufficient legal information is important to create trust by candidate reusers. Without explicit legal information there is a risk involved in the reuse of data, because one has no way of knowing what is allowed. While in the past legislation and initiatives such as Findable, Accessible, Interoperable and Reusable [[FAIR]] data stress the importance of this information, recent legislations, such as the High Value Datasets regulation, have become more demanding by imposing a minimum level of reuse. As a general principle, it is recommended that publishers consult their legal services to provide the appropriate values.

Agent Roles

The first version of DCAT Application Profile [[vocab-dcat-1]] had a single property to relate an Agent (typically, an organisation) to a Dataset. The only such ‘agent role’ that could be expressed in that version of the profile is through the property publisher, defined as “An entity responsible for making the dataset available”. A second property is available in that DCAT recommendation [[vocab-dcat-1]], contact point, defined as “Link a dataset to relevant contact information which is provided using vCard”, but this is not an agent role as the value of this property is contact data, rather than a representation of the organisation as such. In specific cases, for example in exchanging data among domain-specific portals, it may be useful to express other, more specific agent roles. In such cases, extensions to DCAT-AP may be defined using additional properties with more specific meanings.

Two possible approaches have been discussed, particular in the context of the development of the domain-specific GeoDCAT Application Profile [[geodcat-ap]]. The first possible approach is based on the use of a predicate vocabulary that provides a set of properties that represent additional types of relationships between Datasets and Agents. For example, properties could be defined, such as foo:owner, foo:curator or foo:responsibleParty, in addition to the use of existing well-known properties, such as dct:creator and dct:rightsHolder. A possible source for such additional properties is the Roles Named Authority List maintained by the Publications Office of the EU. Other domain-specific sources for additional properties are the INSPIRE Responsible Party roles ,the Library of Congress’ MARC relators and DataCite’s contributor types. To enable the use of such properties, they must be defined as RDF properties with URIs in a well-managed namespace. GeoDCAT-AP has introduced a number of specific properties.

A second approach is based on the use of W3C’s PROV ontology [[prov-o]] which provides a powerful mechanism to express a set of classes, properties, and restrictions that can be used to represent and interchange provenance information generated in different systems and under different contexts. In the context of work on GeoDCAT-AP, a PROV-conformant solution for expressing agent roles was agreed . This solution uses prov:qualifiedAttribution in combination with a dct:type assertion pointing to the code list for Responsible Party Role in the INSPIRE registry. To enable the use of such types, they must be defined with URIs in a well-managed namespace.

Based on the experience gained with the use of domain-specific extensions for additional ‘agent roles’ in the exchange of information about Datasets and on the requests of implementors and stakeholders, the DCAT Application Profile release 2.0.0 is extended with additional roles as proposed by DCAT Version 2 [[vocab-dcat-2]] that have proven to be useful across domains. Precisely, properties creator, qualified attribution and qualified relation have been added to Dataset class to further facilitate relationships between datasets and agents.

In the most recent DCAT Version 3 [[vocab-dcat-3]] a dedicated section on the relationship with Agents is provided. The DCAT-AP guidelines for Agents Roles are conformant to this.

As a technical note: While both approaches represent equivalent effort for publishers of datasets and editors of the specifications overall, their governance models differ. For the first approach using direct properties, the relationship definition occurs within the profile/extension. The governance is external for the second approach using codelists. This may lead to unwanted overhead, e.g. to manage mappings between the distinct codelists different profiles may use. The impact between both is primary operational on querying the data catalogue. Direct properties support simple queries. Codelists necessitate complex queries, often extended with dynamic mapping logic between the external lists.

It should be noted that, even if the second approach is used in a particular implementation, the provision of information using dct:publisher for the Catalogue is still mandatory under the rules laid down in the Conformance Statement in section [[[#conformance]]], while the provision of information using dct:publisher is strongly recommended for Dataset. The provision of such information using dct:publisher will ensure interoperability with implementations that use the basic approach of DCAT-AP. As long the first approach is actively be used, the second approach is thus limited for use to a profile specific context. For that reason, it is recommended to use the direct properties including the ones defined by GeoDCAT-AP such as custodian, distributor, originator, principalInvestigator, processor, resourceProvider in favor of the second approach using the prov:qualifiedAttribution.

Accessibility and Multilingual Aspects

Accessibility in the context of this Application Profile is limited to information about the technical format of distributions of datasets. The properties dcat:mediaType and dct:format provide information that can be used to determine what software can be deployed to process the data. The accessibility of the data within the datasets needs to be taken care of by the software that processes the data and is outside of the scope of this Application Profile.

Multilingual aspects related to this Application Profile concern all properties whose contents are expressed as strings (i.e. rdfs:Literal) with human-readable text. Wherever such properties are used, the string values are of one of two types:

• The string is free text. Examples are descriptions and labels. Such text may be translated into several languages.
• The string is an appellation of a ‘named entity’. Examples are names of organisations or persons. These names may have parallel versions in other languages but those versions don’t need to be literal translations.

Wherever values of properties are expressed with either type of string, the property can be repeated with translations in the case of free text and with parallel versions in case of named entities. For free text, e.g. in the cases of titles, descriptions and keywords, the language tag is mandatory.

Language tags to be used with rdfs:Literal are defined by BCP47 [[rfc5646]], which allows the use of the "t" extension for text transformations defined in RFC6497 [[rfc6497]] with the field "t0" indicating a machine translation.

A language tag will look like: "en-t-es-t0-abcd", which conveys the information that the string is in English, translated from Spanish by machine translation using a tool named "abcd".

For named entities, the language tag is optional and should only be provided if the parallel version of the name is strictly associated with a particular language. For example, the name ‘European Union’ has parallel versions in all official languages of the union, while a name like ‘W3C’ is not associated with a particular language and has no parallel versions.

For linking to different language versions of associated web pages (e.g. landing pages) or documentation, a content negotiation mechanism may be used whereby different content is served based on the Accept-Languages indicated by the browser. Using such a mechanism, the link to the page or document can resolve to different language versions of the page or document.

All the occurrences of the property dct:language, which can be repeated if the metadata is provided in multiple languages, MUST have a URI [[rfc3986]] as their object, not a literal string from the ISO 639 code list.

How multilingual information is handled in systems, for example in indexing and user interfaces, is outside of the scope of this Application Profile.

General usage guidelines

Usage guide on Datasets, Distributions and Data Services

The introduction of Data Services as first class citizens in DCAT 2.0 raised questions about the usage of Data Services and Distributions. This section provides a guideline for publishers what to consider as a Distribution and what as a Data Service.

A first distinction between distributions and data services is their dependency on a dataset for their existence. A distribution cannot exist without its dataset. It is a specific representation of a dataset (cfr definition W3C Distribution). Whereas a data service is an entity in its own right. It provides access to datasets or it provides data processing functions. The independence also holds between the distributions of a dataset, and the data service which provides access to that dataset. The distributions are not required to be the result of the data service operations. However, they may.

Many of the properties of distributions are file oriented (downloadURL, format, byte size, checksum, modification date, ...). The relevance of this information is reduced for data services, related information is present in a very different form and thus under different terminology. For instance, data services do and can provide format transformations, language transformations and schema transformations on request. Also the handling of trust is different. While tampering of downloadable content is detected by e.g. checksums, data services create often a trusted channel using security measures such as authentication and encryption. This reduces the need for additional trust checks on the data.

The difference between downloading a file or accessing the data through a service have resulted in the following guidelines:

• Datasets are the conceptual entity denoting a collection of data.
• Distributions are specific representations of a dataset, described with the intent to facilitate the delivery of the data as file to a reuser. The access URL (preferably download URL) will provide a simple way to obtain (download) the content of the dataset in the representation specified by the distribution. The obtained (downloaded) content is fully determined by the publisher of the distribution. Only after obtaining the data the (re)user can change the data according to its needs.
• Anything that has not the intent to provide a downloadable representation of a dataset is a data service. Data services offer smarter, more interactive ways to the data.

Orthogonal to the nature clarification of distributions and data services, there might be need for a granularity clarification between datasets and distributions. Commonly, at first sight, it is expected that all distributions of a dataset are identical in content, only differing in the representation of the data. But when considering dataset series, this interpretation seems not valid anymore. In the upcoming release of DCAT 3.0, dataset series and dataset versioning are addressed. Implementors are advised to already take this proposal into account when creating guidelines for distributions. Note that this is less an issue between datasets and data services as both are independent entities. Data services usually address the granularity by providing the necessary query interface language so that the user can get the data according its needs.

These guidelines will be able to capture many access patterns, corresponding to most users' expectations. However there might be cases that are more vague. In that case the DCAT(-AP) community can be questioned for a recommended approach.

Usage guide on Dataset Series

The need for sharing this grouping explicitly is strongly use case dependent, and therefore as this will require additional metadata management effort by the publisher, the use of Dataset Series is optional. It should fit the objectives. For instance, if a publisher is sharing an active updated dataset accessible via an API, that provides current as historic data, then it is not mandatory to created metadata records for each snapshot per year. Only if these snapshots are created intentionally and the publisher wants to share the life cycle of them with the public, then Dataset Series come into the picture.

In order to harmonise the use of Dataset Series, the following guidelines are to be considered:

• To introduce a Dataset Series only if the publisher has the intent to manage a collection of datasets. Dataset Series with a single dataset in the collection should be avoided.
• Dataset Series without a dataset in the collection should be avoided. In that case, publishers should ensure that the metadata, in particular, the life cycle metadata is up to date. For instance, if dataset get retracted and as consequence the collection becomes empty, then publishers should consider to retract also the Dataset Series. In case, publishers use persistent identifiers, then this should be reflected in the status of the Dataset Series.
• A Dataset Series is an conceptual dataset without distributions. When interpreting a Dataset Series as a Dataset, then it is expected that the directly associated Distributions contain the data that is the union of all Datasets in the collection. But the presence of these Distributions raise semantical conflicts such as whether the property of the Dataset Series frequency refers to the update frequency of the associated Distributions or the update frequency of the collection. To avoid these semantical conflicts, it is recommended not to associate distributions with a Dataset Series.
• The members of a Dataset Series should be described in such a way that without the knowledge of the Dataset Series, the metadata is interpreted correctly. The existence of a dataset should not be based on the membership in a Dataset Series. The interpretation of a member with its associated Distributions and Data Services is conform the above guidelines.
• Whenever a Data Service is providing access to all the data of a Dataset Series, i.e. the union of all datasets in the collection of the Dataset Series, then the Data Service may refer to the Dataset Series using serves dataset. The ensures that whenever in the future the Dataset Series is extended with new Datasets the Data Service description does not need to be adapted. In that case the Dataset Series should also be considered as a DCAT-AP Dataset.
• Keep the usage simple. Complex nesting such as Dataset Series being a member of Dataset Series, or Dataset belonging to multiple Dataset Series are to be handled with care. While from a data knowledge graph these nestings are feasible and not so problematic, it must be realised that these complicate the presentation on a webpage. Since data portal representations are the main communication from metadata owners to potential data consumers and there is no clear expectations on how to present the members of a Dataset Series in a UI (e.g. because ordering is not imposed), DCAT-AP users may not expect from other portals which harvest and process their Dataset Series, that these are more than a collection of Datasets. Any interpretation, such as the collection is ordered, or grouped according to time or any wellknown dimension, is a specific expectation beyond the agreemenents made in DCAT-AP.
• Conceptually, a Dataset Series is a special kind of Dataset. It is recommended to apply the same metadata quality of Datasets to Dataset Series (see [[[#quick-reference]]] ). The dedicated usage guidelines for Dataset Series and the Dataset members of a Dataset Series take priority over the generic DCAT-AP Dataset guidelines.

In general it is expected that the members of Dataset Series are strongly connected. However, there are no common criteria or rules how this connection could be determined. Usually, the shared characteristics are expressed as a data domain, e.g. the population of bees, and some evolution in space and time, e.g. in Greece for the period of 2019-2023, and published by a single publisher. Nevertheless, other characteristics could give raise to the creation of a Dataset Series.

The DCAT-AP working group has investigated to find and express unique characteristics of Datasets that are members of a Dataset Series. Over time, during the exchanges it became clear that today no consensus exists on restricting the use of Dataset Series to a more limited use. Therefore the DCAT-AP working group has decided to retract the notion of a Dataset Member of a Dataset Series (a subclass of DCAT-AP Datasets). Profile builders may reintroduce this notion when they want to express specific constraints for that usage scope.

In case the connection is very strong or the result of an automated process, versioning terminology is a natural way to express the connection between the Datasets in the Dataset Series collection. For guidelines on expressing versioning information in DCAT-AP, consult the DCAT guidelines on versioning. As versioning is not always the most appropriate terminology, DCAT introduces properties (e.g. next, previous, inSeries, last, etc.) to interconnect the Dataset Series with its members and among the members themselves. It is recommended to use always these properties in combination with Dataset Series, and versioning when appropriate.

The membership in a Dataset Series may influence the (descriptive) metadata to highlight differences from other Datasets in the collection. For instance, a usual adaption is the addition of the release data in the title.

DCAT-AP ecosystem

European DCAT-AP profiles

• DCAT-AP: the base European profile DCAT-AP, https://semiceu.github.io/DCAT-AP/releases/3.0.1
• DCAT-AP HVD: DCAT-AP for High Value Datasets https://semiceu.github.io/DCAT-AP/releases/3.0.0-hvd/
• GeoDCAT-AP: DCAT-AP for geospatial datasets https://semiceu.github.io/GeoDCAT-AP/releases/3.0.0
• StatDCAT-AP: DCAT-AP for statistical datasets https://github.com/SEMICeu/StatDCAT-AP
• Health DCAT-AP: DCAT-AP for health datasets https://healthdcat-ap.github.io/
• Mobility DCAT-AP: DCAT-AP for mobility datasets https://mobilitydcat-ap.github.io/mobilityDCAT-AP/releases/1.1.0/index.html

Member State DCAT-AP profiles

High Value Datasets

In light of the growing importance of data, the European Commission has adopted an implementing regulation focused on high-value datasets on 21 December 2022 [[HVD]]. The ambition of the High-Value Datasets implementing regulation (HVD IR) is to improve the quality, the accessibility and use of a selective number of core datasets within the public sector. To achieve this the HVD IR sets, among others, requirements on the metadata associated with the disclosed datasets. These requirements can be expressed as additional constraints and usage notes on DCAT-AP and are collected in an annex dedicated to the implementation of the HVD IR [[DCAT-AP-HVD]].

Support for implementation

JSON-LD context file

The JSON-LD context file downloadable here.

Identifiers

Validation

The shapes that can be derived from the DCAT-AP specification have the following usage limitations. Shapes may check situations that never occur in the implemented data exchange and result in messages that confuse publishers, or they are too general and more detailed checks should be implemented. For example, the nature of a licence is recommended by DCAT-AP, but most DCAT-AP exchanges assume that based on the URI of the licence this information is known. Thus the check that a licence should have a licence type is considered often not relevant for the individual publishers. However they might support catalogue maintainers to guide individual publishers to use well-documented licence URIs. The case that a title cannot be an empty string and must have a value in a specific language are examples of more detailed constraints that are not present in the provided DCAT-AP SHACL shapes.

More on the validation and the provided SHACL shapes can be found in section [[[#validation-of-dcat-ap]]].

Validation of DCAT-AP

To support the check whether or not a catalogue satisfies the expressed constraints in this Application Profile, the constraints in this specification are expressed using SHACL [[shacl]]. Each constraint in this specification that could be converted into a SHACL expression has been included. As such this collection of SHACL expressions that can be used to build a validation check for a data exchange between two systems. A typical use case is the harvesting of one catalogue into another.

It is up to the implementers of the data exchange to define the validation they expect. Each data exchange happens within a context, and that context is beyond the SHACL expressions here.

For instance, it might be known that the exchanged data does not contain the details of the organisations, as they are all uniquely identified by a deferenceable URI. In such case, the rules regarding the check of the mandatory existence of a name for each organisation is probably not relevant. A strict execution of the DCAT-AP SHACL expressions will raise errors, although the data is available through a different channel. In this example, it is valid to not include this check in this validation step.

The example illustrates that in order to get the right user experience of a validation process, one has to consider what is actually transferred between the systems. And use the constraints that are in scope of the data exchange. To assist this process the SHACL expressions are grouped in distinct files, matching already typical validation setups.

• shapes.ttl: constraints concerning existence, domain and literal range, and cardinalities.
• range.ttl: constraints concerning object range.
• mdr-vocabularies.shape.ttl: constraints concerning the usage of controlled vocabularies.

• shapes_recommended.ttl: constraints concerning existence of recommended properties.

• imports.ttl: imports the vocabulary knowledge.
• mdr_imports.ttl: imports the recommended codelists.

In addition to these SHACL shapes extending the approach from DCAT-AP 2.x, the constraints are also provided in a new design where each constraint has got a unique identifier. This new shape is addressing the request for having a design in which DCAT-AP implementers could cherry pick the constraints they want. The shape is downloadable here.

Example Dataset Series

This section describes the use of Dataset Series via examples. All examples are fictitious and are created to facilitate the understanding of Dataset Series. This section complements the usage explained in section Dataset series in DCAT 3 [[vocab-dcat-3]], because it illustrates the decision making process when switching from publishing a single Dataset to a Dataset Series.

Consider the dataset 'Bee population in Greece" published by the Greek Environment Agency. Users can download the data as a CSV or an RDF file.

To see the evolution in the bee population, the Greek Environment Agency publishes an update each year. This update can be reflected in the DCAT-AP description in various ways.

The first option is to update and renew the metadata each update. For instance, the temporal coverage changes from the period "2020 - 2022" to "2020 - 2023".

This solution has as benefit that new information is shared with a minimal effort of publisher. However, this approach removes the previous state of affairs from the metadata and also does not reflect the aggregation and publishing effort for the dataset in the metadata. In some cases this is a desired outcome.

The alternative option is the creation of a new dataset description. Often this choice is a voluntary decision, however it might be enforced by legislation. Legislation can impose to share an audited dataset reflecting the state of affairs on a reference date.

The creation of new dataset also impacts the metadata, as the generic title "Bee population" cannot be maintained for both. A publisher must perform a minimal disambiguation effort to make the two datasets distinct.

Independent of the metadata descriptions, a core decision to take is whether the dataset BeePopulation2022 is also contained in the dataset BeePopulation2023. Either case is valuable and seen in practice.

However, without expressing any relationships between the datasets this approach has a major drawback: a search on the catalogue for bee population will result in several variants of the same dataset. The catalogue user is not provided with any guidance to select the most recent or most appropriate one. To address that use case, Dataset Series come into the picture. Dataset Series offer a structured approach to order or organise the datasets by the publisher for the catalogue user.

To support the data portal users, the Greek Environment Agency starts managing a Dataset Series BeePopulation.

As the dataset BeePopulation2023 is more recent than BeePopulation2022, a navigation ordering is added.

In most cases, the navigation is a single chain sequence, although tree organised structures are not prohibited.

Similarly as the introduction of a new dataset description, the introduction of a Dataset Series impacts the metadata of the involved datasets.

For instance the notion of the update frequency becomes more precise. In case of a single dataset description, the statement, in the example below, means that the dataset is updated each year.

So each year the actual data to which this dataset provides access to changes, and the most recent information can be found via this metadata.

However; when the publisher will introduce a new dataset description each year, the update frequency of an existing dataset is none. Namely it will not be updated. Without a Dataset Series notion the information of the yearly update is lost.

Users expect that metadata of Dataset Series is coherent with the collection of datasets it refers to. Up to 2022, the Greek Environment Agency only had the ability to provide the information for the region of Thessaloniki, and shared that via the spatial coverage.

From 2023 onwards, data for the region of Athens is also provided. As a consequence, not only the spatial coverage of the new dataset BeePopulation2023 will reflect that, but also the one of the Dataset Series. Namely the geographical coverage of a dataset series will cover the union of all places of each dataset associated with the Dataset Series.

A similar coherency concern applies to the temporal coverage of datasets. Publisher are advised to verify the properties title and description to match any reference to temporal and geographical coverage information with the properties.

The metadata descriptions of a Dataset Series provide insight in how this collection of datasets evolves. The release date of the Dataset Series BeePopulation correspond to the moment the collection has been issued for the first time. In many cases this moment corresponds to the earliest release date of the dataset in the collection. In the example, the Dataset BeePopulation2022 was first issued on 2022-04-01 while shortly after, the Dataset Series was created.

This relationship between the issuing date and the oldest dataset in the collection can reduce over time. The Greek Environment Agency and the National History Archives start to collaborate on the use and production of honey. That project results in new datasets on bee populations predating 2022. To make that information visible, the Dataset Series collection is updated with these historic datasets.

The modification date of the Dataset Series is the moment the collection has changed. The addition of the historic datasets changes the collection, and thus the modification date of the Dataset Series is set to 2023-06-01.

To make the datasets easier to use, the Greek Environment Agency introduces a new API platform. Via this API platform the datasets are not only downloadable in tabular format, but also in the international Biodiversity JSON format.

To support the users even more, the API platform is extended with a possibility to get life counts of the bee population. This means that there is a dataset which data is updated very frequently, eg. every second. In that case providing a modification date for the dataset becomes a technical challenge, but also it reduces it value. Namely one gets the data at query time.

Thus to conclude the example of maintaining a Dataset Series, the Greek Environment Agency has created a Dataset Series about the Bee population in Greece. It contains the dataset with actual life data, the yearly snapshots for 2023 and 2022 and also the historic data of 2000 maintained by the National History Archives.

Inverse properties

To harmonise the exchange of DCAT based data and reduce the need for inference engines, W3C DCAT imposes implementation guidelines expressing a preference in use. The guidelines state which property must be used while the other, called the inverse, may be used. In a proper DCAT profile the inverse property cannot be used to replace completely the property.

In line with these guidelines with the objective to reduce the burden of implementers DCAT-AP will follow the W3C guidelines in a more strict way. When a property is subject to this W3C guideline then DCAT-AP will include only the use of this property and not the inverse. Implementers are still free to use the inverse to optimise their data management.

As note to this discussion: when exchanging DCAT as an RDF graph the impact of the directionality is reduced because

• RDF statements are non-ordered and grouping according to directionality is more for human readibility than technically enforced.
• SPARQL allows to query in any direction simply by switching the subject and object variables.