This repository contains the dataset and ontologies used for the paper:

@inproceedings{
    author = {Ding, Junsheng and Zhang, Haifan and Li, Weihang and Zhou, Liangwei and Perzylo, Alexander},
    title = {Knowledge-based Programming by Demonstration using semantic action models for industrial assembly},
    booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
    year = {2024},
    month = oct,
    address = {Abu Dhabi, UAE},
}

The exemplary LSTM network from Tensorflow is modified and utilized to recognize different grasp types.

The LSTM takes a [1, 63] input, which represents the flattened [3, 21] hand landmarks generated by Mediapipe hands.

The exemplary dataset of 5 different grasp types used during the evaluation is provided in /dataset. The code for training the LSTM is provided under /grasp_rocognition_lstm with a trained model grasp_recognition_lstm.h5 that was used during the experiment.

The grasp recognition network can be extended for more grasp types with additional dataset. A data recorder is provided, which subscribes the hand landmarks from each frame and can be controlled via ros parameters. During our dataset collection process, the operator hold certain objects in hand to perform a single grasp type.

The /ontologies/hand-action.ttl contains the ontology classes for representing hand actions, including grasp types, primitive actions, composite actions, etc. The product-specific actions are defined as skills and represented as defined classes with restrictions in action properties, which allows them to be automatically classified using OWL Reasoner, e.g., OWL DL2.

The /ontologies/programming-by-demonstration-result.ttl contains a minimal example of the recognized human actions within an assembly process. To populate this ontology, it can be hosted on a graph database, e.g. GraphDB, and accessed using the REST API.

This project is licensed under the Apache License 2.0.

The Apache License 2.0 is a permissive free software license written by the Apache Software Foundation. It allows you to use the software for any purpose, to distribute it, and to modify it. The license also provides an express grant of patent rights from the contributors to the users, ensuring that you have the freedom to use, modify, and distribute the software without fear of patent litigation. However, it also requires that any redistributed software or modified versions include a copy of the license and provide proper attribution to the original authors.

For more information, you can read the full text of the license at Apache License 2.0.