{ "cells": [ { "cell_type": "markdown", "id": "9cb473cc-4590-4820-bd51-47cfb26f056e", "metadata": {}, "source": [ "# HDF5 and ontologies\n", "\n", "HDF5 itself is considered self-describing due to the ability to store metadata (attributes) with raw data. However, this is only the prerequisite. Especially, achieving (easy) re-usability, requires standardized metadata, which is publically defined and accessible. This means, that data must be describable with persistent identifiers, as known from linked data solutions.\n", "\n", "One solution of describing data is using controlled vocabularies or even better ontologies. In fact, an [ontology exists](http://purl.allotrope.org/ontologies/hdf5/1.8#), which allows describing the structural content of an HDF5 file (groups, datasets, attributes, properties etc.). The ``h5rdmtoolbox`` has implemented a conversion function, translating an HDF5 into a JSON-LD file. This is outlined here." ] }, { "cell_type": "code", "execution_count": 1, "id": "5815196c-e3be-44d8-a1f8-07e563ce7dc2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Failed to import module h5tbx\n" ] } ], "source": [ "import h5rdmtoolbox as h5tbx" ] }, { "cell_type": "markdown", "id": "cf5946b6-53f5-46a6-8365-1b90f2f82eb4", "metadata": {}, "source": [ "Let's create a sample HDF5 file first:" ] }, { "cell_type": "code", "execution_count": 2, "id": "01fcc4c6-c564-4094-a4d4-bf4a9985c477", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "
\n", "\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "with h5tbx.File(mode='w') as h5:\n", " h5.create_dataset('test_dataset', shape=(3, ))\n", " grp = h5.create_group('grp')\n", " sub_grp = grp.create_group('Fan')\n", " sub_grp.create_dataset('D3', data=300)\n", " sub_grp['D3'].attrs['units', 'http://w3id.org/nfdi4ing/metadata4ing#hasUnits'] = 'mm'\n", " sub_grp['D3'].rdf['units'].object = 'https://qudt.org/vocab/unit/MilliM'\n", " sub_grp['D3'].attrs['standard_name', 'https://matthiasprobst.github.io/ssno/#standard_name'] = 'blade_diameter3'\n", " h5.dump(False)" ] }, { "cell_type": "markdown", "id": "d94305c3-914f-4396-8650-ca74cd34a764", "metadata": {}, "source": [ "## Dump the semantic metadata to JSON-LD format\n", "\n", "The semantic metadata is stored in the RDF dictionaries of the HDF5 file, which the `h5rdmtoolbox` can work with. Call `h5tbx.jsonld.dumps()` to extract it:" ] }, { "cell_type": "code", "execution_count": 3, "id": "0e2765c8-79ec-4116-96cb-55ffca7d8ddc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\n", " \"@context\": {\n", " \"hdf5\": \"http://purl.allotrope.org/ontologies/hdf5/1.8#\",\n", " \"m4i\": \"http://w3id.org/nfdi4ing/metadata4ing#\",\n", " \"schema\": \"http://schema.org/\",\n", " \"ssno\": \"https://matthiasprobst.github.io/ssno/#\"\n", " },\n", " \"@graph\": [\n", " {\n", " \"@id\": \"_:N1\",\n", " \"@type\": \"hdf5:File\",\n", " \"hdf5:rootGroup\": {\n", " \"@id\": \"_:N0\",\n", " \"@type\": \"hdf5:Group\",\n", " \"hdf5:member\": [\n", " {\n", " \"@id\": \"_:N2\",\n", " \"@type\": \"hdf5:Group\",\n", " \"hdf5:member\": {\n", " \"@id\": \"_:N3\",\n", " \"@type\": \"hdf5:Group\",\n", " \"hdf5:member\": {\n", " \"@id\": \"_:N4\",\n", " \"@type\": \"hdf5:Dataset\",\n", " \"hdf5:attribute\": [\n", " {\n", " \"@id\": \"_:N5\",\n", " \"@type\": \"hdf5:Attribute\",\n", " \"hdf5:name\": \"standard_name\",\n", " \"hdf5:value\": \"blade_diameter3\"\n", " },\n", " {\n", " \"@id\": \"_:N6\",\n", " \"@type\": \"hdf5:Attribute\",\n", " \"hdf5:name\": \"units\",\n", " \"hdf5:value\": \"mm\"\n", " }\n", " ],\n", " \"hdf5:datatype\": \"H5T_INTEGER\",\n", " \"hdf5:dimension\": 0,\n", " \"hdf5:name\": \"/grp/Fan/D3\",\n", " \"hdf5:size\": 1,\n", " \"hdf5:value\": {\n", " \"@id\": \"https://qudt.org/vocab/unit/MilliM\"\n", " },\n", " \"m4i:hasUnits\": {\n", " \"@id\": \"https://qudt.org/vocab/unit/MilliM\"\n", " },\n", " \"ssno:standard_name\": \"blade_diameter3\"\n", " },\n", " \"hdf5:name\": \"/grp/Fan\"\n", " },\n", " \"hdf5:name\": \"/grp\"\n", " },\n", " {\n", " \"@id\": \"_:N7\",\n", " \"@type\": \"hdf5:Dataset\",\n", " \"hdf5:datatype\": \"H5T_FLOAT\",\n", " \"hdf5:dimension\": 1,\n", " \"hdf5:name\": \"/test_dataset\",\n", " \"hdf5:size\": 3\n", " }\n", " ],\n", " \"hdf5:name\": \"/\"\n", " }\n", " }\n", " ]\n", "}\n" ] } ], "source": [ "print(h5tbx.jsonld.dumps(h5.hdf_filename,\n", " indent=2,\n", " context={'schema': 'http://schema.org/',\n", " \"ssno\": \"https://matthiasprobst.github.io/ssno/#\",\n", " \"m4i\": \"http://w3id.org/nfdi4ing/metadata4ing#\"}))" ] }, { "cell_type": "markdown", "id": "88f8984e-083c-44e7-b6a0-983f9c6a85c7", "metadata": {}, "source": [ "## Dump the structural metadata to JSON-LD format\n", "\n", "The structural or organizational metadata is the HDF5 internal layout, like groups, datasets, attributes and their properties including their relations:" ] }, { "cell_type": "code", "execution_count": 4, "id": "a36d5ec7-e7e5-4ead-9401-be05cf94552a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"@context\": {\"hdf5\": \"http://purl.allotrope.org/ontologies/hdf5/1.8#\", \"units\": \"http://w3id.org/nfdi4ing/metadata4ing#hasUnits\"}, \"@graph\": [{\"@id\": \"_:N9\", \"@type\": \"hdf5:File\", \"hdf5:rootGroup\": {\"@id\": \"_:N8\", \"@type\": \"hdf5:Group\", \"hdf5:member\": [{\"@id\": \"_:N10\", \"@type\": \"hdf5:Group\", \"hdf5:member\": {\"@id\": \"_:N11\", \"@type\": \"hdf5:Group\", \"hdf5:member\": {\"@id\": \"_:N12\", \"@type\": \"hdf5:Dataset\", \"hdf5:attribute\": [{\"@id\": \"_:N13\", \"@type\": \"hdf5:Attribute\", \"hdf5:name\": \"standard_name\", \"hdf5:value\": \"blade_diameter3\"}, {\"@id\": \"_:N14\", \"@type\": \"hdf5:Attribute\", \"hdf5:name\": \"units\", \"hdf5:value\": \"mm\"}], \"hdf5:datatype\": \"H5T_INTEGER\", \"hdf5:dimension\": 0, \"hdf5:name\": \"/grp/Fan/D3\", \"hdf5:size\": 1, \"hdf5:value\": {\"@id\": \"https://qudt.org/vocab/unit/MilliM\"}, \"https://matthiasprobst.github.io/ssno/#standard_name\": \"blade_diameter3\", \"units\": {\"@id\": \"https://qudt.org/vocab/unit/MilliM\"}}, \"hdf5:name\": \"/grp/Fan\"}, \"hdf5:name\": \"/grp\"}, {\"@id\": \"_:N15\", \"@type\": \"hdf5:Dataset\", \"hdf5:datatype\": \"H5T_FLOAT\", \"hdf5:dimension\": 1, \"hdf5:name\": \"/test_dataset\", \"hdf5:size\": 3}], \"hdf5:name\": \"/\"}}]}\n" ] } ], "source": [ "hdf_jsonld = h5tbx.dump_jsonld(h5.hdf_filename, skipND=None)\n", "print(hdf_jsonld)" ] }, { "cell_type": "markdown", "id": "e7e25ff7-049d-4593-9881-24558fe9e807", "metadata": {}, "source": [ "## Query the HDF-JSONLD file\n", "\n", "The obtained JSON-LD file can be used to search for specific information. In the below example, all datasets with their sizes are extracted:" ] }, { "cell_type": "code", "execution_count": 5, "id": "254527e3-6229-4887-9666-9dc3921aab58", "metadata": {}, "outputs": [], "source": [ "sparql_query = \"\"\"PREFIX rdf: \n", "PREFIX hdf5: \n", "\n", "SELECT ?ds_name ?ds_size\n", "WHERE {\n", " ?group rdf:type hdf5:Dataset .\n", " ?group hdf5:name ?ds_name .\n", " ?group hdf5:size ?ds_size .\n", "}\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 6, "id": "177fe328-d61e-444c-9b2e-d4678142bff2", "metadata": {}, "outputs": [], "source": [ "import rdflib\n", "g = rdflib.Graph()\n", "g.parse(data=hdf_jsonld, format='json-ld')\n", "results = g.query(sparql_query)" ] }, { "cell_type": "code", "execution_count": 7, "id": "1a066906-ac07-4b45-b1aa-2babbf1a4522", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{rdflib.term.Variable('ds_name'): rdflib.term.Literal('/grp/Fan/D3'), rdflib.term.Variable('ds_size'): rdflib.term.Literal('1', datatype=rdflib.term.URIRef('http://www.w3.org/2001/XMLSchema#integer'))}\n", "{rdflib.term.Variable('ds_name'): rdflib.term.Literal('/test_dataset'), rdflib.term.Variable('ds_size'): rdflib.term.Literal('3', datatype=rdflib.term.URIRef('http://www.w3.org/2001/XMLSchema#integer'))}\n" ] } ], "source": [ "for b in results.bindings:\n", " print(b)" ] }, { "cell_type": "code", "execution_count": 8, "id": "16d0a2b3-74c6-4b52-9f59-014b8e2a3158", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ds_nameds_size
0/grp/Fan/D31
1/test_dataset3
\n", "
" ], "text/plain": [ " ds_name ds_size\n", "0 /grp/Fan/D3 1\n", "1 /test_dataset 3" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# convert results to dataframe:\n", "import pandas as pd\n", "df = pd.DataFrame(results.bindings)\n", "df" ] }, { "cell_type": "code", "execution_count": null, "id": "03aa0c67-8db4-4e98-9d40-d88785f7f800", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.18" } }, "nbformat": 4, "nbformat_minor": 5 }