\n",
"
"
],
"text/plain": [
"<xarray.DataArray 'vel' (time: 21)>\n",
"0.0 0.6745 0.9959 0.7962 0.1797 -0.5308 ... -0.6615 0.01737 0.6872 0.9973 0.7855\n",
"Coordinates:\n",
" * time (time) float64 0.0 0.03927 0.07854 0.1178 ... 0.7069 0.7461 0.7854\n",
"Attributes:\n",
" long_name: air velocity in pipe\n",
" units: m/s\n",
"\n",
"
"
],
"text/plain": [
"- \n",
"
- \n",
" \n",
" \n",
" \n",
"
- \n",
"
- \n",
" \n",
" \n",
" (21) [float64]\n",
"
- long_name: measurement time
- units: s \n", "
- \n",
" \n",
" \n",
" (time: 21) [float64]\n",
"
- long_name: air velocity in pipe
- units: m/s \n", "
\n",
"
\n",
"\n",
"
"
],
"text/plain": [
"- \n",
"
- \n",
" \n",
" \n",
" \n",
"
- \n",
"
\n",
"
\n",
"\n",
"
"
],
"text/plain": [
"- \n",
"
- \n",
" \n",
" \n",
" \n",
"
- \n",
"
- \n",
"
- \n",
" \n",
" \n",
" \n",
"
- first_name
![\"IRI_ICON\"](\"https://github.com/matthiasprobst/h5RDMtoolbox/blob/dev/h5rdmtoolbox/data/iri_icon.png?raw=true\")
http://xmlns.com/foaf/0.1/firstName: Matthias - orcid http://w3id.org/nfdi4ing/metadata4ing#orcidId:
![\"ORCID](\"https://info.orcid.org/wp-content/uploads/2019/11/orcid_16x16.png\")
https://orcid.org/0000-0001-8729-0482 \n",
"
\n",
" - first_name
\n",
" - \n",
" \n",
" \n",
" \n",
"
\n", "The module *layouts* makes use of the database solution for HDF5 files. The idea is, that we should be able to formulate our expectations/specifications in the form of a query. For more detailed information, see [here](https://h5rdmtoolbox.readthedocs.io/en/latest/layouts.html). So we write down our queries, which we expect to find HDF5 objects in a file, when we validate one in the future.\n", "\n", "Let's design a simple one, which requires all datasets to have the attribute \"units\":" ] }, { "cell_type": "code", "execution_count": 22, "id": "e4aae76a-0e2a-4adc-9522-6707cc080bed", "metadata": {}, "outputs": [], "source": [ "from h5rdmtoolbox.layout import Layout" ] }, { "cell_type": "code", "execution_count": 23, "id": "b9284d79-3462-45ef-b445-bdedf5a12387", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[LayoutSpecification(kwargs={'flt': {}, 'objfilter': 'dataset'})]" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lay = Layout()\n", "\n", "spec_all_dataset = lay.add(\n", " FileDB.find, # query function\n", " flt={},\n", " objfilter='dataset',\n", " n=None\n", ")\n", "\n", "# The following specification is added to the previous.\n", "# This will apply the query only on results found by the previous query\n", "spec_compression = spec_all_dataset.add(\n", " FileDB.find,\n", " flt={'units': {'$exists': True}}, # attribute \"units\" exists\n", " n=1\n", ")\n", "\n", "# we added one specification to the layout. let's check:\n", "lay.specifications # note, that the second specification is not shown, because it is part of the first one" ] }, { "cell_type": "code", "execution_count": 24, "id": "f3408c38-a67a-476f-818c-5ac139eeb9a6", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = lay.validate(vel_hdf_filename)\n", "res.is_valid()" ] }, { "cell_type": "code", "execution_count": 25, "id": "fe9731be-02c6-4a84-bfee-cdc3c9257ab6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Summary of layout validation\n", "+--------------------------------------+----------+--------+----------------------+---------------+-----------------------------------------+\n", "| id | called | flag | flag description | description | func |\n", "|--------------------------------------+----------+--------+----------------------+---------------+-----------------------------------------|\n", "| 1b900c80-2e76-45c7-8fdf-ac44fe95fa6b | True | 17 | SUCCESSFUL, OPTIONAL | | h5rdmtoolbox.database.hdfdb.filedb.find |\n", "| 3229534f-a3a2-44aa-8d45-13af20b9da8f | True | 1 | SUCCESSFUL | | h5rdmtoolbox.database.hdfdb.filedb.find |\n", "| 3229534f-a3a2-44aa-8d45-13af20b9da8f | True | 1 | SUCCESSFUL | | h5rdmtoolbox.database.hdfdb.filedb.find |\n", "+--------------------------------------+----------+--------+----------------------+---------------+-----------------------------------------+\n", "--> Layout is valid\n" ] } ], "source": [ "res.print_summary(exclude_keys=('kwargs', 'target_name', 'target_type'))" ] }, { "cell_type": "markdown", "id": "4724c12c-4b0b-40c3-9986-7f53c8bc2d37", "metadata": {}, "source": [ "The above layout successfully validate the file.\n", "\n", "Now, let's add the specification:\n", "- The file must have one dataset named \"pressure\".\n", "- The exact location within the file does not play a role.\n", "- This specific dataset must have the unit \"Pa\":\n", "- The shape of the dataset must be equal to (21, )" ] }, { "cell_type": "code", "execution_count": 26, "id": "298c4bea-4396-48a1-8514-40c761c4d099", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[LayoutSpecification(kwargs={'flt': {}, 'objfilter': 'dataset'}),\n", " LayoutSpecification(kwargs={'flt': {'$name': {'$regex': 'pressure'}, '$shape': (21,), 'units': 'Pa'}, 'objfilter': 'dataset'})]" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "lay.add(\n", " FileDB.find_one, # query function\n", " flt={'$name': {'$regex': 'pressure'}, \n", " '$shape': (21, ),\n", " 'units': 'Pa'},\n", " objfilter='dataset',\n", " n=1\n", ")\n", "lay.specifications" ] }, { "cell_type": "markdown", "id": "06d9a40a-90bf-469f-97b3-7e9fa0f5aa04", "metadata": {}, "source": [ "The validation now fails:" ] }, { "cell_type": "code", "execution_count": 27, "id": "7bfb6954-4b82-46a4-ba23-65f716524861", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-07-28_10:36:50,572 ERROR [core.py:330] Applying spec. \"LayoutSpecification(kwargs={'flt': {'$name': {'$regex': 'pressure'}, '$shape': (21,), 'units': 'Pa'}, 'objfilter': 'dataset'})\" failed due to not matching the number of results: 1 != 0\n" ] }, { "data": { "text/plain": [ "False" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = lay.validate(vel_hdf_filename)\n", "res.is_valid()" ] }, { "cell_type": "markdown", "id": "c7764cb0-cb3b-41cc-b681-ab2fd0d894ea", "metadata": {}, "source": [ "Let's add such a dataset:" ] }, { "cell_type": "code", "execution_count": 28, "id": "d537c483-9c2d-4361-9ed8-371df1574314", "metadata": {}, "outputs": [], "source": [ "with h5tbx.File(vel_hdf_filename, 'r+') as h5:\n", " h5.create_dataset('subgrp/pressure', shape=(21,), attrs={'units': 'Pa'})" ] }, { "cell_type": "markdown", "id": "b271037d-d0d4-482b-8ce0-beb6e7f0f0e6", "metadata": {}, "source": [ "And perform the validation again:" ] }, { "cell_type": "code", "execution_count": 29, "id": "8572d50e-9f4d-4c38-aa4e-d47a9b682c87", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = lay.validate(vel_hdf_filename)\n", "res.is_valid()" ] }, { "cell_type": "markdown", "id": "b575c1a3-f194-4959-9ebc-1fc960056e6c", "metadata": {}, "source": [ "Feel free to play with the layout specifications and the HDF5 file content. For sure, knowledge about performing queries with the used database is needed." ] }, { "cell_type": "markdown", "id": "894b7854-339f-4fa0-b1f2-03e74f77d07e", "metadata": {}, "source": [ "## Repositories\n", "\n", "Finally, we can publish our data. The toolbox has implemented an interface to [Zenodo](https://zenodo.org/). Using it with the sandbox (testing) environment requires an API TOKEN. For this, please provide the environment variable \"ZENODO_SANDBOX_API_TOKEN\":" ] }, { "cell_type": "code", "execution_count": 30, "id": "136c1e6f-9de1-4abf-a86e-cd61486242c0", "metadata": {}, "outputs": [], "source": [ "# %set_env ZENODO_SANDBOX_API_TOKEN=
\n", "As HDF5 files cannot be previewed in the repository web interface and HDF5 files may become very large, it is handy to upload a metadata file as an additional resource. This allows to preview the metadata content before downloading the larger HDF5 file.\n", "\n", "This functionality is given by the upload method `.upload_file`, which takes the optional parameter `metamapper`. Please provide a function which creates a file with metadata. The toolbox comes with a solution, that maps the HDF5 metadata content into a JSON-LD file:" ] }, { "cell_type": "code", "execution_count": 35, "id": "24029142-a2ff-4417-8aca-6bc614355abc", "metadata": {}, "outputs": [], "source": [ "from h5rdmtoolbox import jsonld" ] }, { "cell_type": "code", "execution_count": 36, "id": "ba875bb7-40da-4938-bb6c-74aeebce2fa6", "metadata": {}, "outputs": [], "source": [ "deposit.upload_file(filename=vel_hdf_filename, metamapper=jsonld.hdf2jsonld, skipND=1) # skipND is needed by hdf2jsonld" ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.19" } }, "nbformat": 4, "nbformat_minor": 5 }