@@ -21,13 +27,14 @@ A series of tutorials giving you an overview on how to store or convert your dat
- [Converting your data to NeXus](tutorial/converting-data-to-nexus.md)
- [Uploading NeXus data to NOMAD](tutorial/nexus-to-nomad.md)
+- [Troubleshooting guide](tutorial/troubleshooting.md)
### How-to guides
-How-to guides provide step-by-step instructions for a wide range of tasks:
+How-to guides provide step-by-step instructions for a wide range of tasks.
- [Writing an application definition](how-tos/writing-an-appdef.md)
- [Storing data in multiple application definitions](how-tos/using-multiple-appdefs.md)
@@ -36,18 +43,22 @@ How-to guides provide step-by-step instructions for a wide range of tasks:
- [Representing experimental geometries](how-tos/transformations.md)
- [Using pynxtools test framework](how-tos/using-pynxtools-test-framework.md)
-
### Learn
-An introduction to NeXus and its design principles.
+#### An introduction to NeXus and its design principles
- [An introduction to NeXus](learn/nexus-primer.md)
- [Rules for storing data in NeXus](learn/nexus-rules.md)
- [The concept of multiple application definitions](learn/multiple-appdefs.md)
+
+#### pynxtools
+
+- [Data conversion in `pynxtools`](learn/dataconverter-and-readers.md)
+- [NeXus verification in `pynxtools`](learn/nexus-verification.md)
- [The MultiFormatReader as a reader superclass](learn/multi-format-reader.md)
@@ -55,15 +66,22 @@ An introduction to NeXus and its design principles.
### Reference
+#### NeXus definitions
+[Here](reference/definitions.md), you find the detailed list of application definitions and base classes and their respective fields.
+
+Or go directly to the [official NIAC](https://manual.nexusformat.org/classes/index.html)
+ or [latest FAIRmat](https://fairmat-nfdi.github.io/nexus_definitions/) definitions.
+
+#### pynxtools
+
`pynxtools` has a number of command line tools that can be used to convert data and verify NeXus files. You can more information about the
API [here](reference/cli-api.md).
-Within FAIRmat, we maintain a number of reader plugins for different experimental techniques. You can find more information [here](reference/plugins.md).
+Within FAIRmat, we maintain a number of pynxtools readers as well as reader plugins for different experimental techniques. Here you can find more information:
-[Here](reference/definitions.md), you find the detailed list of application definitions and base classes and their respective fields.
+- [Built-in pynxtools readers](reference/built-in-readers.md)
+- [FAIRMat-suppored pynxtools plugins](reference/plugins.md)
-Or go directly to the [official NIAC](https://manual.nexusformat.org/classes/index.html)
- or [latest FAIRmat](https://fairmat-nfdi.github.io/nexus_definitions/) definitions.
diff --git a/docs/learn/dataconverter-and-readers.md b/docs/learn/dataconverter-and-readers.md
new file mode 100644
index 000000000..234edb68f
--- /dev/null
+++ b/docs/learn/dataconverter-and-readers.md
@@ -0,0 +1,75 @@
+# Data conversion in pynxtools
+One of the main motivations for pynxtools is to develop a tool for combining various instrument output formats and electronic lab notebook (ELN) into an [HDF5](https://support.hdfgroup.org/HDF5/) file according to [NeXus application definitions](https://fairmat-nfdi.github.io/nexus_definitions/classes/index.html).
+
+The `dataconverter` API in pynxtools provides exactly that: it converts experimental data to NeXus/HDF5 files based on any provided [NXDL schemas](https://manual.nexusformat.org/nxdl.html#index-1).
+
+The dataconverter has essentially three functionalities:
+
+1. read in experimental data using ```readers```
+2. validate the data and metadata against any of the NeXus application definitions
+3. write a valid NeXus/HDF5 file
+
+For step 1, a set of readers has been which the converter calls to accomplish this task for a specific set of application definition (NXDL XML file) plus a set of experiment/method-specific file(s). These files can be files in a proprietary format, or of a certain format used in the respective scientific community, or text files. Only in combination, these files hold all the required pieces of information which the application definition demands and which are thus required to make a NeXus/HDF5 file compliant. Users can store additional pieces of information in an NeXus/HDF5 file. In this case readers will issue a warning that these data are not properly documented from the perspective of NeXus.
+
+There exists two different subsets of readers:
+
+1. [Built-in readers](../reference/built-in-readers.md), which are implemented directly in pynxtools and are typically used either as superclasses for new reader implementations or for generic reading purposes not directly related to any specific technique.
+2. [Reader plugins](../reference/plugins.md) for `pynxtools, which are used for reading data of specific experimental techniques and are typically available as their own Python packages.
+
+## Matching to NeXus application definitions
+
+The purpose of the dataconverter is to create NeXus/HDF5 files with content that matches a specific NeXus application definition. Such application definitions are useful for collecting a set of pieces of information about a specific experiment in a given scientific field. The pieces of information are metadata and numerical data. The application definition is used to provide these data in a format that serves a data delivery contract: The HDF5 file, or so-called NeXus file, delivers all those pieces of information which the application definition specifies. Required and optional pieces of information are distinguished. NeXus classes can recommend the inclusion of certain pieces of information. Recommended data are essentially optional. The idea is that flagging these data as recommended motivates users to collect them but does not require to write dummy
+or nonsense data if the user is unable to collect recommended data.
+
+## Getting started
+
+Each of the built-in reader comes with the main `pynxtools` package, therefore they are avaible after pip installation:
+```console
+user@box:~$ pip install pynxtools
+```
+
+The different FAIRmat-supported plugins can be installed together with pynxtools by passing the name of the plugin as an extra to the pip install call. For example, for the `pynxtools-mpes` plugin:
+```console
+pip install pynxtools[mpes]
+```
+
+In addition, it is also possible to install all of the pynxtools reader plugins which are maintained by FAIRmat by passing the `[convert]` extra to the pip install call:
+
+```console
+pip install pynxtools[convert]
+```
+
+## Usage
+See [here](../reference/cli-api.md#data-conversion) for the documentation of the `dataconverter` API.
+
+### Use with multiple input files
+
+```console
+user@box:~$ dataconverter --nxdl nxdl metadata data.raw otherfile
+```
+
+### Merge partial NeXus files into one
+
+```console
+user@box:~$ dataconverter --nxdl nxdl partial1.nxs partial2.nxs
+```
+
+### Map an HDF5/JSON/(Python Dict pickled in a pickle file)
+
+```console
+user@box:~$ dataconverter --nxdl nxdl any_data.hdf5 --mapping my_custom_map.mapping.json
+```
+
+You can find actual examples with data files at [`examples/json_map`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/examples/json_map/).
+
+
+## Example data for testing and development purposes
+
+Before using your own data we strongly encourage you to download a set of open-source test data for testing the plug-ins. For this purpose pynxtools comes with a tests directory with a data/dataconverter sub-directory including reader-specific jupyter-notebook examples. These examples can be used for downloading test data and use specific readers as a standalone converter to translate given data into a NeXus/HDF5 file.
+
+Once you have practised with these tools how to convert these examples, feel free to use the tools for converting your own data. You should feel invited to contact the respective corresponding author(s) of each reader if you run into issues with the reader or feel there is a necessity to include additional data into the NeXus file for the respective application.
+
+We are looking forward for learning from your experience and see the interesting use cases.
+You can find the contact persons in the respective README.md of each reader.
+
+You can read specific README's of the readers and find usage examples [here](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/examples/).
\ No newline at end of file
diff --git a/docs/learn/multi-format-reader.md b/docs/learn/multi-format-reader.md
index 56f9872e8..40efc33ef 100644
--- a/docs/learn/multi-format-reader.md
+++ b/docs/learn/multi-format-reader.md
@@ -1,14 +1,14 @@
# The MultiFormatReader as a reader superclass
-There are three options for building a new pynxtools reader:
+There are three options for building a new `pynxtools` reader:
1. build the reader from scratch
2. inherit and extend the [**`BaseReader`**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/dataconverter/readers/base/reader.py)
3. inherit and extend the [**`MultiFormatReader`**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/dataconverter/readers/multi/reader.py)
-While option 1 is generally not recommended, inheriting and extending the `BaseReader` has traditionally been the default solution for all existing pynxtools readers and reader plugins. The `BaseReader`, which is an abstract base class, has an essentially empty ```read``` function and is thus only helpful for implementing the correct input/ouput design of the ```reader```function of any reader which is implemented off of it.
+While option 1 is generally not recommended, inheriting and extending the `BaseReader` has traditionally been the default solution for all existing pynxtools readers and reader plugins. The `BaseReader`, which is an abstract base class, has an essentially empty ```read``` function and is thus only helpful for implementing the correct input/output design of the ```read``` function of any reader which is implemented off of it.
-While building on the ```BaseReader``` allows for the most flexibility, in most cases it is desirable to implement a reader that can read in multiple file formats and then populate the template based on the read data. For this purpose, `pynxtools` has the [**`MultiFormatReader`**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/dataconverter/readers/multi/reader.py), which can be readily extended for any new data.
+While building on the ```BaseReader``` allows for the most flexibility, in most cases it is desirable to implement a reader that can read in multiple file formats and then populate the NeXus file based on the read data, in compliance with a NeXus application definition. For this purpose, `pynxtools` has the [**`MultiFormatReader`**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/dataconverter/readers/multi/reader.py), which can be readily extended for any new data.
Here, we will explain the inner workings of the `MultiFormatReader`. Note that there is also a [how-to guide](../how-tos/use-multi-format-reader.md) on how to implement a new reader off of the `MultiFormatReader` using a concrete example. In case you simply want to use the `MultiFormatReader` without understanding its inner logic, we recommend you start there.
diff --git a/docs/learn/nexus-rules.md b/docs/learn/nexus-rules.md
index 6d8bf8700..aa4a0133b 100644
--- a/docs/learn/nexus-rules.md
+++ b/docs/learn/nexus-rules.md
@@ -1,7 +1,5 @@
# Rules for storing data in NeXus
-!!! danger "Work in progress"
-
There are several rules which apply for storing single data items in NeXus. There exists a [summary](https://manual.nexusformat.org/datarules.html) in the NeXus documentation outlining most of these rules. However, this explanation is not exhaustive and thus, we have compiled here additional information.
diff --git a/docs/learn/nexus-verification.md b/docs/learn/nexus-verification.md
new file mode 100644
index 000000000..9c8c8488c
--- /dev/null
+++ b/docs/learn/nexus-verification.md
@@ -0,0 +1,45 @@
+# NeXus verification
+!!! info "Work in progress"
+
+One of the main advantages of using pynxtools is that it comes with its own verification. That is, it can be used to verify that a given NeXus/HDF5 file is compliant with a NeXus application definition.
+
+## As part of the dataconverter
+During [data conversion](./dataconverter-and-readers.md), before writing the HDF5 file, the data is first checked against the provided application definition.
+
+
+
+## read-nexus: NeXus file reader and debugger
+
+This utility outputs a debug log for a given NeXus file by annotating the data and metadata entries with the schema definitions from the respective NeXus base classes and application definitions to which the file refers to. See [here](reference/cli-api.html#verify-nexus) for the API documentation.
+
+The following example dataset can be used to test the `read_nexus` module: [src/pynxtools/data/201805_WSe2_arpes.nxs](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/data/201805_WSe2_arpes.nxs).
+
+This is an angular-resolved photoelectron spectroscopy (ARPES) dataset and it is formatted according to
+the [NXarpes application definition of NeXus](https://manual.nexusformat.org/classes/applications/NXarpes.html#nxarpes).
+
+### Using a different set of NeXus definitions
+The environmental variable called "NEXUS_DEF_PATH" can be set to
+a directory, which contains the NeXus definitions as XML files. If this environmental
+variable is not defined, the module will use the definitions in its bundle._
+
+An environmental variable can be set as follows:
+
+```
+export 'NEXUS_DEF_PATH'=