Merge pull request #412 from FAIRmat-NFDI/405-move-readme-content-to-…

…documentation

Move all READMEs to docs, overhaul doc structure

README.md

@@ -10,9 +10,16 @@
 `pynxtools` is a tool designed for making your experimental data FAIR.
 It allows to develop ontologies and to create ontological instances based on the [NeXus format](https://www.nexusformat.org/).
 
+# Scope
+
+`pynxtools` is a parser for combining various instrument output formats and electronic lab notebook (ELN) formats into an [HDF5](https://support.hdfgroup.org/HDF5/) file according to NeXus application definitions.
+
+Additionally, the software can be used as a plugin in the research data management system NOMAD for
+making experimental data searchable and publishable. NOMAD is developed by the FAIRmat consortium which is a consortium of the German National Research Data Infrastructure (NFDI).
+
 # Installation
 
-It is recommended to use python 3.10 with a dedicated virtual environment for this package.
+It is recommended to use python 3.11 with a dedicated virtual environment for this package.
 Learn how to manage [python versions](https://github.com/pyenv/pyenv) and
 [virtual environments](https://realpython.com/python-virtual-environments-a-primer/).
 
@@ -28,36 +35,25 @@ You can also install the latest _development_ version with
 pip install git+https://github.com/FAIRmat-NFDI/pynxtools.git
 ```
 
-# Scope
+# Documentation
+Documentation can be found [here](https://fairmat-nfdi.github.io/pynxtools/).
 
-`pynxtools` (previously called `nexusutils`) is intended as a parser for combining various instrument output formats and electronic lab notebook (ELN) formats to an hdf5 file according to NeXus application definitions.
+# Repository structure
 
-Additionally, the software can be used as a plugin in the research data management system NOMAD for
-making experimental data searchable and publishable.
-NOMAD is developed by the FAIRMAT consortium, as a part of the German National Research Data Infrastructure
-(NFDI).
+The software tools are located inside [`src/pynxtools`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/src/pynxtools). They are shipped with unit tests located in [`tests`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/tests).
+Some examples from the scientific community are provided in [`examples`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/examples). They guide you through the process of converting instrument data into the NeXus standard and visualising the files' content.
 
-The software tools are located inside [`pynxtools`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/src/pynxtools) and they are
-shipped with unit tests located in [`tests`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/tests).
-Some examples with real datasets are provided in [`examples`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/examples).
-It guides you through the process of converting instrument raw
-data into the NeXus standard and visualising the files content.
+# NOMAD integration
 
-# Command line tools
+## Does this software require NOMAD or NOMAD OASIS ?
 
-- [**dataconverter**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/dataconverter/README.md): Creates compliant instances of NeXus/HDF5 files to [NeXus schemas](https://nexusformat.org).
-- [**read_nexus**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/nexus/README.md): Outputs a debug log for a given NeXus file.
-- [**generate_eln**](https://github.com/FAIRmat-NFDI/pynxtools/blob/master/src/pynxtools/eln_mapper/README.md): Outputs ELN files that can be used to add metadata to the dataconverter routine.
+No. The data files produced here can be uploaded to NOMAD. Therefore, this tool acts as the framework to design schemas and instances of data within the NeXus universe. It can, however, be used as a NOMAD plugin to parse nexus files, please see the section below for details.
 
-# NOMAD integration
+## How to use pynxtools with NOMAD
 
 To use pynxtools with NOMAD, simply install it in the same environment as the `nomad-lab` package.
-NOMAD will recognize pynxtools as a plugin automatically and offer automatic parsing of `.nxs` files
-and a schema for NeXus application definitions.
-pynxtools is already included in the NOMAD main deployment and NOMAD NeXus distribution images.
-
-# Documentation
-Documentation for the different tools can be found [here](https://fairmat-nfdi.github.io/pynxtools/).
+NOMAD will recognize pynxtools as a plugin automatically and offer automatic parsing of `.nxs` files. In addition, NOMAD will install a schema for NeXus application definitions.
+By default, `pynxtools` is already included in the NOMAD [production]https://nomad-lab.eu/prod/v1/gui/ and [staging](https://nomad-lab.eu/prod/v1/staging/gui/) deployments.
 
 # Contributing
 
@@ -97,14 +93,17 @@ python -m pytest -sv tests
 ## Run examples
 
 A number of examples exist which document how the tools can be used. For a standalone
-usage convenient jupyter notebooks are available for each tool. To use them jupyter
+usage convenient jupyter notebooks are available for each tool. To use these notebooks, jupyter
 and related tools have to be installed in the development environment as follows:
 
 ```shell
 python -m pip install jupyter
 python -m pip install jupyterlab
 python -m pip install jupyterlab_h5web
 ```
+# Troubleshooting
+
+Please check this [guide](https://fairmat-nfdi.github.io/pynxtools/tutorial/troubleshooting.html) for any issues you face with the tool. If you don't find a solution there, please make a new [Github Issue](https://github.com/FAIRmat-NFDI/pynxtools/issues/new?template=bug.yaml).
 
 # Questions, suggestions?
 
@@ -114,13 +113,4 @@ on how to build on this work, or to get your parser included into NOMAD, you can
 - Open an issue on the [pynxtools GitHub](https://github.com/FAIRmat-NFDI/pynxtools/issues)
 - Use our forums at [matsci.org](https://matsci.org/c/nomad/32)
 - Write to [support@nomad-lab.eu](mailto:support@nomad-lab.eu)
-- Contact directly the lead developers of the individual parsers.
-
-### Does this software require NOMAD or NOMAD OASIS ?
-
-No. The data files produced here can be uploaded to Nomad. Therefore, this acts like the framework to design schemas and instances of data within the NeXus universe. It can, however, be used as a NOMAD plugin to parse nexus files, please see the section above for details.
-
-# Troubleshooting
-
-Please check this [guide](TROUBLESHOOTING.md) for any issues you face with the tool. If you don't find a solution there, please make a new [Github Issue](https://github.com/FAIRmat-NFDI/pynxtools/issues/new?template=bug.yaml).
-
+- Contact directly the lead developers of the individual parsers.

docs/index.md

@@ -5,12 +5,18 @@ hide: toc
 # FAIRmat NeXus documentation
 
 <!-- A single sentence that says what the product is, succinctly and memorably -->
+Within [FAIRmat](https://www.fairmat-nfdi.eu/fairmat/), we are extending the [NeXus data format standard](https://www.nexusformat.org/) to support the FAIR data principles for experimental data in materials science (covering solid-state physics and the chemical physics of solids, as well as materials engineering). This is the documentation for both our contribution to the NeXus standard and for our tools for data conversion and verification.
 
 <!-- A paragraph of one to three short sentences, that describe what the product does. -->
+`pynxtools`, the main tool under development, provides a data converter that maps experimental data and metadata to the NeXus format, performing parsing, normalization, visualization, and ontology matching. It combines various instrument output formats and electronic lab notebook (ELN) formats to an HDF5 file according to NeXus application definitions. In addition, `pynxtools` can be used to validate and verify NeXus files.
 
 <!-- A third paragraph of similar length, this time explaining what need the product meets -->
+`pynxtools` offers scientists a convenient way to use the NeXus format and solves the challenge of unstructured and non-standardized data in experimental materials science. We consider this package useful for meeting the following FAIR principle as defined in [FAIR Principles: Interpretations and Implementation Considerations](https://direct.mit.edu/dint/article/2/1-2/10/10017/FAIR-Principles-Interpretations-and-Implementation): F2-4, I2-I3, and R1.
 
 <!-- Finally, a paragraph that describes whom the product is useful for. -->
+FAIRmat's contribution to the existing NeXus standard, together with the tools provided through `pynxtools`, enable scientists and research groups working with data, as well as helping communities implement standardized FAIR research data.
+
+Additionally, the software is used as a plugin in the research data management system [NOMAD](https://nomad-lab.eu/nomad-lab/) for making experimental data searchable and publishable. NOMAD is developed by the FAIRMAT consortium, as a part of the German National Research Data Infrastructure (NFDI).
 
 <div markdown="block" class="home-grid">
 <div markdown="block">
@@ -27,7 +33,7 @@ A series of tutorials giving you an overview on how to store or convert your dat
 
 ### How-to guides
 
-How-to guides provide step-by-step instructions for a wide range of tasks:
+How-to guides provide step-by-step instructions for a wide range of tasks.
 
 - [Writing an application definition](how-tos/writing-an-appdef.md)
 - [Storing data in multiple application definitions](how-tos/using-multiple-appdefs.md)
@@ -36,35 +42,47 @@ How-to guides provide step-by-step instructions for a wide range of tasks:
 - [Representing experimental geometries](how-tos/transformations.md)
 - [Using pynxtools test framework](how-tos/using-pynxtools-test-framework.md)
 
-
 </div>
 
 <div markdown="block">
 
 ### Learn
 
-An introduction to NeXus and its design principles.
+#### An introduction to NeXus and its design principles
 
 - [An introduction to NeXus](learn/nexus-primer.md)
 - [Rules for storing data in NeXus](learn/nexus-rules.md)
 - [The concept of multiple application definitions](learn/multiple-appdefs.md)
+
+#### pynxtools
+
+- [Data conversion in `pynxtools`](learn/dataconverter-and-readers.md)
+- [Validation of NeXus files](learn/nexus-validation.md)
 - [The MultiFormatReader as a reader superclass](learn/multi-format-reader.md)
 
 </div>
 <div markdown="block">
 
 ### Reference
 
-`pynxtools` has a number of command line tools that can be used to convert data and verify NeXus files. You can more information about the
-API [here](reference/cli-api.md).
-
-Within FAIRmat, we maintain a number of reader plugins for different experimental techniques. You can find more information [here](reference/plugins.md).
-
+#### NeXus definitions
 [Here](reference/definitions.md), you find the detailed list of application definitions and base classes and their respective fields.
 
 Or go directly to the [official NIAC](https://manual.nexusformat.org/classes/index.html)
  or [latest FAIRmat](https://fairmat-nfdi.github.io/nexus_definitions/) definitions.
 
+Note: To connect NeXus concepts with semantic web tools, efforts are underway to represent them using the [W3C Web Ontology Language (OWL)](https://www.w3.org/OWL/). See the [NeXusOntology](https://github.com/FAIRmat-NFDI/NeXusOntology) for more details.
+
+#### pynxtools
+
+`pynxtools` has a number of command line tools that can be used to convert data and verify NeXus files. You can find more information about the API [here](reference/cli-api.md).
+
+Within FAIRmat, we maintain a number of generic built-in pynxtools readers, together with reader plugins for different experimental techniques. Here you can find more information:
+
+- [Built-in pynxtools readers](reference/built-in-readers.md)
+- [FAIRMat-supported pynxtools plugins](reference/plugins.md)
+
+
 </div>
 </div>
 

docs/learn/dataconverter-and-readers.md

@@ -0,0 +1,74 @@
+# Data conversion in pynxtools
+One of the main motivations for pynxtools is to develop a tool for combining various instrument output formats and electronic lab notebook (ELN) into a file according to [NeXus application definitions](https://fairmat-nfdi.github.io/nexus_definitions/classes/index.html). 
+
+The `dataconverter` API in pynxtools provides exactly that: it converts experimental as well as simulation data, together with the results from analysis of such data, to NeXus files based on any provided [NXDL schemas](https://manual.nexusformat.org/nxdl.html#index-1). Here, we are using [HDF5](https://support.hdfgroup.org/HDF5/) as the serialization format.
+
+The dataconverter currently has essentially three functionalities:
+
+1. Read in experimental data using ```readers```
+2. Validate the data and metadata against a NeXus application definition of choice (i.e., check that the output data matches all existence, shape, and format constraints of application definition)
+3. Write a valid NeXus/HDF5 file
+
+A set of readers has been developed which the converter calls to read in a set of experiment/method-specific file(s) and for a specific set of application definitions (NXDL XML file). These data files can be in a proprietary format, or of a certain format used in the respective scientific community, or text files. Only in combination, these files hold all the required pieces of information which the application definition demands and which are thus required to make a NeXus/HDF5 file compliant. Users can store additional pieces of information in an NeXus/HDF5 file. In this case readers will issue a warning that these data are not properly documented from the perspective of NeXus.
+
+There exists two different subsets of readers:
+
+1. [Built-in readers](../reference/built-in-readers.md), which are implemented directly in pynxtools and are typically used either as superclasses for new reader implementations or for generic reading purposes not directly related to any specific technique.
+2. [Reader plugins](../reference/plugins.md) for `pynxtools, which are used for reading data of specific experimental techniques and are typically available as their own Python packages.
+
+## Matching to NeXus application definitions
+
+The purpose of the dataconverter is to create NeXus/HDF5 files with content that matches a specific NeXus application definition. Such application definitions are useful for collecting a set of pieces of information about a specific experiment in a given scientific field. The pieces of information are numerical and categorical (meta)data. The application definition is used to provide these data in a format that serves a data delivery contract: The HDF5 file, or so-called NeXus file, delivers all those pieces of information which the application definition specifies. Required and optional pieces of information are distinguished. NeXus classes can recommend the inclusion of certain pieces of information. Recommended data are essentially optional. The idea is that flagging these data as recommended motivates users to collect these, but does not require to write dummy or nonsense data if the recommended data is not available.
+
+## Getting started
+
+Each of the built-in readers comes with the main `pynxtools` package. Hence, they can be used after after pip installation:
+```console
+user@box:~$ pip install pynxtools
+```
+
+The different FAIRmat-supported plugins can be installed together with pynxtools by passing the name of the plugin as an extra to the pip install call. For example, for the `pynxtools-mpes` plugin:
+```console
+pip install pynxtools[mpes]
+```
+
+In addition, it is also possible to install all of the pynxtools reader plugins which are maintained by FAIRmat by passing the `[convert]` extra to the pip install call:
+
+```console
+pip install pynxtools[convert]
+```
+
+Note that in this case, the latest version of the plugin from PyPI is installed.
+
+## Usage
+See [here](../reference/cli-api.md#data-conversion) for the documentation of the `dataconverter` API.
+
+### Use with multiple input files
+
+```console
+user@box:~$ dataconverter metadata data.raw otherfile --nxdl nxdl --reader <reader-name>
+```
+
+### Merge partial NeXus files into one
+
+```console
+user@box:~$ dataconverter --nxdl nxdl partial1.nxs partial2.nxs
+```
+
+### Map an HDF5 file/JSON file
+
+```console
+user@box:~$ dataconverter --nxdl nxdl any_data.hdf5 --mapping my_custom_map.mapping.json
+```
+
+You can find actual examples with data files at [`examples/json_map`](https://github.com/FAIRmat-NFDI/pynxtools/tree/master/examples/json_map/).
+
+
+## Example data for testing and development purposes
+
+Before using your own data we strongly encourage you to download a set of open-source test data for testing the pynxtools readers andreader  plugins. For this purpose, pynxtools and its plugins come 
+with `examples` and `test` directories including reader-specific examples. These examples can be used for downloading test data and use specific readers as a standalone converter to translate given data into a NeXus/HDF5 file.
+
+Once you have practized with these tools how to convert these examples, feel free to use the tools for converting your own data. You should feel invited to contact the respective corresponding author(s) of each reader if you run into issues with the reader or feel there is a necessity to include additional data into the NeXus file for your respective application.
+
+We are looking forward to learning from your experience and learn from your use cases. You can find the contact persons in the respective README.md of each reader (plugin).