Add requirements files for python dependencies

This commit adds two separate requirements files that you can feed into pip for installing python dependencies. This makes it easier for contributors to install dependencies by having them isolated to a couple specific files, instead of having to reference build documentation.
ComplianceAsCode · May 5, 2023 · b80132d · b80132d
1 parent 5a228b6
commit b80132d
Show file tree

Hide file tree

Showing 4 changed files with 53 additions and 42 deletions.
diff --git a/.github/workflows/gate.yaml b/.github/workflows/gate.yaml
@@ -58,11 +58,11 @@ jobs:
       - name: Update the package repository
         run: apt-get update
       - name: Install Deps
-        run: apt-get install -y ansible-lint bats check cmake libopenscap8 libxml2-utils ninja-build python3-github python3-jinja2 python3-setuptools python3-pip python3-pytest python3-pytest-cov xsltproc
-      - name: Install deps python
-        run: pip3 install ruamel.yaml yamlpath pyyaml==5.4.1
+        run: apt-get install -y ansible-lint bats check cmake libopenscap8 libxml2-utils ninja-build python3-github python3-pip xsltproc
       - name: Checkout
         uses: actions/checkout@v2
+      - name: Install deps python
+        run: pip3 install -r requirements.txt -r test-requirements.txt
       - name: Build
         env:
           ADDITIONAL_CMAKE_OPTIONS: "-DSSG_ANSIBLE_PLAYBOOKS_PER_RULE_ENABLED=ON"
@@ -133,11 +133,11 @@ jobs:
       image: fedora:latest
     steps:
       - name: Install Deps
-        run: dnf install -y cmake make openscap-utils python3-pyyaml python3-setuptools python3-jinja2 bats python3-pytest python3-pytest-cov ansible python3-pip ShellCheck git python3-mypy python3-lxml
-      - name: Install deps python
-        run: pip install ruamel.yaml yamlpath types-openpyxl types-PyYAML xmldiff openpyxl openpyxl-stubs pandas cmakelint
+        run: dnf install -y cmake make openscap-utils python3-pyyaml bats ansible python3-pip ShellCheck git
       - name: Checkout
         uses: actions/checkout@v2
+      - name: Install deps python
+        run: pip install -r requirements.txt -r test-requirements.txt
       - name: Build
         run: |-
           ./build_product \
@@ -183,11 +183,11 @@ jobs:
       -   name: Run Updates
           run: dnf update -y
       -   name: Install Deps
-          run: dnf install -y cmake make openscap-utils python3-pyyaml python3-jinja2 bats python3-pytest python3-pytest-cov ansible python3-pip ShellCheck python-mypy git
-      -   name: Install deps python
-          run: pip install ruamel.yaml yamlpath types-openpyxl types-PyYAML openpyxl openpyxl-stubs pandas
+          run: dnf install -y cmake make openscap-utils bats ansible python3-pip ShellCheck git
       -   name: Checkout
           uses: actions/checkout@v2
+      -   name: Install deps python
+          run: pip install -r requirements.txt -r test-requirements.txt
       -   name: Build
           run: |-
             ./build_product \

diff --git a/docs/manual/developer/02_building_complianceascode.md b/docs/manual/developer/02_building_complianceascode.md
@@ -30,29 +30,45 @@ less ~/OpenSCAP/STARTGUIDE.md
 ## Installing build dependencies
 
 ### Required Dependencies
-On *Red Hat Enterprise Linux 7* make sure the packages `cmake`, `openscap-utils`,
-`PyYAML`, `python-jinja2`, `python-setuptools` and their dependencies are installed:
+On *Red Hat Enterprise Linux 7* make sure the following packages are installed:
 
 ```bash
-yum install cmake make openscap-utils openscap-scanner PyYAML python-jinja2
+yum install cmake make openscap-utils openscap-scanner
 ```
 
-On *Red Hat Enterprise Linux 8* and *Fedora* the package list is the same but python2 packages need to be replaced with python3 ones:
+On *Red Hat Enterprise Linux 8* and *Fedora* the package list but must also include python3:
 
 ```bash
-yum install cmake make openscap-utils openscap-scanner python3 python3-pyyaml python3-jinja2 python3-setuptools
+yum install cmake make openscap-utils openscap-scanner python3
 ```
 
 On *Ubuntu* and *Debian*, make sure the packages `libopenscap8`,
-`libxml2-utils`, `python3-jinja2`, `python3-yaml`, `python3-setuptools`, `xsltproc` and their dependencies are
-installed:
+`libxml2-utils`, `xsltproc` and their dependencies are installed:
 
 ```bash
-apt-get install cmake make libopenscap8 libxml2-utils ninja-build python3-jinja2 python3-yaml python3-setuptools xsltproc
+apt-get install cmake make libopenscap8 libxml2-utils ninja-build xsltproc
 ```
 
 IMPORTANT: Version `1.0.8` or later of `openscap-utils` is required to build the content.
 
+### Python Dependencies
+
+Some python dependencies must be installed through pip because they're not
+packaged by a distribution.
+
+Dependencies for developing are in `requirements.txt` and you can install them using:
+
+```bash
+$ pip3 install -r requirements.txt
+```
+
+Test dependencies are kept in a separate requirements file called
+`test-requirements.txt` and are installed using the same process:
+
+```bash
+$ pip3 install -r test-requirements.txt
+```
+
 ### Git (Clone the Repository)
 
 Install git if you want to clone the GitHub repository to get the
@@ -92,11 +108,9 @@ apt-get install bats
 
 ### xmldiff (Python unit tests)
 
-Install the  `xmldiff` and `lxml` packages to execute Python unit tests that use these packages.
+Install the `lxml` package to execute Python unit tests that use these packages.
 
 ```bash
-pip3 install xmldiff
-
 # Fedora/RHEL
 yum install python3-lxml
 
@@ -120,11 +134,9 @@ apt-get install yamllint ansible-lint
 
 ### Static Ansible Playbooks tests
 
-Install `yamlpath` and `pytest` to run tests cases that analyse the Ansible
+Install `pytest` to run tests cases that analyse the Ansible
 Playbooks' yaml nodes.
 ```bash
-pip3 install yamlpath
-
 # Fedora/RHEL
 yum install python3-pytest
 
@@ -145,14 +157,6 @@ yum install ninja-build
 apt-get install ninja-build
 ```
 
-### json2html (HTML Report Statistics)
-
-Install the `json2html` package if you want to generate HTML report statistics:
-
-```bash
-pip install json2html
-```
-
 ### Sphinx packages (Developer Documentation)
 
 Install Sphinx packages if you want to generate HTML Documentation, from source directory run:
@@ -211,17 +215,6 @@ yum install python3-mypy
 apt-get install python3-mypy
 ```
 
-#### Type stubs
-
-```bash
-pip install types-openpyxl types-PyYAML openpyxl-stubs
-```
-
-### cmakelint (Linting CMake Files)
-```bash
-pip install cmakelint
-```
-
 ## Downloading the source code
 
 Download and extract a tarball from the [list of releases](https://github.com/ComplianceAsCode/content/releases):

diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,14 @@
+cmakelint
+types-openpyxl
+types-PyYAML
+openpyxl-stubs
+openpyxl
+json2html
+pyyaml
+Jinja2
+setuptools
+ruamel.yaml
+pandas
+mypy
+lxml
+pycompliance
diff --git a/test-requirements.txt b/test-requirements.txt
@@ -0,0 +1,4 @@
+yamlpath
+xmldiff
+pytest
+pytest-cov