vault-reader

The purpose of this task is to read secrets from HashiCorp Vault server in order to use it during the pipeline or the release process.

The icons for that task was taken from https://icons8.com/

Version 3.*

The task supported with generic Key-Value in versions 1 & 2 without changing the path (although kv2's API is not compatible with v1).

What Changed on the path:
V1: kv_name/level-1/../level-n
V2: kv_name/data/level-1/../level-n

Currently, the vault-reader adds data level automatically. first, it tries to add data to support v2, if it doesn't work it tries it as is (without adding the data to the path). This option was added to the task because v2 is Incompatibility with v1 and there is no reason for the pipeline's processes to break.
This feature will removed on the next major version to be align with the API.

Version 4.* - Coming Soon

• Remove the support of v1 & v2 that added to the task - it should be align with HashiCorp vault API.
• Allow to run the task on the Pre-job process (Open issue)

Task Configuration

How to create a service connection:

• Open Azure DevOps and select the relevant project.
• go to project settings.
• Select service connections under Pipelines section.
• Create a new service by clicking on the "New service connection" button.
• Select "Vault Reader" type and press Next.

• Fill all details:

  • Server URL - the URL of the HashiCorp server. i.e. https://myvault.com:8200
  • Auth Methods - select the method from the list. i.e. LDAP, Token etc.
  • Username - enter username.
  • Password/Token - enter password or token for that user.
  • Disable strict SSL - select this option if you get the error: unable to verify the first certificate.

• Enter service name and description, click Save.

---

How to work with the task during a build/release process

Open build/release and add the task from the list.

Fields:

• Vault Service - select the Vault Connection Service from the list.
• Source Type:
  • Inline - define instructions (Variables & Actions) in multiline box.
  • File Path - read the instructions from file (there is also multiline box to define variables).

Instructions:

The signs => or <= are part of the task instructions.

General Instructions:

• Comment - add a comment to your instruction by using # at the beginning of the line. the task will ignore that line.
• Message - print a message during the Build/Release process by using @ at the beginning of the line.

Variables:
You can define a variable and use it later with the Action commands.

Format: Variable-Name <= Value

Variable-Name can contain Letters (upper/lower), numbers and underline ('_'). Must start with a letter.

Example:
projectPath <= /project/serviceA

In this example we will create a variable named projectPath that sets the value /project/serviceA

Result variable - for some Action type you can use the Azure-DevOps-Variable as a special variable for the next lines, See comment on the table below.

Define Actions:

General Format: ActionType => Path => Field => Azure-DevOps-Variable

Task Variables can affect the Path and Field of the action
Azure-DevOps-Variable can contain Letters (upper/lower), numbers, period and underline. Must start with a letter.

Action	Description	Azure DevOps Variable 1
var	reads value from Path & Field	assigns value to Azure DevOps variable 2
pre	reads object from Path Field will contain a list of keys (separated by a comma) or * for all keys (not recommended)	assigns multiple values to multiple variables in a single command 2 3
raw	reads value from Path & Field and store the value into a file (as is)	assigns file location into a variable 4 5
base64	reads value from Path & Field, decodes the value from BASE64 and stores the result into a file	assigns file location into a variable 4 5
json	reads json object from Path and stores it into a file as json Field will contain location of the file schema. If the data and the scheme aren't equal it would fail use * to skip the compare process (not recommended)	assigns file location into a variable 5 6
yaml	reads json object from Path and stores it into a file as yaml Field will contain location of the json file schema. If the data and the scheme aren't equal it would fail use * to skip the compare process (not recommended)	assigns file location into a variable 5 6
rep	reads file from Field and replaces the string __[key]__ with a value that reads from Path and stores it into a file	assigns file location into a variable 5 6
exp	export JSON object into file. The line define in the Field as base64, %%KEY%% and %%VALUE%% defined place holder for key & value from the JSON object.	assigns file location into a variable 5

Examples:

How to use var and pre actions

Read values from project-A using var action

# This line is a comment.
# Read environment value from vault into env variable 
var => DemoProjects/project-A => environment => env

# Read username & password from vault into usr & pass variables
var => DemoProjects/project-A => username => usr
var => DemoProjects/project-A => password => pss

Read username & password in one line using pre action and store them in login_username & login_password.

# Read username & password with pre action
pre => DemoProjects/project-A => username,password => login

How to use raw and base64 actions

Read value and save it into a file. The variable will be set with the file location.

@ This line will be printed on the build/release console
@ Read rawData and save it into a file. File path will be stored at variable $(file1)
raw =>  DemoProjects/project-B/service-A => rawData => file1

# Read mySecret from vault, decode it (base64) and save it into a file.
# The location will be stored at the variable secret.
base64 =>  DemoProjects/project-B/service-A => mySecret => secret

How to work with VaultReader variables

This section refers to the previous image.

# Define a variable that named servicePath
servicePath <= DemoProjects/project-B/service-A

# Read value using a variable
raw => {servicePath} => rawData => file1
base64 => {servicePath} => mySecret => secret

You can also use a variable as a part of Path/Field.
For example: {servicePath}/test (equal to DemoProjects/project-B/service-A/test)

How to use json action

We can create a json file that contains a template (value will be empty). This way we can compare the scheme between the objects from the vault server and the json file. The json file can be stored on the source control and will be pulled during the build.

File: service-b-template.json
The json file will be stored under config folder in the git repository.

{
  "service": {
    "name": "",
    "url": ""
  },
  "sql": {
    "ConnectionString": "__sql__"
  }
}

# Define a path
servicePath <= DemoProjects/project-B

# Read value using json action
json => {servicePath}/service-B => config/service-b-template.json => configFile

How to use yaml action

This section is based on the data that presented in the JSON explanation.

# Define a path
servicePath <= DemoProjects/project-B

# Read value using json action
yaml => {servicePath}/service-B => config/service-b-template.json => configFile

How to use replace action

When we have a file that we want to inject secrets during the build/release process to it. For example, a secret.yaml file on the working directory that defines Secret for k8s cluster:

apiVersion: v1
kind: Secret
metadata:
  name: secret-basic-auth
type: kubernetes.io/basic-auth
stringData:
  username: __username__
  password: __password__

On the task instructions:

# Inject secrets into a yaml file
rep => DemoProjects/project-A => secret.yaml => secretFile

The variable $(secretFile) will contain the location of the updated file.
To update kubernetes cluster you can run the following command:

kubectl apply -f $(secretFile)

How to use export action

Export the keys and values from the JSON object (1 level) into a file, every key & value will be placed in deferent line by the format in the Field. The format is set as base64, the %%KEY%% and %%VALUE%% will replace with the key & value of each item.

For example, we would like to convert JSON object to tfvars file (This type of file is used by terraform).

Line format: %%KEY%% = "%%VALUE%%"
Line format (Base64): JSVLRVklJSA9ICIlJVZBTFVFJSUi

The line we should add to the task:

exp => DemoProjects/project-A => JSVLRVklJSA9ICIlJVZBTFVFJSUi => tfvar_file

The file that will create after the task is complete, the path to the file will set to tfvar_file.

environment = "dev"
password = "p@assw0rd"
username = "someone"

How to work with VaultReader special variables

# Define a path
servicePath <= DemoProjects/project-B

# Read value using json action
json/yaml => {servicePath}/service-B => config/service-b-template.json => configFile
rep => database/sql => {{configFile}} => configFile2

Footnotes

1. Azure DevOps Variable - the result of the action will be stored at the variable and can be used in the next tasks as $(variableName)
   ↩

2. The type of the variable is "secret" and therefore it can't be printed in the build & release console.
   ↩ ↩²

3. Variable name: Azure-DevOps-Variable_Field
   ↩

4. On Linux OS set permissions to Read only.
   ↩ ↩²

5. The file that contains the secrets will be deleted at the end of the build/release process (the file is stored under _temp folder).
   ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

6. The Azure-DevOps-Variable can used as a special variable surrounded with {{ }}, with this option output from one action can be the input for the others. ↩ ↩² ↩³