Skip to content

Latest commit

 

History

History
528 lines (368 loc) · 27.3 KB

contributing.md

File metadata and controls

528 lines (368 loc) · 27.3 KB
Raw

Contributing

This document is intended to be an introduction to contributing to the terraform-provider-azuredevops project. Links to background information about the project and general guidance on Terraform providers are included below:

If you are looking for background information on the project or related technologies (Terraform, Go and Azure DevOps), consider checking out some of these resources first:

If you are familiar with the technologies used for this project but are looking for general guidance on Terraform provider development, consider checking out some of these resources first:

If you are still reading, then you are in the right place!

Looking for more?

If, after reading through the content here, you are seeking more detailed information, you may want to checkout some of the following resources:

Workspace Setup & Building Project

This section describes how to get your developer workspace running for the first time so that you're ready to start making contributions. If you have already done this, check out Development SDLC.

These steps assume you are running with bash. If you are using Windows, run all commands using WSL. They are not tested using GitBash.

0. (Optional) Docker

To develop using docker:

Note: It is highly recommended to use VSCode as described for rich intellisense, debugging, and access to standard go development tools.

  • Clone the repo: git clone https://github.com/microsoft/terraform-provider-azuredevops.git

If using VSCode's Remote Development Extension:

  • Open the project in VSCode
  • When prompted by VSCode, select "Reopen in Container"
  • In the command palette (ctrl+shift+p on Windows/Linux, command+shift+p on Mac) type: Go: Install/Update tools and select it
  • Note: SSH credentials are automatically mapped into the container
  • When committing, you will be prompted to configure git user.name and user.email

Without VSCode:

  • Build the development environment: docker build -f .devcontainer/Dockerfile -t dev .
  • From bash or powershell, mount the root directory into the docker container, and drop into a bash shell in the container: docker run -it -v ${PWD}:/workspaces/terraform-provider-azuredevops dev

Continue with the guide to run the provider locally.

1. Install dependencies

The recommended development environment is Linux or Mac. If you're on Windows you should install WSL so that your environment more closely mirrors a Linux environment.

You will need the following dependencies installed in order to get started:

2. Clone repository

Note: This project uses Go Modules making it safe to work with it outside of your existing GOPATH. The instructions that follow assume a directory in your home directory outside of the standard GOPATH.

Note These steps assume you are running with bash. If you are using Windows, run all commands using WSL. They are not tested using GitBash.

Note for Go 1.13

If you are using Go 1.13+ , you can get this below error during the execution of the provided compilation script

verifying golang.org/x/[email protected]: golang.org/x/[email protected]: open /mnt/d/Go/pkg/sumdb/sum.golang.org/latest: no such file or directory

As mentionned in the Changelog 1.13 https://golang.org/doc/go1.13#modules For solve it, run in your terminal the commands:

go env -w GOPROXY=direct
go env -w GOSUMDB=off

Setup your workspace

Setup your workspace

$ DEV_ROOT="$HOME/workspace"
$ mkdir -p "$DEV_ROOT"
$ cd "$DEV_ROOT"

Get the code

$ git clone https://github.com/microsoft/terraform-provider-azuredevops.git
$ cd terraform-provider-azuredevops/

3. Build & Install Provider

To build and install the provider locally execute the make install command which will execute the build.sh --Install --SkipTests command for you.

$ make install

For more information about the process see below.

Build & test the azure devops provider

Running the build.sh script will orchestrate a few things for you:

  • Verify that all required Go packages are installed. This may take a few minutes the first time you run the script as the packages will be cached locally. Subsequent runs will be much faster
  • Run all unit tests
  • Compile the provider codebase
$ ./scripts/build.sh
...
[INFO] Executing unit tests
...
[INFO] Build finished successfully

After this script runs you should see a ./bin/ directory with the compiled terraform provider.

The build.sh suppports the following parameters:

-s | --SkipTests : Skip executing the unit tests -d | --DebugBuild: Execute go build with flags that disables optimization and adds debug symbols to the created binary -i | --Install : After successful build perform a local install of the provider by executing local-install.sh

$ ls -lah ./bin/
...
-rwxrwxrwx 1 ... terraform-provider-azuredevops_v0.0.1

Install the provider

Terraform provider plugins are not intended to be run directly. You can see this for yourself:

$ ./bin/terraform-provider-azuredevops_v0.0.1

This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

In order to use the provider locally it must be installed into a location discoverable by Terraform. The local-install.sh script does this for you.

$ ./scripts/local-install.sh
[INFO] Installing provider to /home/$USER/.terraform.d/plugins/

To learn more about the plugin discovery process, refer to the official Terraform provider discovery documentation.

4. Run provider locally

Note: These steps assume you have built the provider locally using the previous steps. The samples in the ./examples/ folder use syntax specific to Terraform 12 + and are not compatible with older versions of Terraform.

Configure the provider

You can now use the provider just like you normally would. Try it out by using the project examples:

$ cd examples/github-based-cicd-simple/

# AZDO_ORG_SERVICE_URL will be the URL of the AzDO org that you want to provison
# resources inside of.
#   ex: https://dev.azure.com/<your org name>
$ export AZDO_ORG_SERVICE_URL="..."

# AZDO_PERSONAL_ACCESS_TOKEN will be the personal access token that grants access
# to provision and manage resources in Azure DevOps.
# You'll need to configure permissions for AzDO and Github PATS detailed in ./website/docs
#   documentation: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops
$ export AZDO_PERSONAL_ACCESS_TOKEN="..."

# Note: AZDO_GITHUB_SERVICE_CONNECTION_PAT is not specifically required
# by the provider, but it is required by the example in this folder.
#   documentation: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line
$ export AZDO_GITHUB_SERVICE_CONNECTION_PAT="..."

$ terraform init
...
Terraform has been successfully initialized!
...

The provider has now been initialized and can be used like any other provider in Terraform. You can try for yourself by running any of the terraform plan|apply|destroy|... commands.

Development SDLC

This section outlines typical development scenarios for the repository, relevant areas of the codebase and provides guidance for writing tests and documentation for your changes.

1. Pick an issue

Please find an open issue in the backlog that has no asignee and request it to be assigned to you by a core contributor if you don't have the permissions to do it yourself. This will help us track which work items are being worked on.

2. Repository Structure

The repository generally follows the patterns suggested by HashiCorp. This pattern is implemented by all major terraform providers.

Here is a simplified version of the repository:

$ tree -L 2
.
├── azdosdkmocks --------> Generated mocks for AzDO Go SDK
├── azuredevops ---------> Provider implementation
│   ├── config.go -------> AzDO SDK initialization lives here
│   ├── provider.go -----> Exports the AzDO terraform provider
│   ├── data_*.go -------> data_*.go files contain terraform data sources implementations
│   ├── resource_*.go ---> resource_*.go files contain terraform resource implementations
│   └── utils -----------> Utilities used across the codebase
├── docs ----------------> Developer documentation
├── go.mod --------------> Describes project dependencies
├── scripts -------------> All scripts live here
└── website -------------> Client facing documentation

3. Code for Terraform

After you picked an issue and figured out where you will implement it, you will quickly realize that HashiCorp has taken a very opinionated approach to building Terraform providers. The following section will outline a few common scenarios for terraform plugin development, and point you towards different pieces of the code that you may need to work with.

If you have not already gone through the guide published by HashiCorp, now is a good time to do so.

Scenario 1: Change resource or data source schema

If you need to add, remove or modify the schema of a data source or resource, you will need to first identify the relevant file. The naming scheme used is as follows:

  • data_foo.go - an implementation for the Terraform data source for the foo Azure DevOps resource
  • resource_foo.go - an implementation for the Terraform resource for the foo Azure DevOps resource

Open the file and look for the schema. Here is a simple schema found in data_group.go. This is fairly simple and only defines three attributes. More complicated ones can be found in the build definition code. The official documentation for the schema can be found here.

Group Data Source Schema

Scenario 2: Modify an existing resource or data source

If you need to modify the business logic in an existing resource or data source, you will find the relevant code in one of the Create, Read, Update or Delete functions.

Resource CRUD Functions

The prototype of these functions are all quite similar. Here is an example of a create function. Keep note of the following details:

  • d *schema.ResourceData is passed to the provider by Terraform. It contains the resource configuration specified by the client using the provider, along with any data pulled from the Terraform state.
  • m interface{} is, in the case of this provider, a structure containing all of the (intialized) clients needed to make API calls to Azure DevOps.
  • Flatten/Expand is a common "idiom" used across terraform providers. It is a standard approach to marshaling and unmarshaling API data structures into the internal terraform state.

image

Scenario 3: Implement a new resource or data source

If you need to implement a new resource or data source, you should first review a few implementations in order to understand the patterns in the codebase, which generally match the way that other Terraform providers are built.

This scenario is a mix of Scenario 1 and Scenario 2. However, after implementing the schema and CRUD operations, you will need to register your newly created resource or data source. The code for doing this is located in provider.go.

Provider Registration

In order to accelerate development and ensure a common structure of all components (resource, data source), the project offers several Visual Studio Code snippets that should preferably be used to create new code.

General information about how to work with snippets inside Visual Studio Code are available in the official documentation.

If you are going to create a new data source care for the wording of your Go function and the Terraform data source name. If your data source is able to return more than one entity, be sure to use the plural form of the object you're returning with the data source.

Example: returning multiple groups from Azure DevOps

Because the data source returns more than one group object from Azure DevOps

  • Data source name azuredevops_groups
  • Go function name func dataGroups() *schema.Resource

Reference: Terraform Data Source Names

If the new data source is able to return more than one object, be sure you are using a schema.TypeSet to return the elements.

Example: returning multiple projects from Azure DevOps as set with azuredevops_projects

"projects": {
  Type:     schema.TypeSet,
  Computed: true,
  Set:      getProjectHash,
  Elem: &schema.Resource{
    Schema: map[string]*schema.Schema{
      "name": {
        Type:     schema.TypeString,
        Computed: true,
      },
      "project_id": {
        Type:     schema.TypeString,
        Computed: true,
      },
      "project_url": {
        Type:     schema.TypeString,
        Computed: true,
      },
      "state": {
        Type:     schema.TypeString,
        Computed: true,
      },
    },
  },
},

Don't use Sleep

Managing resources in the cloud often means using APIs that return a state even though the affected resource has not yet reached the expected state. One might be tempted to use the Sleep function to wait for the resource to be in the correct state for further actions. But that is considered bad practice, because the time how long a cloud resource needs to reach it's final (expected) state isn't really predictable since the deployment, removal or change of a cloud resource could depend on many factors, like replication, database locks, runtime etc. that, as said, the time how long the client must wait for a resource reaching a state isn't foreseeable.

Thus, as a provider developer, you should take delays in resource APIs into account in the CRUD functions of the resource. Terraform supports configurable timeouts to assist in these situations.

Terraform provides the following mechanisms to wait for a state change of a resource:

  1. Using the retry mechanism of the schema.Resource (Retry)

  2. Using the WaitForState function of resource.StateChangeConf (StateChangeConf)

Both means, if implemented correctly, guarantee that subsequent actions within the provider implementation can be confident that the affected resource has reached an expected state.

Visual Studio Code snippets

Shortcut: tf-azdo-rs Purpose: Create new resource How to use:

  1. Create a new .go file, name it according to the naming scheme and save it.
  2. Type tf-azdo-rs + [Ctrl+Space]. The snippet will prompt you for the new name of the resource.
  3. Press [ESC]
  4. Save the file again

Shortcut: tf-azdo-ds Purpose: Create new data source How to use:

  1. Create a new .go file, name it according to the naming scheme and save it.
  2. Type tf-azdo-ds + [Ctrl+Space]. The snippet will prompt you for the new name of the resource.
  3. Press [ESC]
  4. Save the file again

Shortcut: tf-azdo-test Purpose: Create new test file How to use:

  1. Create a new _test.go file, name it according to the naming scheme and save it.

  2. Type tf-azdo-test + [Ctrl+Space]. The snippet will prompt you for the following parameters:

    • The name of the resource or data source which is under test
    • The function to test
    • Test test you want to perform

  3. Press [ESC]

  4. Save the file.

Shortcut: tf-azdo-test-func Purpose: Create new test function How to use:

  1. Open an existing _test.go file, navigate to the end of the file

  2. Type tf-azdo-test-func + [Ctrl+Space]. The snippet will prompt you for the following parameters:

    • The name of the resource or data source which is under test
    • The function to test
    • Test test you want to perform

  3. Press [ESC]

  4. Save the file.

4. Test changes

Running Unit Tests

The unit tests are executed whenever ./scripts/build.sh is run. This can be run locally, but will also be run on every automated build and will be a gate for any PR against this repository. The tests can also be run in isolation by running the following:

$ ./scripts/unittest.sh

Running Acceptance Tests (Integration Tests)

Note: Running acceptance tests provisions and deletes actual resources in AzDO. This can cost money and can be dangerous if you are not running them in isolation!

The acceptance tests for terraform providers are typically implemented as Acceptance Tests. Acceptance tests can be invoked by running the following:

When generating a new PAT, the acceptance tests require the provisioning of certain resources this token will need to provide permission for creating. For simplicity you can authorize the scope of access associated with this token to Full Access with a short expiration date. For custom defined scope, refer to the Azure DevOps Provider: Authenticating using the Personal Access Token document that provides more information on how to configure this correctly.

# AZDO_ORG_SERVICE_URL will be the URL of the AzDO org that you want to provison
# resources inside of.
#   ex: https://dev.azure.com/<your org name>
$ export AZDO_ORG_SERVICE_URL="..."

# AZDO_PERSONAL_ACCESS_TOKEN will be the personal access token that grants access
# to provision and manage resources in Azure DevOps.
#   documentation: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops
$ export AZDO_PERSONAL_ACCESS_TOKEN="..."

# Note: AZDO_GITHUB_SERVICE_CONNECTION_PAT is not specifically required
# by the provider, but it is required by the acceptance tests in order to test
# the authentication with GitHub for build definitions hosted in GitHub.
#   documentation: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line
$ export AZDO_GITHUB_SERVICE_CONNECTION_PAT="..."

# Note: AZDO_TEST_AAD_USER_EMAIL will be the e-mail address that is not included in the current organization
# ,but needs to be an account in the Azure Active Directory. The e-mail is used for acceptance testing of the User Entitlement resource.
$ export AZDO_TEST_AAD_USER_EMAIL="..."

$ ./scripts/acctest.sh

Writing your own tests

There is a lot of context to cover here, so check out our dedicated testing document for more information on writing unit and acceptance tests.

5. Debug changes

There is a lot of context to cover here, so check out our dedicated debugging document for more information on debugging the provider.

6. Document changes

Most changes should involve updates to the client-facing reference documentation. Please update the existing documentation to reflect any changes you have made to the codebase.

Name Description Link
Index Table of contents index.md
Resources Resources reference resources
Data Sources Data Sources reference data sources
Guides Guide and tutorial docs guides

7. Explore with postman

Azure DevOps has a rich set of REST API's available for integrating with Terraform. A convenient way to explore the APIs is through postman. Once you have postman installed, you can clone/fork this repo and follow the instructions in the README to setup postman to be able to explore a large set of the rest API's. A second resource that is very helpful in understanding how to call/use the apis is the Azure DevOps Cli extension found here.

8. Post a PR

After the feature or bug is completed, you'll want to post a PR to get it reviewed by somebody in the core development team. Here are some useful tips to keep in mind:

Tips for PR authors

  • The steps run by the CI are located here. You should manually execute these commands to make sure that they pass before you post your PR.

    • The below command runs everything, but the acceptance tests:

      $ make ci

  • The PR template for the repository is located here. Please be ready to fill this template out when posting the PR.

  • If your PR is a draft, please mark it as a draft and put WIP (stands for "Work In Progress") in the title. This helps reviewers prioritize which reviews to look at.

  • If your PR contains changes to the go.mod or go.sum files, or the vendor/ directory, please elaborate on why the change is being made.

Tips for PR reviewers

  • If a PR contains changes to the go.mod or go.sum files, or the vendor/ directory, carefully understand them and make sure that these changes are (1) minimal and (2) necessary. This helps the stability of the product over time. Dependency upgrades to get off of old, deprecated or unsupported libraries should be done in isolated PRs, if possible.

Contributing back

If you find an api is missing from the postman collection, please submit a PR. We intend make the collection as complete as possible over time.

9. Creating a release

You can create a github release of this project or a fork of this project using goreleaser. The gorelease will create binaries for Mac, Linux and Windows based of the fork you're making the release from. To be able to make the release you need to have the required permissions to create a release for this project or a fork.

To make the release for your fork change the owner in .goreleaser.yml to be the name of the account holding the fork.

Making the release:

  1. Open this project in the dev container or install gorelaser
  2. Create a Github PAT token with all of the Repo permissions
  3. Set GITHUB_TOKEN enviorment variable to the above PAT token 
    export GITHUB_TOKEN=<Your PAT token>
  4. Set the release version you would like to create 
    RELEASE_VERSION="v0.1.7"
  5. Create 
    git tag $RELEASE_VERSION && goreleaser release --rm-dist

More information about using goreleaser here: https://goreleaser.com/

Note about CLA

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.