Skip to content

Latest commit

 

History

History
155 lines (126 loc) · 4.13 KB

autoscale-agent-hpa.md

File metadata and controls

155 lines (126 loc) · 4.13 KB
Raw

AutoScale DevOps Agent using HPA

After hosting the DevOps agent on a Kubernetes Cluster, it may be need to auto-scale the number of agents depending on the number of queued builds.

To auto-scale the agent, this post explores using Kubernetes Horizontal Pod Autoscaler (HPA) with an external metric exposed by the Azure DevOps REST API.

Note: The pre-requisite to this post is to have an Image for the DevOps agent as detailed here.

Refer:

The components to achieve this are

  • Metrics Provider
  • Metrics Aggregator
  • HPA Controller
  • Agent Deployment

Metrics Provider

The external metric to scale the agents will be the number of builds in a notStarted Status. The Azure DevOps Services REST API provides the List API with a filter.

For ex. https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=6.1-preview.6&statusFilter=notStarted

Refer to sample exporter here

Metrics Aggregator

The aggregator service is responsible to query the DevOps REST API and format the metric as required by Kubernetes

  • Uses a PAT token with access to list builds
  • REST API call to the devops services
  • Format the response in a kubernetes consumable format
  • Expose the API on an endpoint the APIService object expects

The format of each metric should be as below

{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Service",
        "namespace": "default",
        "name": "devops-proxy",
        "apiVersion": "/v1beta1"
      },
      "metricName": "count",
      "timestamp": "2018-01-10T16:49:07Z",
      "value": "10"
    }
  ]
}

The endpoint that will be consumed by Kubernetes is as /apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace}/services/{serviceName}/{metricName}

HPA Controller

The Horizontal Pod Autoscaler fetches the metrics from the aggregator service and decides to scale the target deployment as required.

  • APIService object to register the external API
  • HPA object to define the target, metric and the Scale
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  insecureSkipTLSVerify: true
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 1000
  versionPriority: 5
  service:
    name: devops-proxy
    namespace: default
  version: v1beta1

The HPA object for reference

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache-autoscaler
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {agentDeploymentName}
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      target:
        kind: Service
        name: devops-proxy
      metricName: count
      targetValue: 1

Agent Deployment

The deployment manifest for the agent

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devops-agent
spec:
  selector:
    matchLabels:
      run: devops-agent
  replicas: 1
  template:
    metadata:
      labels:
        run: devops-agent
    spec:
      containers:
      - name: devops-agent
        image: myregistry.azurecr.io/dockeragent:latest
        env:
          – name: AZP_URL
            value: <https://dev.azure.com/orgname>
          – name: AZP_TOKEN
            value: <PAT_token>
          – name: AZP_AGENT_NAME
            value: <agent_name>
          – name: AZP_POOL
            value: <pool_name>
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m

Testing

Once the deployments are done, queue-up new builds to observe that new agent pods are added.