Categories: ADMIN GUIDE CLOUD

Autoscaler configuration

Workflow

The autoscaling Step plugin works as described below:

  • When starting an execution, the Step plan is scanned to estimate the required number of agent tokens per agent type (taking into account the optional agent routing)
  • It identifies the available Agent pools matching the required types (based on the Kubernetes ConfigMap available in your namespace)
  • It provisions the required agents (technically statefulsets are created on the fly for this execution using the matched statutefulsets and configmap as templates)
  • Once the all agents are ready, the actual execution starts
  • After the execution, the agents are deprovisions (the kubernetes resources created for this executions are cleaned up)

Refer to the Agent configuration tutorial for more details about Agent token pool configuration.

Controller configuration

Property

In order to start the Autoscaler plugin, the below property has to be added to your Step Controller configuration (step.properties file): 

grid.tokens.autoscaler.driver=step.plugins.k8s.autoscaling.K8sTokenAutoscalingDriver

Using the Step Helm Chart, that could be achieved by adding the below to your values file: 

controller:
...
  config:
    additionalProperties:
      grid.tokens.autoscaler.driver: step.plugins.k8s.autoscaling.K8sTokenAutoscalingDriver
...

Required Kuberbetes role

Below Kuberbetes roles have to be granted to the Step Controller to be able to perform the autoscaling (Kubernetes API access):

apiVersion: v1
kind: List
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: enterprise-controller-role
  rules:
  - apiGroups:
    - apps
    resources:
    - statefulsets
    verbs:
    - list
    - get
    - create
    - delete
  - apiGroups:
    - ""
    resources:
    - services
    - configmaps
    verbs:
    - get
    - list
    - create
    - delete

Using the Step Helm Chart, that could be achieved by adding the below to your values file: 

controller:
...
  serviceAccount:
    automountServiceAccountToken: true
    create: true
  rbac:
    create: true
    rules:
    - apiGroups:
      - apps
      resources:
      - statefulsets
      verbs:
      - list
      - get
      - create
      - delete
    - apiGroups:
      - ""
      resources:
      - services
      - configmaps
      verbs:
      - get
      - list
      - create
      - delete
...

Please consider the official Kuberbetes documentation available at https://kubernetes.io/docs/reference/access-authn-authz/rbac/.

Agent Token Group configuration

Using the Step Helm Chart, it is pretty easy to deploy an Agent pool and its associated configuration (mind the tokenGroups section).

Below an example of what your values file would contain to create a pool of agent dedicated to UI testing (1 token per Agent, tokenConf attributes type=ui-testing and tech=java)

agent:
  pools:
  - name: "java-ui-testing"
    annotations: {}
    tech:
      type: java
      version: 17
    lifecycleHooks: {}
    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
      fsGroup: 1000
      fsGroupChangePolicy: "OnRootMismatch"
    containerSecurityContext: {}
    replicaCount: 0
    startup:
      xvfb: true
    terminationGracePeriodSeconds: 60
    config:
      STEP_DP: ""
      JAVA_OPTS: ""
      NON_HEAP_MEMORY_MB: "1500"
      # Token groups
      tokenGroups:
      # The number of tokens to be emitted
      - capacity: 1
        tokenConf:
          # A list of key-values that identify the agent.
          # These attributes can then be used to select a specific agent in the grid
          attributes:
            type: ui-testing
            tech: java
          properties:
      properties:
    nodeSelector: {}
    resources:
      requests:
        cpu: "1750m"
        memory: "1800Mi"
      limits:
        cpu: "1750m"
        memory: "1800Mi"

Plan configuration

Routing

Configure your Plan with the appropriate routing, ie in that example with “tech=java” and “type=ui-testing” to target the Agent pool mentioned previously.

UI Autoscaling configuration

It is also possible to modifiy the Autoscaling configuration on a plan basis, by clicking on the dedicated button from the Plan editor view:

autoscaler-settings.png

From here, you can:

  • choose to disable the Autoscaler for this plan
  • choose to determine yourself the number of Agents required for your plan (instead of letting the Controller do the calculation for you)

Execution

Assuming you have deployed your Step instance using the above examples, you should at least see the below material when you looking at your Kuberbetes API resources:

kubectl -n test get cm,sts,svc,role

NAME                                         DATA   AGE
configmap/enterprise-controller-licenses     2      3m49s
configmap/enterprise-controller-setup        3      3m49s
configmap/java-ui-testing-enterprise-agent   3      3m49s
configmap/kube-root-ca.crt                   1      3m50s
configmap/mongodb-common-scripts             3      3m49s

NAME                                                READY   AGE
statefulset.apps/enterprise-controller              1/1     3m49s
statefulset.apps/java-ui-testing-enterprise-agent   0/0     3m49s
statefulset.apps/mongodb                            1/1     3m49s

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/enterprise-agent        ClusterIP   None           <none>        <none>      3m50s
service/enterprise-controller   ClusterIP   10.64.13.181   <none>        8080/TCP    3m50s
service/grid                    ClusterIP   10.64.7.150    <none>        8081/TCP    3m50s
service/mongodb                 ClusterIP   10.64.15.150   <none>        27017/TCP   3m50s

NAME                                                   CREATED AT
role.rbac.authorization.k8s.io/enterprise-controller   2024-05-02T12:12:26Z

Assuming you have created a plan that will require 3 tokens (type=ui-testing, tech=java as properties), the Controller will create the additional Kubernetes resources as per below (mind the ones ending with IDs):

kubectl -n test get cm,sts,svc

NAME                                                  DATA   AGE
configmap/enterprise-controller-licenses              2      7m26s
configmap/enterprise-controller-setup                 3      7m26s
configmap/java-ui-testing-enterprise-agent            3      7m26s
configmap/java-ui-testing-enterprise-agent-pxm9akw6   3      25s
configmap/kube-root-ca.crt                            1      7m27s
configmap/mongodb-common-scripts                      3      7m26s

NAME                                                         READY   AGE
statefulset.apps/enterprise-controller                       1/1     7m26s
statefulset.apps/java-ui-testing-enterprise-agent            0/0     7m26s
statefulset.apps/java-ui-testing-enterprise-agent-pxm9akw6   1/3     24s
statefulset.apps/mongodb                                     1/1     7m26s

NAME                                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/enterprise-agent                            ClusterIP   None           <none>        <none>      7m26s
service/enterprise-controller                       ClusterIP   10.64.13.181   <none>        8080/TCP    7m26s
service/grid                                        ClusterIP   10.64.7.150    <none>        8081/TCP    7m26s
service/java-ui-testing-enterprise-agent-pxm9akw6   ClusterIP   None           <none>        <none>      25s
service/mongodb                                     ClusterIP   10.64.15.150   <none>        27017/TCP   7m26s

See Also