Categories: ADMIN GUIDE CLOUD

Agent Provisioning configuration

When running in Kubernetes, Step supports automatic provisioning of Agents (i.e. automatic provisioning of the containers responsible for executing Keywords). This page describes how to enable and configure the Agent provisioning.

For information on using Agent provisioning, please refer to the Agent Provisioning page.

Enable Agent provisioning

In order to enable Agent provisioning, the Controller configuration section of the Values file has to be set as follows: 

...
controller:
  agentProvisioning:
    enabled: true
...

Technical details about rights

When Agent provisioning is enabled in the Values, the provisioning module is enabled in the Step controller. Under the hood, the Agent provisioning module requires some additional rights to work in Kubernetes. For this, below Kubernetes API objects are created by the Step Helm chart when agent provisioning is enabled:

A role object, declaring the API access points required by the Step Controller to create the Kubernetes objects:

apiVersion: v1
kind: List
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: Role
  metadata:
    name: enterprise-controller-agent-provisioning
  rules:
- apiGroups:
    - apps
  resources:
    - statefulsets
    - statefulsets/scale
  verbs:
    - list
    - get
    - create
    - delete
    - patch
- apiGroups:
    - ""
  resources:
    - pods
    - pods/log
    - services
    - configmaps
    - events
  verbs:
    - get
    - list
    - create
    - delete

A role binding object, to map the previously defined role to the Step Controller service account:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: enterprise-controller-agent-provisioning
roleRef:
  kind: Role
  name: enterprise-controller-agent-provisioning
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: controllerServiceAccountName
  namespace: yourNamespace

Please consider the official Kubernetes documentation about Role Based Access Control available at https://kubernetes.io/docs/reference/access-authn-authz/rbac/.

Configure Agent pools

The provisioning of Agents and their underlying Kubernetes resources is based on existing Kubernetes resources serving as templates. Upon provisioning, the resource templates are copied and enriched by the provisioning engine of Step. The configuration of the agent pools available for agent provisioning involves therefore the configuration of the Kubernetes template resources.

At startup, Step reads the list of ConfigMaps defined in its namespace and having the annotation step.agent-pool/resource-template=true. The ConfigMaps having this annotation define the resource templates of the agent pools available for provisioning.

A valid agent pool configuration consist of following resource templates:

  • A ConfigMap with the annotation step.agent-pool/resource-template=true
  • A StatefulSet having the same name as the ConfigMap
  • A Service having the same name as the ConfigMap and the StatefulSet

The Step Helm Chart is preconfigured with 3 agent pools for the 3 supported agent types. To define new agent pools, it is recommended to start with one of these examples.

When using the Step Helm chart, the configuration of agent pools can be done in the Values file using the value agentProvisioningTemplate. Enabling this for a specific pool, adds the required annotation to all the Kubernetes resources and makes the pool available as template to the agent provisioning engine.

agent:
pools:
- name: "java-ui-testing"
  agentProvisioningTemplate: "true"
  ...

Example

Enable custom docker image support for an existing agent pool

Step’s agent provisioning allows users to define custom docker images to run their tests.

To enable this feature for an existing agent pool, the token property “$supportsCustomDockerImage: true” has to be added to the Agent pool configuration.

The example below illustrates a Values file defining a Java agent pool exposing one token and supporting custom docker images:

agent:
  pools:
  - name: "java-ui-testing"
    annotations: {}
    tech:
      type: java
      version: 17
    lifecycleHooks: {}
    podSecurityContext:
      runAsUser: 1000
      runAsGroup: 1000
      fsGroup: 1000
      fsGroupChangePolicy: "OnRootMismatch"
    containerSecurityContext: {}
    replicaCount: 0
    startup:
      xvfb: true
    terminationGracePeriodSeconds: 60
    config:
      STEP_DP: ""
      JAVA_OPTS: ""
      NON_HEAP_MEMORY_MB: "1500"
      # Token groups
      tokenGroups:
      # The number of tokens to be emitted
      - capacity: 1
        tokenConf:
          # A list of key-values that identify the agent.
          # These attributes can then be used to select a specific agent in the grid
          attributes:
            type: ui-testing
            tech: java
            $supportsCustomDockerImage: true
          properties:
      properties:
    nodeSelector: {}
    resources:
      requests:
        cpu: "1750m"
        memory: "1800Mi"
      limits:
        cpu: "1750m"
        memory: "1800Mi"

Configure provisioning settings in Plans

When Agent provisioning is enabled in Step, users can configure the Agent provisioning settings in each plan (in the UI and in Yaml).

Refer to the Agent provisioning page for details on how to use this feature.

Troubleshooting

Assuming you have deployed your Step instance using the above examples, you should at least see the below material when you looking at your Kuberbetes API resources:

kubectl -n test get cm,sts,svc,role

NAME                                         DATA   AGE
configmap/enterprise-controller-licenses     2      3m49s
configmap/enterprise-controller-setup        3      3m49s
configmap/java-ui-testing-enterprise-agent   3      3m49s
configmap/kube-root-ca.crt                   1      3m50s
configmap/mongodb-common-scripts             3      3m49s

NAME                                                READY   AGE
statefulset.apps/enterprise-controller              1/1     3m49s
statefulset.apps/java-ui-testing-enterprise-agent   0/0     3m49s
statefulset.apps/mongodb                            1/1     3m49s

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/enterprise-agent        ClusterIP   None           <none>        <none>      3m50s
service/enterprise-controller   ClusterIP   10.64.13.181   <none>        8080/TCP    3m50s
service/grid                    ClusterIP   10.64.7.150    <none>        8081/TCP    3m50s
service/mongodb                 ClusterIP   10.64.15.150   <none>        27017/TCP   3m50s

NAME                                                   CREATED AT
role.rbac.authorization.k8s.io/enterprise-controller-agent-provisioning   2024-05-02T12:12:26Z

Assuming you have created a plan that will require 3 tokens (type=ui-testing, tech=java as properties), the Controller will create the additional Kubernetes resources as per below (mind the ones ending with IDs):

kubectl -n test get cm,sts,svc

NAME                                                  DATA   AGE
configmap/enterprise-controller-licenses              2      7m26s
configmap/enterprise-controller-setup                 3      7m26s
configmap/java-ui-testing-enterprise-agent            3      7m26s
configmap/java-ui-testing-enterprise-agent-pxm9akw6   3      25s
configmap/kube-root-ca.crt                            1      7m27s
configmap/mongodb-common-scripts                      3      7m26s

NAME                                                         READY   AGE
statefulset.apps/enterprise-controller                       1/1     7m26s
statefulset.apps/java-ui-testing-enterprise-agent            0/0     7m26s
statefulset.apps/java-ui-testing-enterprise-agent-pxm9akw6   1/3     24s
statefulset.apps/mongodb                                     1/1     7m26s

NAME                                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/enterprise-agent                            ClusterIP   None           <none>        <none>      7m26s
service/enterprise-controller                       ClusterIP   10.64.13.181   <none>        8080/TCP    7m26s
service/grid                                        ClusterIP   10.64.7.150    <none>        8081/TCP    7m26s
service/java-ui-testing-enterprise-agent-pxm9akw6   ClusterIP   None           <none>        <none>      25s
service/mongodb                                     ClusterIP   10.64.15.150   <none>        27017/TCP   7m26s

See Also