4.1. Virtual machine replica sets

Using virtual machine replica sets

A VirtualMachineInstanceReplicaSet resource tries to ensure that a specified number of virtual machines are always in a ready state. The VirtualMachineInstanceReplicaSet is very similar to the Kubernetes ReplicaSet¹.

However, the VirtualMachineInstanceReplicaSet does not maintain any state or provide guarantees about the maximum number of VMs running at any given time. For instance, the VirtualMachineInstanceReplicaSet may initiate new replicas if it detects that some VMs have entered an unknown state, even if those VMs might still be running.

Using a VirtualMachineInstanceReplicaSet

Using the custom resource VirtualMachineInstanceReplicaSet, we can specify a template for our VM. A VirtualMachineInstanceReplicaSet consists of a VM specification just like a regular VirtualMachine. This specification resides in spec.template. Besides the VM specification, the replica set requires some additional metadata like labels to keep track of the VMs in the replica set. This metadata resides in spec.template.metadata.

The amount of VMs we want the replica set to manage is specified as spec.replicas. This number defaults to 1 if it is left empty. If you change the number of replicas in-flight, the controller will react to it and change the VMs running in the replica set.

The replica set controller needs to keep track of the VMs running in this replica set. This is done by specifying a spec.selector. This selector must match the labels in spec.template.metadata.labels.

A basic VirtualMachineInstanceReplicaSet template looks like this:

apiVersion: kubevirt.io/v1 
kind: VirtualMachineInstanceReplicaSet    
metadata:    
  name: vmi-replicaset   
spec:    
  replicas: 2  # desired instances in the replica set
  selector:    
    # VirtualMachineInstanceReplicaSet selector
  template:    
    metadata:    
      # VirtualMachineInstance metadata
    spec:    
      # VirtualMachineInstance template
[...]

Note

Be aware that if spec.selector does not match spec.template.metadata.labels, the controller will do nothing except log an error. Further, it is your responsibility to not create two VirtualMachineInstanceReplicaSet conflicting with each other.

A real world example could look like this:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceReplicaSet
metadata:    
  name: vmi-cirros-replicaset
spec:
  replicas: 2
  selector:    
    matchLabels:
      kubevirt.io/domain: vmi-cirros
  template:    
    metadata:    
      labels:
        kubevirt.io/domain: vmi-cirros
    spec:
[...]

When to use VirtualMachineInstanceReplicaSets

You should use VirtualMachineInstanceReplicaSets whenever you want multiple exactly identical instances not requiring persistent disk state. In other words, you should only use replica sets if your VM is ephemeral and every used disk is read-only. If the VM writes data this should only be allowed in a tmpfs.

Warning

You should expect data corruption if the VM writes data to a storage not being a tmpfs or an ephemeral type.

Volume types which can safely be used with replica sets are:

cloudInitNoCloud
ephemeral
containerDisk
emptyDisk
configMap
secret
any other type if the VM instance just writes to a tmpfs

Note

This is the most important difference to a VirtualMachinePool. If you want to manage multiple unique instances using persistent storage, you have to use a VirtualMachinePool. If you want to manage identical ephemeral instances which do not require persistent storage or different data sources (startup scripts, configmaps, secrets) you should use a VirtualMachineInstanceReplicaSet.

Using a VirtualMachineInstanceReplicaSet

We will create a VirtualMachineInstanceReplicaSet using a CirrOS container disk from a container registry. As we know, container disks are ephemeral, so this fits our use case very well.

Task 4.1.1: Create a VirtualMachineInstanceReplicaSet

Click on the plus + in the upper right menu to import a resource as YAML. Make sure the correct Project is selected.

Import YAML

Paste the following content:

Note

Alternatively, paste the content into a file on your local filesystem, edit it as instructed and only paste the resulting, final YAML definition.

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceReplicaSet
metadata:    
  name: lab05-cirros-replicaset
spec:
  replicas: 2
  selector:
    matchLabels:
      kubevirt.io/domain: lab05-cirros
  template:
    metadata:
      labels:
        kubevirt.io/domain: lab05-cirros
    spec:
[...]

Enhance the spec.template.spec block to start a VM matching these criteria:

Use the container disk quay.io/kubevirt/cirros-container-disk-demo
Use an empty cloudInitNoCloud block
Use 1 replicas
Configure the guest to have 1 cores
Resources
- Request 265Mi of memory
- Request 100m of cpu
- Limit 300m of cpu

Use this empty cloudInitNoCloud block to prevent cirros from trying to instantiate by using a remote url:

      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |
            #cloud-config

Task hint

Your resulting VirtualMachineInstanceReplicaSet should look like this:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceReplicaSet
metadata:
  name: lab05-cirros-replicaset
spec:
  replicas: 2
  selector:
    matchLabels:
      kubevirt.io/domain: lab05-cirros
  template:
    metadata:
      labels:
        kubevirt.io/domain: lab05-cirros
    spec:
      domain:
        cpu:
          cores: 1
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
        resources:
          limits:
            cpu: 300m
          requests:
            cpu: 100m
            memory: 265Mi
      networks:
      - name: default
        pod: {}
      volumes:
      - name: containerdisk
        containerDisk:
          image: quay.io/kubevirt/cirros-container-disk-demo
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |
            #cloud-config

If you haven’t already done so, paste the resulting YAML into the Import YAML editor.

Click Create to create the VirtualMachineInstanceReplicaSet resource.

Task 4.1.2: Access the VirtualMachineInstanceReplicaSet

There is not much the CirrOS disk image provides besides entering the VMs using the console.

Open the web terminal to execute the following commands.

Check the availability of the VirtualMachineInstanceReplicaSet with:

oc get vmirs --namespace lab-<username>

NAME                      DESIRED   CURRENT   READY   AGE
lab05-cirros-replicaset   2         2         2       1m

List the created VirtualMachineInstances using:

oc get vmi --namespace lab-<username>

NAME                           AGE    PHASE     IP             NODENAME            READY
lab05-cirros-replicasetnc5p5   11m    Running   10.244.3.96    training-worker-0   True
lab05-cirros-replicasetp25s2   11m    Running   10.244.3.149   training-worker-0   True

You can access the VM’s console with virtctl using the name of the VMI:

virtctl console lab05-cirros-replicasetnc5p5 --namespace lab-<username>

Scaling the VirtualMachineInstanceReplicaSet

As the VirtualMachineInstanceReplicaSet implements the Kubernetes standard scale sub-command, you could scale the VirtualMachineInstanceReplicaSet using:

oc scale vmirs lab05-cirros-replicaset --replicas 1 --namespace lab-<username>

Horizontal pod autoscaler

The HorizontalPodAutoscaler (HPA)¹ resource can be used to manage the replica count depending on resource usage:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: lab05-cirros-replicaset
spec:
  maxReplicas: 2
  minReplicas: 1
  scaleTargetRef:
    apiVersion: kubevirt.io/v1
    kind: VirtualMachineInstanceReplicaSet
    name: lab05-cirros-replicaset
  targetCPUUtilizationPercentage: 75

This will ensure that the VirtualMachineInstanceReplicaSet is automatically scaled depending on the CPU utilization.

You can check the consumption of your pods with:

oc adm top pod --namespace lab-<username>

NAME                                               CPU(cores)   MEMORY(bytes)   
virt-launcher-lab06-cirros-replicasetck6rw-9s8wd   3m           229Mi

Task 4.1.3: Enable the HorizontalPodAutoscaler

Import another YAML resource by clicking on the + button in the upper right menu. Again, make sure the correct Project is selected.

Paste the following content:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: lab05-cirros-replicaset
spec:
  maxReplicas: 2
  minReplicas: 1
  scaleTargetRef:
    apiVersion: kubevirt.io/v1
    kind: VirtualMachineInstanceReplicaSet
    name: lab05-cirros-replicaset
  targetCPUUtilizationPercentage: 75

Click Create.

Check the status of the HorizontalPodAutoscaler with:

oc get hpa --namespace lab-<username>

NAME                      REFERENCE                                                  TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
lab05-cirros-replicaset   VirtualMachineInstanceReplicaSet/lab05-cirros-replicaset   cpu: 2%/75%   1         2         1          7m44s

Open a second terminal in the webshell and connect to the console of one of your VM instances:

oc get vmi --namespace lab-<username>

NAME                           AGE     PHASE     IP             NODENAME            READY
lab05-cirros-replicasetck6rw   9m47s   Running   10.244.3.171   training-worker-0   True

Pick the VMI and open the console:

virtctl console lab05-cirros-replicaset<pod> --namespace lab-<username>

Start to generate some load. Issue the following command in your webshell:

load() { dd if=/dev/zero of=/dev/null & }; load; read; killall dd

In the other terminal, regularly check the following commands:

oc adm top pod --namespace lab-<username>

oc get hpa --namespace lab-<username>

After a short delay, the HorizontalPodAutoscaler kicks in and scales your replica set to 2:

NAME                      REFERENCE                                                  TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
lab05-cirros-replicaset   VirtualMachineInstanceReplicaSet/lab05-cirros-replicaset   cpu: 283%/75%   1         2         2          11m

And you will see that a second VMI has been started:

oc get vmi --namespace lab-<username>

After the horizontal pod autoscaler scaled up your instances, head over to the console where you generated the load. Hit enter in the console to stop the load generation. By default, the horizontal pod autoscaler tries to stabilize the replica set by using a stabilizationWindowSeconds of 300 seconds. This means that it will keep the replica set stable for at least 300 seconds before issuing a scale down. For more information about the configuration, head over to the Horizontal pod autoscaler documentation .

End of lab

Cleanup resources

You have reached the end of this lab. Please stop your running virtual machines to save resources on the cluster.

Delete your VirtualMachineInstanceReplicaSet:

oc delete vmirs lab05-cirros-replicaset --namespace lab-<username>

Delete the horizontal pod autoscaler

oc delete hpa lab05-cirros-replicaset --namespace lab-<username>

Kubernetes ReplicaSet ↩︎ ↩︎