This is an old revision of the document!
Table of Contents
So there is quite a bit to do when setting up for KVM. In the How To Setup Zramswap And Make Your PC Awesome we touch on setting up zram so that you can get more VM's but there is actually a way to get even better performance that I did not mention in that guide.
Here we will cover some additional tunables so once you have your zram setup so that you can have extra vm's running lets do a bit more so we can have even more memory available.
As like in zram the cost is more cpu cycles so on a very slow cpu that might not be desirable but if you have spare CPU there is no reason not to clear up some ram using things like zram or other kvm switches available. While zram compresses what ram is used to give more available, kvm can deduplicate what is used so that ram is only stored one time across machines.
Lets start by installing virtual machine manager for debian: (by the way if you ever need copy/paste functionality between the vm and your host just install “spice-client-gtk” and “spice-vdagent” package on the vm, then turn it off and back on).
sudo apt-get install virt-manager
There is a lot to install so sit back and wait a while then reboot the machine when done so it can complete.
Once back in the machine you can start enabling some tweaks. First thing is to allow passing through hardware to your VM's so you will want to enable that (as its useful for a variety of reasons) and also to check nested vm's are enabled (normally is).
Nested VM's
First thing to check is if nested vm's are allowed. You can check this by:
cat /etc/modprobe.d/kvm.conf
And you should see this line is returned: 'options kvm_intel nested=1' (for intel cpu's or 'options kvm_amd nested=1' for AMD).
On mine this was already enabled but if you are missing that file or the option first check that your system supports it by either :
cat /sys/module/kvm_intel/parameters/nested or cat /sys/module/kvm_amd/parameters/nested
and if the relevant command returns a 1 then you can enable the function by creating the file or editing it to include the option you need eg:
sudo nano /etc/modprobe.d/kvm.conf
and add either of these two lines depending on CPU:
options kvm_intel nested=1 or options kvm_amd nested=1
Obviously after enabling nested vm support you will need to reboot for it to take effect.
Passthrough Hardware
Next thing we like to tinker with in the ability to pass bits of hardware to the VM so that the VM can use it directly. While it means the host cannot use the hardware passed anymore, it does give the VM direct access to the hardware which has performance benefits, such as passing a GPU to a VM so it can do some encoding using the GPU. While performance is not 100% that of what it is like to run bear metal, its not bad probably like 97% as good and way better than emulation.
To enable this we just need to edit grub:
sudo nano /etc/default/grub
We want to add either intel_iommu=on or amd_iommu=on to the GRUB_CMDLINE_LINUX_DEFAULT line. eg: here is one on a test machine I am using:
You will see that on that line as I have an intel chip I have added the intel_iommu=on parameter onto that line. I also have a couple other parameters such as mitigations=off and nvidia-drm.modeset=1 but these should not be added as they are generally not required and decrease the security of the system. Im simply showing you what I had on a test system so you can see where to make modifications that you may need.
Once we have added this and closed and saved the file when finished with nano we can move onto the next step.
KVM optimizations
So we are already getting some more ram out of the system using some zram but in some cases this wont be super useful (although you can also use both or some hybrid of the two), and another option to optimize might be to deduplicate identical bits of memory to further save RAM. This can be done using the ksmd service so lets configure this and see what to change to get some semi good values.
So two things we are interested are files (everything is a file in Linux) '/sys/kernel/mm/ksm/run' and '/sys/kernel/mm/ksm/pages_to_scan'.
The first file turn on or off the feature, while the second changes the amount of pages it will be scanning every 20ms (by default). So I am interested in being able to find values for this file that are reasonable. To enable the feature we can use
sudo su echo 1 > /sys/kernel/mm/ksm/run
And by default it will scan 100 pages every 20ms
cat /sys/kernel/mm/ksm/pages_to_scan
100 is not a super useful value as it is a little under 400KB or 19.5MB per second or around 1GB per minute. This might sound like a lot but if you have 128GB of RAM assigned to VM's then for it to scan all the way through the RAM would take about 2 hours to do one complete scan through your RAM. While this will eventually get through everything it can be more efficient to find a slightly more reasonable value.
The cost of this is that the CPU will be more taxed. The ksmd service appears to be single threaded so it will use 1 core of your available CPU cores only, so once it reaches 100%, increasing the value will make it less efficient as you are asking it to do more than the CPU can keep up with. During my testing I have found that a value of CPU at 40% (on a single core) is alright if you have a lot of spare cores, or perhaps less if you dont have a large number of free cpu cores. I have 12 on my system I tested from so giving 1 core up for better memory management seemed like an alright trade off (leaving it running at 40% all the time). However I did feel this was a little on the high side and would be happy with a lower value as well. You can experiment yourself based on what I will show you here but probably a good low value would be around 10% CPU, a mid value around 25% and a high value around 40%). The higher the value the more the processor will scale and use power, cooling to compensate etc.
To check what is reasonable to you just increase the pages_to_scan value eg:
echo 4096 > /sys/kernel/mm/ksm/pages_to_scan
On my system a value of 4096 is 16 megabytes every cycle or 800MB a second (4096*4/1024 = 16 and the default sleep is every 20 milliseconds or 50 times a second). Thus every minute 46GB is being scanned and deduplicated if possible. On a modern system with lots of RAM getting through all the RAM in around 15 minutes would be good, so that you are checking the RAM can be optimized again and again. As its not uncommon for systems with around 256GB of RAM these days this value seems fairly reasonable to me at this time. Mine is probably a little high but the final optimization I leave to you, try aim to keep CPU under 40% and the amount of time around 15 minutes or so. That way when you fire up a VM, 15 minutes later some of the RAM can be claimed back. Setting this value too high is detrimental so try experiment to find the lowest value that is reasonable. There is no point scanning so quickly that everything is done every minute for example, try think of a reasonable goal like getting through the RAM in 15 minutes and work to find the lowest value that hits that. My test value above is high and not something I would use unless I was testing to see what I saved quickly just to benchmark if it was worth it at all.
For example after running 2 windows VM's for a short while I can check how much actual memory is saved by
cat /sys/kernel/mm/ksm/pages_shared
My value was 311881 pages which is 311881*4/1024 = 1218MB or about 1GB of memory saved (both VMs together were using 22GB). If I fired up more VM's I would continue to save a bit over each VM. its not massive and does tax the CPU but its something and when used in conjunction with other optimisations can help. I only got 1/22 or about 5% RAM reclaimed from this basic test and had to give up a CPU core running at 40% all the time.
To make the changes permanent, as they are lost on reboot we will need to make a startup script. So once you have experimented and found values you like, then you can move onto doing this. The final value I ended up choosing was 1024 pages each scan so this will be reflected below in the example.
First I make a script in a directory of my choosing. In my case I created a directory on my root /Scripts and added a file in there.
For example:
sudo mkdir /Scripts sudo nano /Scripts/KSMScriptRun.sh
Then added the following to the script:
#!/bin/bash echo 1 > /sys/kernel/mm/ksm/run echo 1024 > /sys/kernel/mm/ksm/pages_to_scan
