VM for dpdk

VM Setup for DPDK testing

Host

Check that host CPU has 1Gb hugepages flag:

grep pdpe1gb /proc/cpuinfo | uniq
flags           : [...] pdpe1gb [...]
<PRE>

Edit /etc/default/grub
<PRE>
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1GB hugepages=30"

Reboot and check that all params have been applyed

1. cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic root=UUID=5f20cc6c-ff53-11e8-8f82-ecf4bbc26e30 ro maybe-ubiquity intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=128.0 transparent_hugepage=never isolcpu=2-11 rcu_nocbs=2-11 nohz=off Check hugepages

1. cat /proc/meminfo | grep Huge

AnonHugePages: 0 kB ShmemHugePages: 0 kB HugePages_Total: 100 HugePages_Free: 99 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB Get NIC NUMA node:

1. cat /sys/bus/pci/devices/0000\:d8\:00.1/numa_node

1 where device ID can be found with command lshw -class network -businfo Get CPUs with needed NUMA node for VM CPU pinning

1. lscpu | grep NUMA

NUMA node(s): 2 NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47

VMs: Outside VM

Edit VM settings with virsh using command: virsh edit <VM NAME>

CPU Pinning (manual configuration, need to add cputune section <vcpu placement='static' cpuset='1,3,5,7'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='7'/> </cputune> Add hugepages configuration

 <memoryBacking>
   <hugepages/>
 </memoryBacking>

Enable CPU flags as on host CPU

 <cpu mode='host-model' check='partial'>
   <model fallback='allow'/>
 </cpu>

Video Card MUST be set to virtio (may be configured using virt-manager) <video> <model type='virtio' heads='1' primary='yes'> <acceleration accel3d='no'/> </model> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video>

Inside VM

manual module load modprobe vfio enable_unsafe_noiommu_mode=Y

OR add to configuration  in file /etc/modprobe.d/vfio.conf options vfio enable_unsafe_noiommu_mode=Y

Kernel settings 

modprobe vfio enable_unsafe_noiommu_mode=Y modprobe uio

/etc/grub.d/50-curtin-settings.cfg  GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity default_hugepagesz=1G hugepagesz=1G hugepages=128.0 transparent_hugepage=never intel_iommu=on iommu=pt" Check that hugepages have been applyed AnonHugePages: 0 kB ShmemHugePages: 0 kB HugePages_Total: 3 HugePages_Free: 2 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB