VM for dpdk

VM Setup for DPDK testing

Host

Check that host CPU has 1Gb hugepages flag:

grep pdpe1gb /proc/cpuinfo | uniq
flags           : [...] pdpe1gb [...]

Edit /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1GB hugepages=30"

Reboot and check that all params have been applied

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-43-generic root=UUID=5f20cc6c-ff53-11e8-8f82-ecf4bbc26e30 ro maybe-ubiquity intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=128.0 transparent_hugepage=never isolcpu=2-11 rcu_nocbs=2-11 nohz=off
Check hugepages
# cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
HugePages_Total: 100
HugePages_Free: 99
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB

Get NIC NUMA node:

# cat /sys/bus/pci/devices/0000\:d8\:00.1/numa_node
1

where device ID can be found with command

lshw -class network -businfo

Get CPUs with needed NUMA node for VM CPU pinning

# lscpu  | grep NUMA
NUMA node(s):          2
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47

VMs

Outside VM

Edit VM settings with virsh using command:

virsh edit <VM NAME>

CPU Pinning (manual configuration, need to add cputune section

<vcpu placement='static' cpuset='1,3,5,7'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='5'/>
<vcpupin vcpu='3' cpuset='7'/>
</cputune>
Add hugepages configuration
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
Enable CPU flags as on host CPU
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>

Video Card MUST be set to virtio (may be configured using virt-manager)

<video>
<model type='virtio' heads='1' primary='yes'>
<acceleration accel3d='no'/>
</model>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>

Inside VM

manual module load

modprobe vfio enable_unsafe_noiommu_mode=Y

OR add to configuration  in file /etc/modprobe.d/vfio.conf

options vfio enable_unsafe_noiommu_mode=Y

Kernel settings 

modprobe vfio enable_unsafe_noiommu_mode=Y
modprobe uio

/etc/grub.d/50-curtin-settings.cfg 

GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity default_hugepagesz=1G hugepagesz=1G hugepages=128.0 transparent_hugepage=never intel_iommu=on iommu=pt"

Check that hugepages have been applyed

AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:       3
HugePages_Free:        2
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB