Shihta's Note

© Copyright 2023 | Shihta's Note | All rights reserved

DDP

25 May 2018

Glossary

  • RTE: Run Time Environment. Provides a fast and simple framework for fast packet processing, in a lightweight environment as a Linux* application and using Poll Mode Drivers (PMDs) to increase speed.
  • RSS Receive Side Scaling

IOMMU

  • Enabling

    1
2
3
4
5
6
7

    # modify /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"

# reboot

# check it
dmesg | grep -e DMAR -e IOMMU

Preset - Host

  • instructions

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

    # or run follow instructions after boot up
echo 4    > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages
echo 4    > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

modprobe igb_uio
dpdk-devbind -b igb_uio 0000:1c:00.3
dpdk-devbind -b igb_uio 0000:da:00.0
dpdk-devbind -b igb_uio 0000:da:00.1
dpdk-devbind --status

echo 2 > /sys/bus/pci/devices/0000\:da\:00.0/max_vfs
echo 4 > /sys/bus/pci/devices/0000\:da\:00.1/max_vfs
echo 2 > /sys/bus/pci/devices/0000\:1c\:00.3/max_vfs
rmmod i40evf

echo 512 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

  • lshw -c network -businfo

    1
2
3
4
5
6
7
8
9
10
11
12
13
14

    Bus info          Device          Class          Description
============================================================
pci@0000:1c:00.0  enp28s0f0       network        Ethernet Controller XXV710 for 25GbE SFP28
pci@0000:1c:00.1  enp28s0f1       network        Ethernet Controller XXV710 for 25GbE SFP28
pci@0000:3d:00.0  enp61s0f0       network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:3d:00.1  enp61s0f1       network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:3d:00.2  enp61s0f2       network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:3d:00.3  enp61s0f3       network        Ethernet Connection X722 for 10GbE SFP+
pci@0000:da:00.0                  network        Ethernet Controller XXV710 for 25GbE SFP28
pci@0000:da:00.1                  network        Ethernet Controller XXV710 for 25GbE SFP28
pci@0000:da:02.0                  network        Illegal Vendor ID
pci@0000:da:02.1                  network        Illegal Vendor ID
pci@0000:da:0a.0                  network        Illegal Vendor ID
pci@0000:da:0a.1                  network        Illegal Vendor ID

  • difference between igb_uio & i40e on /sys/bus/pci/devices/0000:da:00.0

    igb_uio i40e
    max_vfs  
      msi_irqs/
      net/
      ptp/
    uio/  

  • status

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

    root@host# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:1c:00.3 'Ethernet Controller X710 for 10GbE SFP+ 1572' drv=igb_uio unused=i40e,vfio-pci
0000:1c:0e.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=i40evf,igb_uio
0000:da:00.0 'Ethernet Controller XXV710 for 25GbE SFP28 158b' drv=igb_uio unused=i40e,vfio-pci
0000:da:02.0 'XL710/X710 Virtual Function 154c' drv=vfio-pci unused=i40evf,igb_uio

root@guest:~# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:00:04.0 'XL710/X710 Virtual Function 154c' drv=igb_uio unused=i40evf
0000:00:09.0 'XL710/X710 Virtual Function 154c' drv=igb_uio unused=i40evf

Disable something by kernel parameters

  • edit /etc/default/grub

    1
2
3
4
5

    # Disable disk
GRUB_CMDLINE_LINUX_DEFAULT="libata.force=2.00:disable"

# Disable IPV6
GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1"

Guest VM

  • mount qcow2

    1
2

    apt-get install libguestfs-tools
guestmount -a /var/lib/libvirt/images/xenserver.qcow2 -m /dev/sda1 /mnt

  • resize qcow2

    • in the host:

      1
2

      qemu-img info u1604-test0.img
qemu-img resize u1604-test0.img +20G

    • in the guest:

      1
2
3

      growpart /dev/vda 1
resize2fs /dev/vda1
xfs_growfs /

  • try to fix the ubuntu hang on boot

    • modify /etc/default/grub & /boot/grub/grub.cfg, remove the console=ttyS0
    • apt-get purge cloud-init
    • Add your ssh key
    • change root password

testpmd

EAL Command-line Options

There are several EAL command-line options that can be used in conjunction with the testpmd, or any other DPDK application. See the DPDK Getting Started Guides for more information on these options.

Such as:

  • -c COREMASK

    Set the hexadecimal bitmask of the cores to run on.

  • -n NUM

    Set the number of memory channels to use.

  • -b, --pci-blacklist domain:bus:devid.func

    Blacklist a PCI device to prevent EAL from using it. Multiple -b options are allowed.

  • -w, --pci-whitelist domain:bus:devid:func

    Add a PCI device in white list.

  • --syslog

    Set the syslog facility.

Testpmd Command-line Options

There are several command-line options for the testpmd applications. They must be separated from the EAL options, with a -- separator.

  • --rxq=N

    Set the number of RX queues per port to N, where 1 <= N <= 65535. The default value is 1.

  • --txq=N

    Set the number of TX queues per port to N, where 1 <= N <= 65535. The default value is 1.

  • --coremask=0xXX

    Set the hexadecimal bitmask of the cores running the packet forwarding test. The master lcore is reserved for command line parsing only and cannot be masked on for packet forwarding.

  • --portmask=0xXX

    Set the hexadecimal bitmask of the ports used by the packet forwarding test.

Some useful instructions in interactive mode

  • start it: testpmd -- -i

  • Get loaded dynamic device personalization (DDP) package info list:

    testpmd> ddp get list (port_id)

  • Display information about dynamic device personalization (DDP) profile:

    testpmd> ddp get info (profile_path)

  • Load a dynamic device personalization (DDP) package:

    testpmd> ddp add (port_id) (package_path[,output_path])

  • Delete a dynamic device personalization package:

    testpmd> ddp del (port_id) (package_path)

  • Display VF statistics:

    testpmd> show vf stats (port_id) (vf_id)

  • Display port info:

    show port info all

Enable DDP hw ptype

testpmd -- -i --rxq=8 --txq=8 --forward-mode=rxonly

1
2
3
4
5

port stop all
ddp add 0 /root/ddp/gtp.pkgo,/root/ddp/gtp.bak
set verbose 8
port start all
start

Scapy example

1
2
3
4
5
6
7
8
9
10
11
12

#!/usr/bin/env python
from scapy.all import *
load_contrib('gtp')

sendp(Ether(dst="FF:FF:FF:FF:FF:FF")/IP(dst="172.16.0.7")/UDP(dport=2123)/GTPHeader(teid=2807)/Raw('x'*20), iface="ens8")
sendp(Ether(dst="FF:FF:FF:FF:FF:FF")/IP(dst="172.16.0.7")/UDP(dport=2123)/GTP_U_Header(teid=0x3456)/Raw('x'*20), iface="ens8")
sendp(Ether(dst="3C:FD:FE:A9:EB:5D")/IP(dst="172.16.0.8")/UDP(dport=2152)/GTP_U_Header(teid=287)/IP()/UDP()/Raw('x'*20), iface="ens8")
sendp(Ether(dst="3C:FD:FE:A9:EB:5D")/IP(dst="172.16.0.8")/UDP(dport=2049)/GTPHeader(teid=2807)/GTPCreatePDPContextRequest(), iface="ens8")
sendp(Ether(dst="3C:FD:FE:A9:EB:5D")/IP(dst="172.16.0.8")/UDP(dport=2025), iface="ens8")
sendp(Ether(dst="FF:FF:FF:FF:FF:FF")/IP(dst="172.16.0.7")/UDP(dport=2152)/GTP_U_Header(teid=0x123456)/Raw('x'*20), iface="ens8")
sendp(Ether(dst="FF:FF:FF:FF:FF:FF")/IP(dst="172.16.0.7")/UDP(dport=2152)/GTP_U_Header(teid=0x123456)/IP()/Raw('x'*20), iface="ens8")
sendp(Ether(dst="FF:FF:FF:FF:FF:FF")/IP(dst="172.16.0.7")/UDP(dport=2152)/GTP_U_Header(teid=0x123456)/IPv6()/Raw('x'*20), iface="ens8")

Result

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=FF:FF:FF:FF:FF:FF - type=0x0800 - length=70 - nb_segs=1 - hw ptype: L3_IPV4_EXT_UNKNOWN TUNNEL_GTPC  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - VXLAN packet: packet type =28816, Destination UDP port =2123, VNI = 10 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=FF:FF:FF:FF:FF:FF - type=0x0800 - length=70 - nb_segs=1 - hw ptype: L3_IPV4_EXT_UNKNOWN TUNNEL_GTPC  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - VXLAN packet: packet type =28816, Destination UDP port =2123, VNI = 52 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=3C:FD:FE:A9:EB:5D - type=0x0800 - length=98 - nb_segs=1 - hw ptype: L3_IPV4_EXT_UNKNOWN TUNNEL_GTPU INNER_L3_IPV4_EXT_UNKNOWN INNER_L4_UDP  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - VXLAN packet: packet type =32912, Destination UDP port =2152, VNI = 1 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=3C:FD:FE:A9:EB:5D - type=0x0800 - length=89 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=3C:FD:FE:A9:EB:5D - type=0x0800 - length=60 - nb_segs=1 - hw ptype: L2_ETHER L3_IPV4_EXT_UNKNOWN L4_UDP  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=FF:FF:FF:FF:FF:FF - type=0x0800 - length=70 - nb_segs=1 - hw ptype: L3_IPV4_EXT_UNKNOWN TUNNEL_GTPU  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - VXLAN packet: packet type =32912, Destination UDP port =2152, VNI = 4660 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=FF:FF:FF:FF:FF:FF - type=0x0800 - length=90 - nb_segs=1 - hw ptype: L3_IPV4_EXT_UNKNOWN TUNNEL_GTPU INNER_L3_IPV4_EXT_UNKNOWN  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - VXLAN packet: packet type =32912, Destination UDP port =2152, VNI = 4660 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD
port 0/queue 0: received 1 packets
  src=5E:4F:EE:11:37:0B - dst=FF:FF:FF:FF:FF:FF - type=0x0800 - length=110 - nb_segs=1 - hw ptype: L3_IPV4_EXT_UNKNOWN TUNNEL_GTPU  - sw ptype: L2_ETHER L3_IPV4 L4_UDP  - l2_len=14 - l3_len=20 - l4_len=8 - VXLAN packet: packet type =32912, Destination UDP port =2152, VNI = 4660 - Receive queue=0x0
  ol_flags: PKT_RX_L4_CKSUM_GOOD PKT_RX_IP_CKSUM_GOOD

Known Issues

  • The uio_pci_generic module is not supported by XL710, since the errata of XL710 states that the Interrupt Status bit is not implemented. 13.28. uio pci generic module bind failed in X710/XL710/XXV710

  • Loading DDP is the prerequisite for below GTP relative cases. Load profile again once restarting testpmd to let software detect this event, although has “profile has already existed” reminder. Operate global reset or lanconf tool to recover original setting. DDP GTP-C/GTP-U

    Global reset trigger reg is 0xb8190, first cmd is core reset, second cmd is global reset:

    1
2

    testpmd > write reg 0 0xb8190 1
testpmd > write reg 0 0xb8190 2
: dpdk

Related Posts

comments powered by Disqus