path: root/platform/linux-dpdk/README

Copyright (c) 2014, Linaro Limited
All rights reserved.

SPDX-License-Identifier:        BSD-3-Clause


ERRATA:
- DPDK 16.07 and earlier supports pool names with RTE_MEMZONE_NAMESIZE
  characters (including terminating NULL), which is 6 characters less than
  ODP_POOL_NAME_LEN. Names reaching into this interval might collide if the
  first 25 characters are not unique.


1. Rationale
=================================================

This is an effort to port ODP on top of DPDK to accelerate packet processing on
Intel systems equiped with NIC's that supports DPDK.

Prerequisites and considerations:
--------------------------------
- at least an Intel system is needed for compiling and running odp-dpdk
- 8GB of RAM recommended, 4 might be enough too
- it's recommended to obtain an Intel network card (many come in dual port
  configuration available to buy, mostly dedicated to server usage)
- it's also possible to use odp-dpdk for evaluation purposes without a DPDK
  compatible NIC, using the pcap poll mode driver
- DPDK code must be downloaded, configured and compiled, details below
- odp-dpdk has been compiled and tested on Ubuntu 14.04 LTS with 3.19 kernel


Checking if an Intel NIC is supported by DPDK:
---------------------------------------------
DPDK only works on a selected range of Intel network cards. To check if an
installed NIC is supported one needs to get the product id and check if DPDK
supports it.

Getting the product id is possible using lspci:

    lspci -nn | grep Ethernet

A sample output could be:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection [8086:1502] (rev 04)
01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The list of known and supported devices can be found in the DPDK code:
http://dpdk.org/browse/dpdk/tree/lib/librte_eal/common/include/rte_pci_dev_ids.h?id=v16.04

The definition of a known NIC looks like this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#define E1000_DEV_ID_PCH2_LV_LM                 0x1502
...
#define E1000_DEV_ID_I350_COPPER                0x1521
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For a card to be supported a further PCI dev declaration must exist:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RTE_PCI_DEV_ID_DECL_IGB(PCI_VENDOR_ID_INTEL, E1000_DEV_ID_I350_COPPER)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Notice that 82579LM is known but not actually supported by DPDK.


2. Preparing DPDK
=================================================

Fetching the DPDK code:
----------------------
    git clone http://92.243.14.124/git/dpdk ./<dpdk-dir>

Right now odp-dpdk only supports DPDK v16.04. During upgrade, make sure that
PMD constuctors are correctly and fully referred in refer_constructors():
    git tag -l -- will list all the tags available
    git checkout -b 16.04 tags/v16.04

Compile DPDK:
------------
Please refer to http://dpdk.org/doc for more details on how to build DPDK.
Getting started guide for Linux might be of help.
Best effort is done to provide some help on DPDK cmds below for Ubuntu, where it
has been compiled and tested.


This has to be done only once:
    cd <dpdk-dir>
    make config T=x86_64-native-linuxapp-gcc O=x86_64-native-linuxapp-gcc

Set CONFIG_RTE_BUILD_COMBINE_LIBS=y and CONFIG_RTE_BUILD_SHARED_LIB=n in
./x86_64-native-linuxapp-gcc/.config file:
    cd <dpdk-dir>/x86_64-native-linuxapp-gcc
    sed -ri 's,(CONFIG_RTE_BUILD_COMBINE_LIBS=).*,\1y,' .config

Note: dynamic linking does not work with DPDK v1.7.1, there is a workaround
though but until then it's better to disable shared libs:
    sed -ri 's,(CONFIG_RTE_BUILD_SHARED_LIB=).*,\1n,' .config

Note: to use odp-dpdk without DPDK supported NIC's enable pcap pmd:
    sed -ri 's,(CONFIG_RTE_LIBRTE_PMD_PCAP=).*,\1y,' .config

Note: if non-intel SFP's are used in IXGBE, then:
    sed -ri 's,(CONFIG_RTE_LIBRTE_IXGBE_ALLOW_UNSUPPORTED_SFP=).*,\1y,' .config

Now return to parent directory and build DPDK
    cd ..
    make install T=x86_64-native-linuxapp-gcc EXTRA_CFLAGS="-fPIC"

If "conflicting types for skb_set_hash" error happens during DPDK
build, then please knock-off skb_set_hash function from kcompat.h as
shown below. This was seen in Ubuntu 3.13.0-30-generic.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
diff --git a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
index 19df483..78a794a 100644
--- a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
+++ b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
@@ -3845,11 +3845,6 @@ static inline struct sk_buff *__kc__vlan_hwaccel_put_tag(struct sk_buff *skb,
 #if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) )
 #ifdef NETIF_F_RXHASH
 #define PKT_HASH_TYPE_L3 0
-static inline void
-skb_set_hash(struct sk_buff *skb, __u32 hash, __always_unused int type)
-{
-       skb->rxhash = hash;
-}
 #endif /* NETIF_F_RXHASH */
 #endif /* < 3.14.0 */
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This only ensures building DPDK, but traffic is not tested with this build yet.


3. Compile odp-dpdk
=================================================

    cd <odp-dir>
    ./bootstrap
    ./configure --with-platform=linux-dpdk --with-sdk-install-path=<dpdk-dir>/x86_64-native-linuxapp-gcc
    make


4. Prepare DPDK for running odp-dpdk examples
=================================================

Reserve hugepages:
-----------------
To reserve huge pages, which is needed for DPDK, execute following commands
(these are usually needed only once after the system has started):
    sudo sh -c 'echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages'

If you are running on a multi-node machine then hugepages have to be reserved on
each node:
    ls /sys/devices/system/node
    sudo sh -c 'echo 1024 > /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages'

Mount hugetlbfs:
---------------
    sudo mkdir /mnt/huge
    sudo mount -t hugetlbfs nodev /mnt/huge

Insert DPDK kernel module:
-------------------------
DPDK uses userspace poll mode drivers, so it's necessary to insert a couple of
modules to allow DPDK to map the NIC's registers to userspace:
    sudo /sbin/modprobe uio
    ulimit -Sn 2048

    cd <dpdk-dir>
    sudo insmod x86_64-native-linuxapp-gcc/kmod/igb_uio.ko

Bind NIC's to DPDK:
------------------
The DPDK code contains a tool used to bind drivers to the network cards.

    cd <dpdk-dir>
    ./tools/dpdk_nic_bind.py --status

This command produces output that is similar to the one given below:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Network devices using IGB_UIO driver
====================================
0000:05:00.0 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused=
0000:05:00.1 'Ethernet 10G 2P X520 Adapter' drv=igb_uio unused=

Network devices using kernel driver
===================================
0000:01:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth0 drv=bnx2 unused=<none> *Active*
0000:01:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=eth1 drv=bnx2 unused=<none>
0000:07:00.0 'T320 10GbE Dual Port Adapter' if=eth2,eth3 drv=cxgb3 unused=<none>

Other network devices
=====================
<none>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Bind using interface name:
-------------------------
The easiest way is to let the tool automatically switch the regular drivers. For
that the interface must not be active i.e. no IP addresses assigned:
    ifconfig eth0 0
    ifconfig eth1 0
    sudo ./tools/dpdk_nic_bind --bind=igb_uio eth0
    sudo ./tools/dpdk_nic_bind --bind=igb_uio eth1


Bind using PCI ids:
------------------
Another way is to remove the regular drivers and use PCI ids:
    sudo rmmod ixgbe

If the SFP's used are non-intel, then
    sudo modprobe ixgbe allow_unsupported_sfp=1

    sudo ./tools/igb_uio_bind.py --bind=igb_uio 05:00.0
    sudo ./tools/igb_uio_bind.py --bind=igb_uio 05:00.1

Unbind network cards from DPDK:
------------------------------
To restore the NIC's back to kernel use something like this:
    sudo ./tools/igb_uio_bind.py --bind=ixgbe 05:00.0
    sudo ./tools/igb_uio_bind.py --bind=ixgbe 05:00.1


5. Running ODP apps
=================================================

You need to supply the DPDK command line parameters either as a null-terminated
array of char's to odp_global_init()'s platform_params parameter:

	odp_global_init([params], "-n 4");

Or, if it's NULL the platform tries to read the ODP_PLATFORM_PARAMS environment
variable:

	export ODP_PLATFORM_PARAMS="-n 4"

You need to pass "-n [1..4]" at least to specify memory channels. The coremask
(-c) is calculated by ODP-DPDK based on the process affinity at startup. You can
influence that with 'taskset'. DPDK init changes the affinity of the calling
thread, so after it returns the original affinity is restored. Only the first
active core is passed to rte_eal_init(), as the rest would be used for DPDK's
special lcore threads, which are only available through
rte_eal_[mp_]remote_launch(), but not through ODP API's. Nevertheless,
odp_local_init() makes sure for the rest of the DPDK libraries ODP threads look
like proper DPDK threads.
Some useful ODP examples and how to run them:

    l2fwding app:
    sudo ODP_PLATFORM_PARAMS="-n 4" ./test/performance/odp_l2fwd -i 0,1 -m 0 -c 2

    loopback app:
    sudo ODP_PLATFORM_PARAMS="-n 4" ./example/packet/odp_pktio -i 0,1 -m 0 -c 2

	-i 0,1 - interface number
	-m 0   - burst mode
	-c 2   - number of cpus


6. Howto debug DPDK apps on the host
=================================================

For example you need to debug some l2fwd application. Then network configuration
might be:

<veth1-2> <-----> <veth2-1> (iface0 DPDK L2FWD APP iface1) <veth2-3> <-----> <veth3-2>

Where:
vethX-Y - virtual devices for host.

Packet sent to veth1-2 goes to chain and appears on veth3-2

Steps:
Recompile with:
CONFIG_RTE_LIBRTE_PMD_PCAP=y

    ip link add veth1-2 type veth peer name veth2-1
    ip link add veth2-3 type veth peer name veth3-2
    ifconfig veth1-2 up -arp
    ifconfig veth2-1 up -arp
    ifconfig veth2-3 up -arp
    ifconfig veth3-2 up -arp

    mount -t hugetlbfs none /mnt/huge

Finally give l2fwd fake devices:
    ./l2fwd -c '0xf' -n 4 --vdev "eth_pcap0,iface=veth2-1" --vdev="eth_pcap1,iface=veth2-3" -- -p 3

7. Build with devbuild.sh
=================================================

scripts/devbuild.sh contains an example script aimed for developers. It uses
the CI scripts from https://git.linaro.org/lng/check-odp.git to build DPDK and
ODP-DPDK. It can also run "make check" or individual unit tests, but you need to
install CUnit as a prerequisite.
If you have build problems, try to run it and see if it works. An example:
	export REPOS=${PWD}
	git clone https://git.linaro.org/lng/check-odp.git
	git clone https://git.linaro.org/lng/odp-dpdk.git
	odp-dpdk/scripts/devbuild.sh dpdk
	odp-dpdk/scripts/devbuild.sh odp
	odp-dpdk/scripts/devbuild.sh odp-check

It can also run unit tests individually, optionally with gdb. If the first
parameter is not dpdk, odp or odp-check, it tries to run it in
"$CHECK_ODP_DIR/new-build/bin/", with the help of the wrapper script. That's
where example programs are, but you can also use it to run the unit tests, by
traversing the directories:

	odp-dpdk/scripts/devbuild.sh \
	../../../odp-dpdk/test/common_plat/validation/api/atomic/atomic_main
	odp-dpdk/scripts/devbuild.sh \
	../../../odp-dpdk/test/linux-dpdk/validation/api/pktio/pktio_run.sh

The wrapper will prepend the executable with ODP_GDB env. variable, or pass it
down if its name ends ".sh". With prepending your command line with ODP_GDB=gdb
you can run the tests in GDB. If the unit test is wrapped into yet another
shell script (like pktio), it has to do the prepending itself!

8. Upgrading ODP-DPDK to newer ODP API level
=================================================

This repository is based on odp.git, it also retains the history of that. There
are some modifications in configure.ac plus a few other files, but most of the
changes are in platform/linux-dpdk. That directory's Makefile.am builds our
code and the required parts from platform/linux-generic.
This allows us to easily pull the necessary changes from odp.git with git:

git remote add odp_base https://git.linaro.org/lng/odp.git
git pull odp_base master

This will result in a merge commit, and possibly some conflict resolving if
one of the files we touch outside platform/linux-dpdk is modified. After
resolving this conflict you need to implement the changes in the API. Also,
some of the code in our files are exact copy of the linux-generic counterpart,
it's important to port the fixes to the original. The git-transplant script
can do this:

scripts/git-transplant.py platform/linux-generic/ platform/linux-dpdk/ \
[the previous last commit merged from odp.git]..HEAD

It prints the list of prospective patches to be ported. See its comments about
what it does.