cxgbe.rst revision 7b53c036
1..  BSD LICENSE
2    Copyright 2015 Chelsio Communications.
3    All rights reserved.
4
5    Redistribution and use in source and binary forms, with or without
6    modification, are permitted provided that the following conditions
7    are met:
8
9    * Redistributions of source code must retain the above copyright
10    notice, this list of conditions and the following disclaimer.
11    * Redistributions in binary form must reproduce the above copyright
12    notice, this list of conditions and the following disclaimer in
13    the documentation and/or other materials provided with the
14    distribution.
15    * Neither the name of Chelsio Communications nor the names of its
16    contributors may be used to endorse or promote products derived
17    from this software without specific prior written permission.
18
19    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
20    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
21    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
22    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
23    OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
24    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
25    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
26    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
27    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
28    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
31CXGBE Poll Mode Driver
32======================
33
34The CXGBE PMD (**librte_pmd_cxgbe**) provides poll mode driver support
35for **Chelsio T5** 10/40 Gbps family of adapters. CXGBE PMD has support
36for the latest Linux and FreeBSD operating systems.
37
38More information can be found at `Chelsio Communications Official Website
39<http://www.chelsio.com>`_.
40
41Features
42--------
43
44CXGBE PMD has support for:
45
46- Multiple queues for TX and RX
47- Receiver Side Steering (RSS)
48- VLAN filtering
49- Checksum offload
50- Promiscuous mode
51- All multicast mode
52- Port hardware statistics
53- Jumbo frames
54
55Limitations
56-----------
57
58The Chelsio T5 devices provide two/four ports but expose a single PCI bus
59address, thus, librte_pmd_cxgbe registers itself as a
60PCI driver that allocates one Ethernet device per detected port.
61
62For this reason, one cannot whitelist/blacklist a single port without
63whitelisting/blacklisting the other ports on the same device.
64
65Supported Chelsio T5 NICs
66-------------------------
67
68- 1G NICs: T502-BT
69- 10G NICs: T520-BT, T520-CR, T520-LL-CR, T520-SO-CR, T540-CR
70- 40G NICs: T580-CR, T580-LP-CR, T580-SO-CR
71- Other T5 NICs: T522-CR
72
73Prerequisites
74-------------
75
76- Requires firmware version **1.13.32.0** and higher. Visit
77  `Chelsio Download Center <http://service.chelsio.com>`_ to get latest firmware
78  bundled with the latest Chelsio Unified Wire package.
79
80  For Linux, installing and loading the latest cxgb4 kernel driver from the
81  Chelsio Unified Wire package should get you the latest firmware. More
82  information can be obtained from the User Guide that is bundled with the
83  Chelsio Unified Wire package.
84
85  For FreeBSD, the latest firmware obtained from the Chelsio Unified Wire
86  package must be manually flashed via cxgbetool available in FreeBSD source
87  repository.
88
89  Instructions on how to manually flash the firmware are given in section
90  :ref:`linux-installation` for Linux and section :ref:`freebsd-installation`
91  for FreeBSD.
92
93Pre-Installation Configuration
94------------------------------
95
96Config File Options
97~~~~~~~~~~~~~~~~~~~
98
99The following options can be modified in the ``.config`` file. Please note that
100enabling debugging options may affect system performance.
101
102- ``CONFIG_RTE_LIBRTE_CXGBE_PMD`` (default **y**)
103
104  Toggle compilation of librte_pmd_cxgbe driver.
105
106- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG`` (default **n**)
107
108  Toggle display of generic debugging messages.
109
110- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_REG`` (default **n**)
111
112  Toggle display of registers related run-time check messages.
113
114- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_MBOX`` (default **n**)
115
116  Toggle display of firmware mailbox related run-time check messages.
117
118- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_TX`` (default **n**)
119
120  Toggle display of transmission data path run-time check messages.
121
122- ``CONFIG_RTE_LIBRTE_CXGBE_DEBUG_RX`` (default **n**)
123
124  Toggle display of receiving data path run-time check messages.
125
126.. _driver-compilation:
127
128Driver Compilation
129~~~~~~~~~~~~~~~~~~
130
131To compile CXGBE PMD for Linux x86_64 gcc target, run the following "make"
132command:
133
134.. code-block:: console
135
136   cd <DPDK-source-directory>
137   make config T=x86_64-native-linuxapp-gcc install
138
139To compile CXGBE PMD for FreeBSD x86_64 clang target, run the following "gmake"
140command:
141
142.. code-block:: console
143
144   cd <DPDK-source-directory>
145   gmake config T=x86_64-native-bsdapp-clang install
146
147Linux
148-----
149
150.. _linux-installation:
151
152Linux Installation
153~~~~~~~~~~~~~~~~~~
154
155Steps to manually install the latest firmware from the downloaded Chelsio
156Unified Wire package for Linux operating system are as follows:
157
158#. Load the kernel module:
159
160   .. code-block:: console
161
162      modprobe cxgb4
163
164#. Use ifconfig to get the interface name assigned to Chelsio card:
165
166   .. code-block:: console
167
168      ifconfig -a | grep "00:07:43"
169
170   Example output:
171
172   .. code-block:: console
173
174      p1p1      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C0
175      p1p2      Link encap:Ethernet  HWaddr 00:07:43:2D:EA:C8
176
177#. Install cxgbtool:
178
179   .. code-block:: console
180
181      cd <path_to_uwire>/tools/cxgbtool
182      make install
183
184#. Use cxgbtool to load the firmware config file onto the card:
185
186   .. code-block:: console
187
188      cxgbtool p1p1 loadcfg <path_to_uwire>/src/network/firmware/t5-config.txt
189
190#. Use cxgbtool to load the firmware image onto the card:
191
192   .. code-block:: console
193
194      cxgbtool p1p1 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
195
196#. Unload and reload the kernel module:
197
198   .. code-block:: console
199
200      modprobe -r cxgb4
201      modprobe cxgb4
202
203#. Verify with ethtool:
204
205   .. code-block:: console
206
207      ethtool -i p1p1 | grep "firmware"
208
209   Example output:
210
211   .. code-block:: console
212
213      firmware-version: 1.13.32.0, TP 0.1.4.8
214
215Running testpmd
216~~~~~~~~~~~~~~~
217
218This section demonstrates how to launch **testpmd** with Chelsio T5
219devices managed by librte_pmd_cxgbe in Linux operating system.
220
221#. Change to DPDK source directory where the target has been compiled in
222   section :ref:`driver-compilation`:
223
224   .. code-block:: console
225
226      cd <DPDK-source-directory>
227
228#. Load the kernel module:
229
230   .. code-block:: console
231
232      modprobe cxgb4
233
234#. Get the PCI bus addresses of the interfaces bound to cxgb4 driver:
235
236   .. code-block:: console
237
238      dmesg | tail -2
239
240   Example output:
241
242   .. code-block:: console
243
244      cxgb4 0000:02:00.4 p1p1: renamed from eth0
245      cxgb4 0000:02:00.4 p1p2: renamed from eth1
246
247   .. note::
248
249      Both the interfaces of a Chelsio T5 2-port adapter are bound to the
250      same PCI bus address.
251
252#. Unload the kernel module:
253
254   .. code-block:: console
255
256      modprobe -ar cxgb4 csiostor
257
258#. Request huge pages:
259
260   .. code-block:: console
261
262      echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages/nr_hugepages
263
264#. Mount huge pages:
265
266   .. code-block:: console
267
268      mkdir /mnt/huge
269      mount -t hugetlbfs nodev /mnt/huge
270
271#. Load igb_uio or vfio-pci driver:
272
273   .. code-block:: console
274
275      insmod ./x86_64-native-linuxapp-gcc/kmod/igb_uio.ko
276
277   or
278
279   .. code-block:: console
280
281      modprobe vfio-pci
282
283#. Bind the Chelsio T5 adapters to igb_uio or vfio-pci loaded in the previous
284   step:
285
286   .. code-block:: console
287
288      ./tools/dpdk-devbind.py --bind igb_uio 0000:02:00.4
289
290   or
291
292   Setup VFIO permissions for regular users and then bind to vfio-pci:
293
294   .. code-block:: console
295
296      sudo chmod a+x /dev/vfio
297
298      sudo chmod 0666 /dev/vfio/*
299
300      ./tools/dpdk-devbind.py --bind vfio-pci 0000:02:00.4
301
302   .. note::
303
304      Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
305
306#. Start testpmd with basic parameters:
307
308   .. code-block:: console
309
310      ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i
311
312   Example output:
313
314   .. code-block:: console
315
316      [...]
317      EAL: PCI device 0000:02:00.4 on NUMA socket -1
318      EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
319      EAL:   PCI memory mapped at 0x7fd7c0200000
320      EAL:   PCI memory mapped at 0x7fd77cdfd000
321      EAL:   PCI memory mapped at 0x7fd7c10b7000
322      PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8
323      PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
324      Interactive-mode selected
325      Configuring Port 0 (socket 0)
326      Port 0: 00:07:43:2D:EA:C0
327      Configuring Port 1 (socket 0)
328      Port 1: 00:07:43:2D:EA:C8
329      Checking link statuses...
330      PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
331      PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
332      Port 0 Link Up - speed 10000 Mbps - full-duplex
333      Port 1 Link Up - speed 10000 Mbps - full-duplex
334      Done
335      testpmd>
336
337.. note::
338
339   Flow control pause TX/RX is disabled by default and can be enabled via
340   testpmd. Refer section :ref:`flow-control` for more details.
341
342FreeBSD
343-------
344
345.. _freebsd-installation:
346
347FreeBSD Installation
348~~~~~~~~~~~~~~~~~~~~
349
350Steps to manually install the latest firmware from the downloaded Chelsio
351Unified Wire package for FreeBSD operating system are as follows:
352
353#. Load the kernel module:
354
355   .. code-block:: console
356
357      kldload if_cxgbe
358
359#. Use dmesg to get the t5nex instance assigned to the Chelsio card:
360
361   .. code-block:: console
362
363      dmesg | grep "t5nex"
364
365   Example output:
366
367   .. code-block:: console
368
369      t5nex0: <Chelsio T520-CR> irq 16 at device 0.4 on pci2
370      cxl0: <port 0> on t5nex0
371      cxl1: <port 1> on t5nex0
372      t5nex0: PCIe x8, 2 ports, 14 MSI-X interrupts, 31 eq, 13 iq
373
374   In the example above, a Chelsio T520-CR card is bound to a t5nex0 instance.
375
376#. Install cxgbetool from FreeBSD source repository:
377
378   .. code-block:: console
379
380      cd <path_to_FreeBSD_source>/tools/tools/cxgbetool/
381      make && make install
382
383#. Use cxgbetool to load the firmware image onto the card:
384
385   .. code-block:: console
386
387      cxgbetool t5nex0 loadfw <path_to_uwire>/src/network/firmware/t5fw-*.bin
388
389#. Unload and reload the kernel module:
390
391   .. code-block:: console
392
393      kldunload if_cxgbe
394      kldload if_cxgbe
395
396#. Verify with sysctl:
397
398   .. code-block:: console
399
400      sysctl -a | grep "t5nex" | grep "firmware"
401
402   Example output:
403
404   .. code-block:: console
405
406      dev.t5nex.0.firmware_version: 1.13.32.0
407
408Running testpmd
409~~~~~~~~~~~~~~~
410
411This section demonstrates how to launch **testpmd** with Chelsio T5
412devices managed by librte_pmd_cxgbe in FreeBSD operating system.
413
414#. Change to DPDK source directory where the target has been compiled in
415   section :ref:`driver-compilation`:
416
417   .. code-block:: console
418
419      cd <DPDK-source-directory>
420
421#. Copy the contigmem kernel module to /boot/kernel directory:
422
423   .. code-block:: console
424
425      cp x86_64-native-bsdapp-clang/kmod/contigmem.ko /boot/kernel/
426
427#. Add the following lines to /boot/loader.conf:
428
429   .. code-block:: console
430
431      # reserve 2 x 1G blocks of contiguous memory using contigmem driver
432      hw.contigmem.num_buffers=2
433      hw.contigmem.buffer_size=1073741824
434      # load contigmem module during boot process
435      contigmem_load="YES"
436
437   The above lines load the contigmem kernel module during boot process and
438   allocate 2 x 1G blocks of contiguous memory to be used for DPDK later on.
439   This is to avoid issues with potential memory fragmentation during later
440   system up time, which may result in failure of allocating the contiguous
441   memory required for the contigmem kernel module.
442
443#. Restart the system and ensure the contigmem module is loaded successfully:
444
445   .. code-block:: console
446
447      reboot
448      kldstat | grep "contigmem"
449
450   Example output:
451
452   .. code-block:: console
453
454      2    1 0xffffffff817f1000 3118     contigmem.ko
455
456#. Repeat step 1 to ensure that you are in the DPDK source directory.
457
458#. Load the cxgbe kernel module:
459
460   .. code-block:: console
461
462      kldload if_cxgbe
463
464#. Get the PCI bus addresses of the interfaces bound to t5nex driver:
465
466   .. code-block:: console
467
468      pciconf -l | grep "t5nex"
469
470   Example output:
471
472   .. code-block:: console
473
474      t5nex0@pci0:2:0:4: class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00
475
476   In the above example, the t5nex0 is bound to 2:0:4 bus address.
477
478   .. note::
479
480      Both the interfaces of a Chelsio T5 2-port adapter are bound to the
481      same PCI bus address.
482
483#. Unload the kernel module:
484
485   .. code-block:: console
486
487      kldunload if_cxgbe
488
489#. Set the PCI bus addresses to hw.nic_uio.bdfs kernel environment parameter:
490
491   .. code-block:: console
492
493      kenv hw.nic_uio.bdfs="2:0:4"
494
495   This automatically binds 2:0:4 to nic_uio kernel driver when it is loaded in
496   the next step.
497
498   .. note::
499
500      Currently, CXGBE PMD only supports the binding of PF4 for Chelsio T5 NICs.
501
502#. Load nic_uio kernel driver:
503
504   .. code-block:: console
505
506      kldload ./x86_64-native-bsdapp-clang/kmod/nic_uio.ko
507
508#. Start testpmd with basic parameters:
509
510   .. code-block:: console
511
512      ./x86_64-native-bsdapp-clang/app/testpmd -c 0xf -n 4 -w 0000:02:00.4 -- -i
513
514   Example output:
515
516   .. code-block:: console
517
518      [...]
519      EAL: PCI device 0000:02:00.4 on NUMA socket 0
520      EAL:   probe driver: 1425:5401 rte_cxgbe_pmd
521      EAL:   PCI memory mapped at 0x8007ec000
522      EAL:   PCI memory mapped at 0x842800000
523      EAL:   PCI memory mapped at 0x80086c000
524      PMD: rte_cxgbe_pmd: fw: 1.13.32.0, TP: 0.1.4.8
525      PMD: rte_cxgbe_pmd: Coming up as MASTER: Initializing adapter
526      Interactive-mode selected
527      Configuring Port 0 (socket 0)
528      Port 0: 00:07:43:2D:EA:C0
529      Configuring Port 1 (socket 0)
530      Port 1: 00:07:43:2D:EA:C8
531      Checking link statuses...
532      PMD: rte_cxgbe_pmd: Port0: passive DA port module inserted
533      PMD: rte_cxgbe_pmd: Port1: passive DA port module inserted
534      Port 0 Link Up - speed 10000 Mbps - full-duplex
535      Port 1 Link Up - speed 10000 Mbps - full-duplex
536      Done
537      testpmd>
538
539.. note::
540
541   Flow control pause TX/RX is disabled by default and can be enabled via
542   testpmd. Refer section :ref:`flow-control` for more details.
543
544Sample Application Notes
545------------------------
546
547.. _flow-control:
548
549Enable/Disable Flow Control
550~~~~~~~~~~~~~~~~~~~~~~~~~~~
551
552Flow control pause TX/RX is disabled by default and can be enabled via
553testpmd as follows:
554
555.. code-block:: console
556
557   testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 0
558   testpmd> set flow_ctrl rx on tx on 0 0 0 0 mac_ctrl_frame_fwd off autoneg on 1
559
560To disable again, run:
561
562.. code-block:: console
563
564   testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 0
565   testpmd> set flow_ctrl rx off tx off 0 0 0 0 mac_ctrl_frame_fwd off autoneg off 1
566
567Jumbo Mode
568~~~~~~~~~~
569
570There are two ways to enable sending and receiving of jumbo frames via testpmd.
571One method involves using the **mtu** command, which changes the mtu of an
572individual port without having to stop the selected port. Another method
573involves stopping all the ports first and then running **max-pkt-len** command
574to configure the mtu of all the ports with a single command.
575
576- To configure each port individually, run the mtu command as follows:
577
578  .. code-block:: console
579
580     testpmd> port config mtu 0 9000
581     testpmd> port config mtu 1 9000
582
583- To configure all the ports at once, stop all the ports first and run the
584  max-pkt-len command as follows:
585
586  .. code-block:: console
587
588     testpmd> port stop all
589     testpmd> port config all max-pkt-len 9000
590