README revision aa97dd1c
11. INTRODUCTION
2
3   l4fwd is a sample application to demonstrate and test TLDK TCP/UDP
4   functionalities. Depending on configuration it can do simple send, recv or
5   both over opened TCP/UDP streams. Also it implements ability to do TCP/UDP
6   packet forwarding between different streams, so it is possible to use the
7   l4fwd application as a TCP/UDP proxy.
8
9   The l4fwd application logically is divided into two parts, Back End (BE) and
10   Front End (FE).
11
121.1 Back End (BE)
13
14   BE is responsible for:
15   - RX over DPDK ports and feed them into TCP/UDP TLDK context(s)
16     (via tle_*_rx_bulk).
17
18   - retrieve packets ready to be send out from TCP/UDP TLDK context(s) and TX
19     them over destined DPDK port.
20
21   - Multiple RX/TX queues per port are supported by RSS. Right now the number
22     of TX is same as the number of RX queue.
23
24- Each BE lcore can serve multiple DPDK ports, TLDK TCP/UDP contexts.
25
26   BE configuration record format:
27
28   port=<uint>,addr=<ipv4/ipv6>,masklen=<uint>,mac=<ether><mtu>
29
30   port -    DPDK port id to be used to send packets to the destination.
31             It is an mandatory option.
32   addr -    destination network address. It is an mandatory option.
33   masklen - destination network prefix length. It is an mandatory option.
34   mac -     destination Ethernet address. It is an mandatory option.
35   mtu -     MTU to be used on that port (= application data size + L2/L3/L4
36             headers sizes, default=1514). It is an optional option.
37
38   Below are some example of BE entries
39
40   port=0,masklen=16,addr=192.168.0.0,mac=01:de:ad:be:ef:01
41   port=0,addr=2001:4860:b002::,masklen=64,mac=01:de:ad:be:ef:01
42
43   These examples are also available in be.cfg file.
44
451.2 Front End (FE)
46
47   FE is responsible for:
48   - to open configured TCP/UDP streams and perform send/recv over them.
49     These streams can belong to different TCP/UDP contexts.
50
51   Each lcore can act as BE and/or FE.
52
53   In UDP mode the application can reassemble input fragmented IP packets, and
54   fragment outgoing IP packets (if destination MTU is less then packet size).
55
56   FE configuration record format:
57
58   lcore=<uint>,op=<"rx|tx|echo|fwd">,\
59   laddr=<ip>,lport=<uint16>,raddr=<ip>,rport=<uint16>,\
60   [txlen=<uint>,fwladdr=<ip>,fwlport=<uint16>,fwraddr=<ip>,fwrport=<uint16>,\
61   belcore=<uint>]
62
63   lcore -   EAL lcore to manage that stream(s) in the FE. It is an mandatory
64             option.
65   belcore - EAL lcore to manage that stream(s) in the BE. It is an optional
66             option. lcore and belcore can specify the same cpu core.
67   op -      operation to perform on that stream:
68             "rx" - do receive only on that stream.
69             "tx" - do send only on that stream.
70             "echo" - mimic recvfrom(..., &addr);sendto(..., &addr);
71             on that stream.
72             "fwd" - forward packets between streams.
73             It is an mandatory option.
74   laddr -   local address for the stream to open. It is an mandatory option.
75   lport -   local port for the stream to open. It is an mandatory option.
76   raddr -   remote address for the stream to open. It is an mandatory option.
77   rport -   remote port for the stream to open. It is an mandatory option.
78   txlen -   data length sending in each packet (mandatory for "tx" mode only).
79   fwladdr - local address for the forwarding stream(s) to open
80             (mandatory for "fwd" mode only).
81   fwlport - local port for the forwarding stream(s) to open
82             (mandatory for "fwd" mode only).
83   fwraddr - remote address for the forwarding stream(s) to open
84             (mandatory for "fwd" mode only).
85   fwrport - remote port for the forwarding stream(s) to open
86             (mandatory for "fwd" mode only).
87
88   Below are some example of FE entries
89
90   lcore=3,op=echo,laddr=192.168.1.233,lport=0x8000,raddr=0.0.0.0,rport=0
91
92   lcore=3,op=tx,laddr=192.168.1.233,lport=0x8001,raddr=192.168.1.56,\
93   rport=0x200,txlen=72
94
95   lcore=3,op=rx,laddr=::,lport=0x200,raddr=::,rport=0,txlen=72
96
97   lcore=3,op=fwd,laddr=0.0.0.0,lport=11211,raddr=0.0.0.0,rport=0,\
98   fwladdr=::,fwlport=0,fwraddr=2001:4860:b002::56,fwrport=11211
99
100   These examples are also available in fe.cfg file with some more explanation.
101
1021.3 Configuration files format
103
104   - each record on a separate line.
105   - lines started with '#' are treated as comments.
106   - empty lines (containing whitespace chars only) are ignored.
107   - kvargs style format for each record.
108   - each FE record correspond to at least one stream to be opened
109     (could be multiple streams in case of op="fwd").
110   - each BE record define a ipv4/ipv6 destination.
111
1122. REQUIREMENTS
113
114   DPDK libraries (16.11 or higher)
115   TLDK libraries (1.0)
116   Back-End (BE) configuration file
117   Front-End(FE) configuration file
118
1193. USAGE
120
121   l4fwd <DPDK EAL parameters> -- \
122      -P | --promisc          /* promiscuous mode enabled. */    \
123      -R | --rbufs <num>      /* max recv buffers per stream. */ \
124      -S | --sbufs <num>      /* max send buffers per stream. */ \
125      -s | --streams <num>    /* streams to open per context. */ \
126      -b | --becfg <filename> /* backend configuration file. */  \
127      -f | --fecfg <filename> /* frontend configuration file. */ \
128      -U | --udp /* run the app to handle UDP streams only. */ \
129      -T | --tcp /* run the app to handle TCP streams only. */ \
130      -L | --listen /* open TCP streams in server mode (listen). */ \
131      -a | --enable-arp /* enable arp responses (request not supported) */ \
132      -v | --verbose /* different level of verbose mode */ \
133      <port0_params> <port1_params> ... <portN_params>
134
135   Note that: options -U and -T cannot be used together.
136   Option -L can be used only with option -T.
137
138   portX_params: port=<uint>,lcore=<uint>[-<uint>],[lcore=<uint>[-<uint>],]\
139   [rx_offload=<uint>,tx_offload=<uint>,mtu=<uint>,ipv4=<ipv4>,ipv6=<ipv6>]
140
141   portX_params are used to configure the particular DPDK device
142   (rte_ethdev port), and specify BE lcore that will handle RX/TX from/to the
143   device and manage BE part of corresponding TCP/UDP context.
144   Multiple BE lcore can be specified.
145
146   port -       DPDK port id (RSS are supported when multiple lcores are
147                specified for a port). It is an mandatory option.
148   lcore -      EAL lcore id to handle IO over that port (rx_burst/tx_burst).
149                several ports can be managed by the same lcore, and same port
150                can be managed by more than one lcore.
151                It is an mandatory option. At least one lcore option has to be
152                specified. lcore range can be specified in one lcore option.
153                e.g. lcore=2-3,lcore=6 will enable lcores 2, 3, and 6 to
154                handle BE.
155   rx_offload - RX HW offload capabilities to enable/use on this port.
156                (bitmask of DEV_RX_OFFLOAD_* values). It is an optional option.
157   tx_offload - TX HW offload capabilities to enable/use on this port.
158                (bitmask of DEV_TX_OFFLOAD_* values).
159   mtu -        MTU to be used on that port (= application data size + L2/L3/L4
160                headers sizes, default=1514).
161   ipv4 -       ipv4 address to assign to that port.
162   ipv6 -       ipv6 address to assign to that port.
163
164   At least one of ipv4/ipv6 values have to be specified for each port.
165
1663.1 RSS
167
168   If multiple lcore is specified per DPDK port, the following RSS hash will
169   be enabled on that port:
170      ETH_RSS_UDP, or ETH_RSS_TCP
171
172   The RSS queue qid will handle the stream according to the TCP/UDP source
173   ports of the stream. The qid can be calculated as below
174   
175   qid = (src_port % power_of_2(n)) % n
176   
177   where n is number of lcore used to mane the DPDK port.
178
1794. EXAMPLES
180
1814.1 Sample testbed
182
183+----------------------------+                +-------------------------------+
184|                   TLDK Box |                | Linux Box                     |
185|                            |                |                               |
186|                     port 0 +----------------+ port 0                        |
187|                192.168.1.1 |                | 192.168.1.2                   |
188|          2001:4860:b002::1 |                | 2001:4860:b002::2             |
189|          AA:BB:CC:DD:EE:F1 |                | AA:BB:CC:DD:EE:F2             |
190+----------------------------+                +-------------------------------+
191
1924.2 UDP, "rx" mode, IPv4-only, Single core
193
194   This example shows receiving data from a IPv4 stream. The TLDK UDP server
195   runs on single core where both BE and FE run on cpu core 3.
196
197   be.cfg file contains:
198
199   port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
200
201   fe.cfg file contains (udp server listening to port 6000):
202
203   lcore=3,op=rx,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0
204
205   run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
206
207   l4fwd --lcores='3' -w 01:00.0 -- \
208   --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
209   --becfg be.cfg -U port=0,lcore=3,ipv4=192.168.1.1
210
211   This will create TLDK UDP context on lcore=3 (BE lcore) to manage
212   DPDK port 0. The port 0 will have IPv4 address 192.168.1.1.
213   All the streams will be in server mode and also managed by lcore 3.
214
2154.3 UDP, "echo" mode, IPv6-only, Multicore
216
217   This example shows receiving data from a IPv6 stream and sending the data
218   back through the same IPv6 stream. The TLDK UDP server runs on multicore
219   where BE runs on cpu core 2 and FE runs on cpu core 3.
220
221   be.cfg file contains:
222
223   port=0,masklen=64,addr=2001:4860:b002::,mac=AA:BB:CC:DD:EE:F2
224
225   fe.cfg file contains (udp server listening to port 6000):
226
227   lcore=3,op=rx,laddr=2001:4860:b002::1,lport=6000,raddr=::,rport=0
228
229   run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
230
231   l4fwd --lcores='2,3' -w 01:00.0 -- \
232   --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
233   --becfg be.cfg -U port=0,lcore=2,ipv6=2001:4860:b002::1
234
235   This will create TLDK UDP context on lcore=2 (BE lcore) to manage
236   DPDK port 0. The port 0 will have IPv4 address 2001:4860:b002::1.
237   All the streams will be in server mode and managed by lcore 3 (FE lcore).
238   In this case, the UDP server will send the incoming data back to the sender.
239
2404.4 TCP, "echo" mode, IPv4-only, Multicore, RX-Offload
241
242   This example shows receiving data from a IPv4 stream and sending the data
243   back through the same IPv4 stream. The TLDK TCP server runs on multicore
244   where BE runs on cpu core 2 and FE runs on cpu core 3. The BE also uses
245   receive offload features of the NIC.
246
247   be.cfg file contains:
248
249   port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
250
251   fe.cfg file contains (tcp server listening to port 6000):
252
253   lcore=3,op=echo,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0
254
255   run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
256
257   l4fwd --lcores='2,3' -w 01:00.0 -- \
258   --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
259   --becfg be.cfg -T -L port=0,lcore=2,rx_offload=0xf,tx_offload=0,\
260   ipv4=192.168.1.1
261
262   This will create TLDK TCP context on lcore=2 (BE lcore) to manage
263   DPDK port 0. The port 0 will have IPv4 address 192.168.1.1. The following
264   DPDK RX HW offloads will be enabled on that port.
265      DEV_RX_OFFLOAD_VLAN_STRIP,
266      DEV_RX_OFFLOAD_IPV4_CKSUM,
267      DEV_RX_OFFLOAD_UDP_CKSUM,
268      DEV_RX_OFFLOAD_TCP_CKSUM
269   No HW TX offloads will be enabled.
270   All the streams will be in server mode and managed by lcore 3 (FE core).
271   In this case, the TCP server will send the incoming data back to the sender.
272
2734.5 TCP, "fwd" (proxy) mode, IPv4-to-IPv6, Multi-core, RX-Offload
274
275   This example shows receiving data from a IPv4 stream and forwarding the
276   data to a IPv6 stream. The TLDK TCP server runs on multicore
277   where BE runs on cpu core 2 and FE runs on cpu core 3. The BE also uses
278   receive offload features of the NIC.
279
280   be.cfg file contains:
281
282   port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
283
284   fe.cfg file contains (tcp server listening to port 6000):
285
286   lcore=3,op=fwd,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0,\
287      rladdr=::,lport=0,raddr=2001:4860:b002::2,rport=7000
288
289   run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
290
291   l4fwd --lcores='2,3' -w 01:00.0 -- \
292   --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
293   --becfg be.cfg -T -L port=0,lcore=2,rx_offload=0xf,tx_offload=0,\
294   ipv4=192.168.1.1,ipv6=2001:4860:b002::1
295
296   This will create TLDK TCP context on lcore=2 (BE lcore) to manage
297   DPDK port 0. The port 0 will have IPv4 address 192.168.1.1. The following
298   DPDK RX HW offloads will be enabled on that port.
299      DEV_RX_OFFLOAD_VLAN_STRIP,
300      DEV_RX_OFFLOAD_IPV4_CKSUM,
301      DEV_RX_OFFLOAD_UDP_CKSUM,
302      DEV_RX_OFFLOAD_TCP_CKSUM
303   No HW TX offloads will be enabled.
304   All the streams will be in server mode and managed by lcore 3 (FE core).
305   In this case, the IPv4 TCP server will forward the incoming data to the IPv6
306   TCP server 2001:4860:b002::2 listening to port 7000.
307
3084.6 TCP, "echo" mode, RSS, IPv4-only, Multicore, RX-Offload
309
310   This example shows receiving data from a IPv4 stream and sending the data
311   back through the same IPv4 stream. The TLDK TCP server runs on multicore
312   where BE runs on cpu cores 1-2 and FE runs on cpu core 3. As BE runs on
313   multicore, Receive Side Scaling (RSS) feature will be automatically enabled.
314   The BE also uses receive offload features of the NIC.
315
316   be.cfg file contains:
317
318   port=0,masklen=24,addr=192.168.1.0,mac=AA:BB:CC:DD:EE:F2
319
320   fe.cfg file contains (tcp server listening to port 6000):
321
322   lcore=3,op=echo,laddr=192.168.1.1,lport=6000,raddr=0.0.0.0,rport=0
323   lcore=3,op=echo,laddr=192.168.1.1,lport=6001,raddr=0.0.0.0,rport=0
324
325   run the l4fwd application as below (DPDK port 0 (pci 01:00.0)):
326
327   l4fwd --lcores='1,2,3' -w 01:00.0 -- \
328   --promisc --rbufs 0x100 --sbufs 0x100 --streams 0x100 --fecfg fe.cfg \
329   --becfg be.cfg -T -L port=0,lcore="1-2",rx_offload=0xf,tx_offload=0,\
330   ipv4=192.168.1.1
331
332   This will create TLDK TCP context on lcore=1-2 (BE lcore) to manage
333   DPDK port 0. The port 0 will have IPv4 address 192.168.1.1. The following
334   DPDK RX HW offloads will be enabled on that port.
335      DEV_RX_OFFLOAD_VLAN_STRIP,
336      DEV_RX_OFFLOAD_IPV4_CKSUM,
337      DEV_RX_OFFLOAD_UDP_CKSUM,
338      DEV_RX_OFFLOAD_TCP_CKSUM
339   No HW TX offloads will be enabled.
340   All the streams will be in server mode and managed by lcore 3 (FE core).
341   In this case, the TCP server will send the incoming data back to the sender.
342
343   As RSS is enabled, all the packets with destination port 6000 and 6001 will
344   be managed by HW queue 0 and queue 1 respectively. Please note that RSS
345   is not supported on the interface when both IPv4 and IPv6 are enabled.
346   Only one of IPv4 or IPv6 has to be enabled in the port.
347