diff options
Diffstat (limited to 'doc/users-guide/users-guide.adoc')
| -rw-r--r-- | doc/users-guide/users-guide.adoc | 317 |
1 files changed, 263 insertions, 54 deletions
diff --git a/doc/users-guide/users-guide.adoc b/doc/users-guide/users-guide.adoc index 18d8cb8fb..73ec55c41 100644 --- a/doc/users-guide/users-guide.adoc +++ b/doc/users-guide/users-guide.adoc @@ -40,11 +40,29 @@ by abstract handles of type `odp_packet_t`, and packet-related APIs take arguments of this type. What an `odp_packet_t` actually is is not part of the ODP API specification--that is the responsibility of each ODP implementation. +.API Specification Principles +The ODP API specification is designed to permit wide latitude on the part of +implementations while at the same time supporting highly efficient processing, +especially for APIs that are executed frequently. + +Both applications and implementations must comply with the API +specification. If not otherwise documented, results are undefined if an +application acts against the specification. For example, if an application +passes bad parameters to an ODP API one implementation may report an error, +while another may not check them (to maximize performance) but would just +crash while using the bad values. + +Note that many ODP component areas provide an `odp_xxx_capability()` API that +returns platform-specific information regarding valid input to other APIs in +that component. For best portability applications should always use these +capability APIs to determine valid parameter input. + .Summary: ODP API attributes: * Open Source, open contribution, BSD-3 licensed. * Vendor and platform neutral. * Application-centric. Covers functional needs of data plane applications. * Ensures portability by specifying the functional behavior of ODP. +* Both applications and implementations must conform to the API specification. * Defined jointly and openly by application writers and platform implementers. * Architected to be implementable on a wide range of platforms efficiently * Sponsored, governed, and maintained by the Linaro Networking Group (LNG) @@ -133,7 +151,7 @@ of the specification or other minor changes that do not affect either the syntax or semantics of the specification. Such changes in the API specification are expected to be rare. Increments to the minor level represent the introduction of new APIs or functional capabilities, or changes -to he specified syntax or functional behavior of APIs and thus may require +to the specified syntax or functional behavior of APIs and thus may require application source code changes. Such changes are well documented in the release notes for each revision of the specification. Finally, increments to the major level represent significant structural changes that most likely @@ -229,14 +247,14 @@ polled by ODP _Threads_, or can pass through the _Classifier_ and sorted into Queues that represent individual flows. These queues can then be dispatched to application threads via the _Scheduler_. -Threads, in term can invoke various ODP APIs to manipulate packet contents +Threads, in turn can invoke various ODP APIs to manipulate packet contents prior to disposing of them. For output processing, packets make by directly queued to a PktIO output queue or else they may be handed to the _Traffic Manager_ for programmatic _Quality of Service (QoS)_ processing before winding up being transmitted (TX). Note that output interfaces may operate in _loopback_ mode, in which case packets sent to them are re-routed back to the -input lines for "second pass" processing. For example, an incoming IPSec packet -cannot be properly classified (beyond being IPSec traffic) until it is +input lines for "second pass" processing. For example, an incoming IPsec packet +cannot be properly classified (beyond being IPsec traffic) until it is decrypted. Once decrypted and its actual contents made visible, it can then be classified into its real flow. @@ -294,7 +312,7 @@ appropriate type represented by the event. A queue is a message passing channel that holds events. Events can be added to a queue via enqueue operations or removed from a queue via dequeue operations. The endpoints of a queue will vary depending on how it is used. -Queues come in two major types: polled and scheduled, which will be +Queues come in two major types: plain and scheduled, which will be discussed in more detail when the event model is introduced. Queues may also have an associated context, which represents a persistent state for all events that make use of it. These states are what permit threads to perform @@ -386,7 +404,7 @@ To get rate of time source `odp_time_local_res()`, `odp_time_global_res()` are used. To wait, `odp_time_wait_ns()` and `odp_time_wait_until()` are used, during which a thread potentially busy loops the entire wait time. -The `odp_time_t` opaque type represents local or global timestamps. +The `odp_time_t` opaque type represents local, global and PktIO timestamps. ==== Portability Considerations The ODP Time APIs are designed to permit high-precision relative time @@ -540,20 +558,44 @@ calling the terminate functions should only be done when the application is sure it has closed the ingress and subsequently drained all queues, etc. === Startup -The first API that must be called by an ODP application is 'odp_init_global()'. +The first API that must be called by an ODP application is `odp_init_global()`: +[source,c] +----- +int odp_init_global(odp_instance_t *instance, + const odp_init_t *param, + const odp_platform_init_t *platform_param); +----- This takes two pointers. The first, `odp_init_t`, contains ODP initialization data that is platform independent and portable, while the second, `odp_platform_init_t`, is passed unparsed to the implementation to be used for platform specific data that is not yet, or may never be -suitable for the ODP API. +suitable for the ODP API. Each of these parameters is optional and may be +specified as NULL to accept the implementation-defined default initialization +values. -Calling odp_init_global() establishes the ODP API framework and MUST be +Calling `odp_init_global()` establishes the ODP API framework and MUST be called before any other ODP API may be called. Note that it is only called -once per application. Following global initialization, each thread in turn +once per application. A successful call to `odp_init_global()` returns rc = 0 +and sets the `instance` variable supplied as input to the call to a handle +representing this unique ODP instance. + +The `odp_init_t` parameter is used to specify various customizations to the +ODP environment being established by this call. For example, the caller can +specify the maximum number of worker threads it will use, the thread masks +associated with these threads, as well as whether the default logging or +abort functions are to be overridden with an application-supplied handler. + +The application may also provide optimization hints to the ODP implementation +if it knows that it will never use specific ODP feature sets, such as the +packet classifier or traffic manager. Implementations may use such hints to +provide optimized behavior to applications that are known not to need these +features. + +Following global initialization, each thread in turn calls 'odp_init_local()'. This establishes the local ODP thread context for that thread and MUST be called before other ODP APIs may be -called by that thread. The sole argument to this call is the _thread type_, -which is either `ODP_THREAD_WORKER` or `ODP_THREAD_CONTROL`. +called by that thread. The sole argument to this call is the `instance` +variable returned by `odp_init_global()`. === Shutdown Shutdown is the logical reverse of the initialization procedure, with @@ -619,7 +661,7 @@ area and how best to use ODP to achieve these goals. === Portability and Coexistence Because ODP offers a programming _framework_ rather than a programming _environment_, it is designed to be able to work alongside APIs offered by -other frameworks with minimual interference. Therefore when we speak of +other frameworks with minimal interference. Therefore when we speak of portability in an ODP context, we of necessity speak of portability of those portions of the application that make use of ODP APIs. If an application uses non-ODP APIs then those must be taken into consideration as well when @@ -662,7 +704,7 @@ Embedded applications will typically work with a copy of ODP downloaded from a git repository so that it can be configured for the application's precise needs. To specify that the application wishes to use the embedded profile: -`./configure --enable-abi-compat=no ...` +`./configure --disable-abi-compat ...` should be used as part of the ODP configuration options. This allows applications to use inline forms of ODP APIs to give optimal performance @@ -682,11 +724,7 @@ that is installed via `sudo apt-get install` or equivalent command). When using a copy of ODP downloaded from a repository, the cloud profile is selected at configure time: -`./configure --enable-abi-compat=yes ...` - -Note that `--enable-abi-compat=yes` is the default, so this need not be -specified. Unless `no` is specified for this option, the result will be -applications designed to run in the cloud profile. +`./configure --enable-abi-compat ...` === ABI Characteristics An ABI consists of several conventions that ensure that a program compiled @@ -714,10 +752,10 @@ Architecture (ISA), such as x86-64 or AArch64. Binaries cannot directly port between ISAs--that requires a recompilation. Each ODP implementation will identify which ABI definition it supports, if any. -When compiling against an ODP implementation in ABI compabitilty mode, the +When compiling against an ODP implementation in ABI compatibility mode, the resulting binary is automatically binary compatible with all other ODP implementations that share this ABI. For example, for the x86-64 ISA, both -the `odp-linux` and `odp-dpdk` implemtations are a common ABI. +the `odp-linux` and `odp-dpdk` implementations are a common ABI. == Shared memory === Allocating shared memory @@ -756,16 +794,16 @@ shared_data_t *shared_data; shared_data = odp_shm_addr(shm); ---- -The address returned by `odp_shm_addr()` is valid only in the calling ODP -thread space: odp_shm_t handles can be shared between ODP threads and remain -valid within any threads, whereas the address returned by `odp_shm_addr(shm)` -may differ from ODP threads to ODP threads (for the same 'shm' block), and -should therefore not be shared between ODP threads. -For instance, it would be correct to send a shm handle using IPC between two -ODP threads and let each of these thread do their own `odp_shm_addr()` to -get the block address. Directly sending the address returned by -`odp_shm_addr()` from one ODP thread to another would however possibly fail -(the address may have no sense in the receiver address space). +The address returned by `odp_shm_addr()` is normally valid only in the calling +ODP thread space: odp_shm_t handles can be shared between ODP threads and +remain valid within any threads, whereas the address returned by +`odp_shm_addr(shm)` may differ from ODP threads to ODP threads (for the same +'shm' block), and should therefore not be shared between ODP threads. For +instance, it would be correct to send a shm handle using IPC between two ODP +threads and let each of these thread do their own `odp_shm_addr()` to get the +block address. Directly sending the address returned by `odp_shm_addr()` from +one ODP thread to another would however possibly fail (the address may make no +sense in the receiver address space). The address returned by `odp_shm_addr()` is nevertheless guaranteed to be aligned according to the alignment requirements provided at block creation @@ -777,7 +815,13 @@ All shared memory blocks are contiguous in any ODP thread addressing space: as provided in the `odp_shm_reserve()` call) is read and writeable and mapping the shared memory block. There is no fragmentation. -=== Memory behaviour +The exception to this rule is if the `odp_shm_t` is created with the +`ODP_SHM_SINGLE_VA` flag. This requests that `odp_shm_addr()` return the same +virtual address for all ODP threads in this instance. Note that there may be a +performance cost or shm size limit associated with providing this function in +some implementations. + +=== Memory behavior By default ODP threads are assumed to behave as cache coherent systems: Any change performed on a shared memory block is guaranteed to eventually become visible to other ODP threads sharing this memory block. @@ -896,15 +940,6 @@ to other ODP instances running on the same OS. Other ODP instances willing to see this exported memory should use the `odp_shm_import()` ODP function. -==== ODP_SHM_SW_ONLY -This flag tells ODP that the shared memory will be used by the ODP application -software only: no HW (such as DMA, or other accelerator) will ever -try to access the memory. No other ODP call will be involved on this memory -(as ODP calls could implicitly involve HW, depending on the ODP -implementation), except for `odp_shm_lookup()` and `odp_shm_free()`. -ODP implementations may use this flag as a hint for performance optimization, -or may as well ignore this flag. - ==== ODP_SHM_SINGLE_VA This flag is used to guarantee the uniqueness of the address at which the shared memory is mapped: without this flag, a given memory block may be @@ -917,28 +952,28 @@ same value on all ODP threads, for a given memory block, in this case) Note that ODP implementations may have restrictions of the amount of memory which can be allocated with this flag. -== Queues +== Queues and the Scheduler Queues are the fundamental event sequencing mechanism provided by ODP and all ODP applications make use of them either explicitly or implicitly. Queues are created via the 'odp_queue_create()' API that returns a handle of type `odp_queue_t` that is used to refer to this queue in all subsequent APIs that -reference it. Queues have one of two ODP-defined _types_, POLL, and SCHED that -determine how they are used. POLL queues directly managed by the ODP +reference it. Queues have one of two ODP-defined _types_, PLAIN, and SCHED that +determine how they are used. PLAIN queues directly managed by the ODP application while SCHED queues make use of the *ODP scheduler* to provide automatic scalable dispatching and synchronization services. -.Operations on POLL queues +.Operations on PLAIN queues [source,c] ---- -odp_queue_t poll_q1 = odp_queue_create("poll queue 1", ODP_QUEUE_TYPE_POLL, NULL); -odp_queue_t poll_q2 = odp_queue_create("poll queue 2", ODP_QUEUE_TYPE_POLL, NULL); +odp_queue_t plain_q1 = odp_queue_create("poll queue 1", ODP_QUEUE_TYPE_PLAIN, NULL); +odp_queue_t plain_q2 = odp_queue_create("poll queue 2", ODP_QUEUE_TYPE_PLAIN, NULL); ... -odp_event_t ev = odp_queue_deq(poll_q1); +odp_event_t ev = odp_queue_deq(plain_q1); ...do something -int rc = odp_queue_enq(poll_q2, ev); +int rc = odp_queue_enq(plain_q2, ev); ---- -The key distinction is that dequeueing events from POLL queues is an +The key distinction is that dequeueing events from PLAIN queues is an application responsibility while dequeueing events from SCHED queues is the responsibility of the ODP scheduler. @@ -950,7 +985,7 @@ odp_queue_param_init(&qp); odp_schedule_prio_t prio = ...; odp_schedule_group_t sched_group = ...; qp.sched.prio = prio; -qp.sched.sync = ODP_SCHED_SYNC_[NONE|ATOMIC|ORDERED]; +qp.sched.sync = ODP_SCHED_SYNC_[PARALLEL|ATOMIC|ORDERED]; qp.sched.group = sched_group; qp.lock_count = n; /* Only relevant for ordered queues */ odp_queue_t sched_q1 = odp_queue_create("sched queue 1", ODP_QUEUE_TYPE_SCHED, &qp); @@ -1006,8 +1041,8 @@ Three types of queue scheduler synchronization area supported: Parallel, Atomic, and Ordered. ==== Parallel Queues -SCHED queues that specify a sync mode of ODP_SCHED_SYNC_NONE are unrestricted -in how events are processed. +SCHED queues that specify a sync mode of ODP_SCHED_SYNC_PARALLEL are +unrestricted in how events are processed. .Parallel Queue Scheduling image::parallel_queue.svg[align="center"] @@ -1136,11 +1171,179 @@ until the locking order for this lock for all prior events has been resolved and then enters the critical section. The *odp_schedule_order_unlock()* call releases the critical section and allows the next order to enter it. +=== Scheduler Capabilities and Configuration +As with other ODP components, the ODP scheduler offers a range of capabilities +and configuration options that are used by applications to control its +behavior. + +The sequence of API calls used by applications that make use of the scheduler +is as follows: + +.ODP API Scheduler Usage +[source,c] +----- +odp_schedule_capability() +odp_schedule_config_init() +odp_schedule_config() +odp_schedule() +----- +The `odp_schedule_capability()` API returns an `odp_schedule_capability_t` +struct that defines various limits and capabilities offered by this +implementation of the ODP scheduler: + +.ODP Scheduler Capabilities +[source,c] +----- +/** + * Scheduler capabilities + */ +typedef struct odp_schedule_capability_t { + /** Maximum number of ordered locks per queue */ + uint32_t max_ordered_locks; + + /** Maximum number of scheduling groups */ + uint32_t max_groups; + + /** Number of scheduling priorities */ + uint32_t max_prios; + + /** Maximum number of scheduled (ODP_BLOCKING) queues of the default + * size. */ + uint32_t max_queues; + + /** Maximum number of events a scheduled (ODP_BLOCKING) queue can store + * simultaneously. The value of zero means that scheduled queues do not + * have a size limit, but a single queue can store all available + * events. */ + uint32_t max_queue_size; + + /** Maximum flow ID per queue + * + * Valid flow ID range in flow aware mode of scheduling is from 0 to + * this maximum value. So, maximum number of flows per queue is this + * value plus one. A value of 0 indicates that flow aware mode is not + * supported. */ + uint32_t max_flow_id; + + /** Lock-free (ODP_NONBLOCKING_LF) queues support. + * The specification is the same as for the blocking implementation. */ + odp_support_t lockfree_queues; + + /** Wait-free (ODP_NONBLOCKING_WF) queues support. + * The specification is the same as for the blocking implementation. */ + odp_support_t waitfree_queues; + +} odp_schedule_capability_t; +----- +This struct indicates the various scheduling limits supported by this ODP +implementation. Of note is the `max_flow_id` capability, which indicates +whether this implementation is able to operate in _flow aware mode_. + +==== Flow Aware Scheduling +A _flow_ is a sequence of events that share some application-specific meaning +and context. A good example of a flow might be a TCP connection. Various +events associated with that connection, such as packets containing +connection data, as well as associated timeout events used for transmission +control, are logically connected and meaningful to the application processing +that TCP connection. + +Normally a single flow is associated with an ODP queue. That is, all events +on a given queue belong to the same flow. So the queue id is synonymous with +the flow id for those events. However, this is not without drawbacks. Queues +are relatively heavyweight objects and provide both synchronization as well as +user contexts. The number of queues supported by a given implementation +(`max_queues`) may be less than the number of flows an application needs to +be able to process. + +To address these needs, ODP allows schedulers to operate in flow aware mode +in which flow id is maintained separately as part of each event. Two new +APIs: + +* `odp_event_flow_id()` +* `odp_event_flow_id_set()` + +are used to query and set a 32-bit flow id associated with individual events. +The assignment and interpretation of individual flow ids is under application +control. + +When operating in flow aware mode, it is the combination of flow id and +queue id that is used by the scheduler in making scheduling decisions. So, +for example, an Atomic queue would normally be able to dispatch events only a +single thread at a time. When operating in flow aware mode, however, the +scheduler will provide this exclusion only when two events on the same atomic +queue have the same flow id. If they have different flow ids, then they can be +scheduled concurrently to different threads. + +Note that when operating in this mode, any sharing of queue context must be +done with application-provided synchronization controls (similar to how +parallel queues behave). + +==== Scheduler Configuration +After determining the scheduler's capabilities, but before starting to use +the scheduler to process events, applications must configure the scheduler +by calling `odp_schedule_config()`. + +The argument to this call is the `odp_schedule_config_t` struct: + +.ODP Scheduler Configuration +[source,c] +----- +/** + * Schedule configuration + */ +typedef struct odp_schedule_config_t { + /** Maximum number of scheduled queues to be supported. + * + * @see odp_schedule_capability_t + */ + uint32_t num_queues; + + /** Maximum number of events required to be stored simultaneously in + * scheduled queue. This number must not exceed 'max_queue_size' + * capability. A value of 0 configures default queue size supported by + * the implementation. + */ + uint32_t queue_size; + + /** Maximum flow ID per queue + * + * This value must not exceed 'max_flow_id' capability. Flow aware + * mode of scheduling is enabled when the value is greater than 0. + * The default value is 0. + * + * Application can assign events to specific flows by calling + * odp_event_flow_id_set() before enqueuing events into a scheduled + * queue. When in flow aware mode, the event flow id value affects + * scheduling of the event and synchronization is maintained per flow + * within each queue. + * + * Depending on implementation, there may be much more flows supported + * than queues, as flows are lightweight entities. + * + * @see odp_schedule_capability_t, odp_event_flow_id() + */ + uint32_t max_flow_id; + +} odp_schedule_config_t; +----- +The `odp_schedule_config_init()` API should be used to initialize this +struct to its default values. The application then sets whatever +overrides it needs prior to calling `odp_schedule_config()` to activate +them. Note that `NULL` may be passed as the argument to `odp_schedule_config()` +if the application simply wants to use the implementation-defined default +configuration. In the default configuration, the scheduler does not operate in +flow aware mode. + +Once configured, `odp_schedule()` calls can be made to get events. It is +a programming error to attempt to use the scheduler before it has been +configured. + === Queue Scheduling Summary NOTE: Both ordered and parallel queues improve throughput over atomic queues due to parallel event processing, but require that the application take -steps to ensure context data synchronization if needed. +steps to ensure context data synchronization if needed. The same is true for +atomic queues when the scheduler is operating in flow aware mode. include::users-guide-packet.adoc[] @@ -1150,8 +1353,14 @@ include::users-guide-timer.adoc[] include::users-guide-crypto.adoc[] +include::users-guide-ipsec.adoc[] + +include::users-guide-comp.adoc[] + include::users-guide-tm.adoc[] include::users-guide-cls.adoc[] +include::users-guide-utilities-examples.adoc[] + include::../glossary.adoc[] |