:doctitle: OpenDataPlane (ODP) Implementers-Guide :description: This document is intended to guide a new OpenDataPlane + implementation developer. :imagesdir: ../images :toc: :numbered!: [abstract] Abstract -------- This document is intended to guide a new ODP implementation developer. Further details about ODP may be found at http://opendataplane.org[ODP homepage] :numbered: == Introduction and Overview ODP consists of three distinct components: [[odp_components]] .OpenDataPlane Components image::odp_components.svg[align="center"] - An abstract API specification. This is the application's writer's view of ODP and defines the syntax and semantics of all ODP routines at a functional level. - An Implementation of the ODP API for a specific target platform. A mapping of the ODP APIs to a specific target platform. This is the focus of this document. - A Validation Test Suite. This is an independent set of routines that when run against an ODP implementation verifies that it correctly implements all of the defined ODP APIs at a functional level. The test suite is used by implementers to self-certify their ODP implementation as well as by third-parties to verify an implementation's claim to be ODP API compliant. The structure of these components, especially the API specification, is more fully defined in the _ODP User's Guide_ === Organization of this Document This document is designed to serve two purposes. Its larger purpose is to provide guidance and practical advice for those wishing to implement ODP on their platform. To help with this, as well as to provide deeper insight into how to think about ODP from an implementer's standpoint, this document also discusses in some depth the design and organization of a specific ODP implementation: the odp-linux reference implementation distributed as part of the main ODP git repository. By grounding theory in practice and discussing a particular example implementation, it is hoped this will provide insight into the trade-offs implementers should consider in approaching how to best implement ODP on their platforms. The section <> discusses the layout of the ODP include tree from an implementer's perspective. Although implementers have wide latitude in how they organize their ODP implementations, it is recommended that this layout be be observed by other implementations. Doing so both simplifies code sharing with the odp-linux reference implementation and also ensure ease of upgrading to future ODP API levels, as this is the basic layout that will be observed by future revisions to the API. This layout also facilitates shared library distribution. The section <> then discusses how validation tests are organized and run to provide ODP API conformance testing for ODP implementations. This is something ODP implementations need to consider from the outset. ODP uses a self-certifying model for API compliance, however this self-certification is performed by the ODP validation test suite (possibly augmented with a vendor's own extensions to this suite). The important point is that because the validation suite is itself free open source code, any potential user of a given ODP implementation is free to do its own validation of a vendor's ODP implementation using this suite. Following this basic material, each ODP API area is then reviewed and its implementation considerations are discussed, illustrating these considerations with discussion of how these are done in *odp-linux*. [[include_structure]] == The Include Structure The implementers view of the include source tree allows the common API definitions and documentation to be reused by all the platforms defined in the tree, but leave the actual definitions to be defined by the specific platform. .Implementers include structure (in repository) ---- ./ ├── include/ │ ├── odp/ │ │ └── api/ │ │ └── spec/ │ │ └── The Public API specification and its documentation. <1> │ │ │ ├── odp_api.h This file should be the only file included by the any ODP │ │ application. <4> │ │ └── platform/ └── / └── include/ ├── Internal header files seen only by the implementation. └── odp/ └── api/ <2> ├── In-line function definitions of the public API for this │ platform seen by the application. │ └── plat/ <3> └── Platform specific types, enums etc as seen by the application but require overriding by the implementation. ---- <1> The specification, defining the ODP application programming interface (API) is held in 'include/odp/api/spec/'. The ODP API is defined by a set of '.h' files including doxygen documentation. <2> Each public specification file is included by a counterpart in 'platform//include/odp/api'. The include of the specification API is AFTER the platform specific definitions to allow the platform to provide definitions that match the underlying hardware. <3> The implementation code may include files from 'platform//include/odp/api/plat' <4> Applications in turn include the include/odp_api.h file which includes the 'platform//include/odp/api' files to provide a complete definition of the API. After ODP installation (make install), the structure becomes as follows: .Installed ODP structure ---- ./ └── include/ ├── odp/ │ └── api/ API In-line for this platform. │ ├── plat/ API Platform specific types. │ └── spec/ The public API specification. └── odp_api.h ---- == ODP library naming recommendations The ODP project supports two implementations *odp-linux* and *odp-dpdk*. The name of the libraries are `libodp-linux` and `libodp-dpdk` respectively. It is recommended that other implementations follow the same schema (*odp-*) to make the representation of the ODP implementations uniform in a distribution. [[validation]] == The Validation Suite ODP provides a comprehensive set of API validation tests that are intended to be used by implementers during development and by application developers to verify that a particular implementation meets their requirements. The list of these tests is expected to grow as ODP grows. Note that validation tests are not typically written by implementers, however their structure and operation needs to be understood so that implementations can properly run them to validate that they conform to the ODP API specification. The only exception to this is platform-specific tests, as described in <>. These may be written by platforms to leverage the CUnit framework for their own internal test needs or to extend the platform-agnostic tests with platform-specific logic. The list of test executables is run by the automake test harness, when running `make check`. Therefore, as required by this harness, each executable should return 0 on success (tests passed), 77 on inconclusive, or any other values on failure. The automake functionality shows a status line (PASSED/FAIL...) for each of the ran test executables. It is expected that ODP developers will need to run tests as early as possible in the development cycle, before all APIs have been implemented. Besides, although there are no APIs that are formally listed as optional, it is also expected that there may be cases where a subset of APIs remain unimplemented on a particular platform. Moreover, some platforms may require specific initialization/termination code prior/after running the standard tests. To accommodate with these platform disparities, the ODP validation has been divided in two distinct areas: * The platform agnostic area, * A platform dependent area (one per platform). === Platform agnostic This grouping defines tests that are expected to be executable and succeed on any platform, though possibly with very different performance, depending on the underlying platform. They are written in plain C code, and may only use functions defined in the standard libC (C99) library (besides the ODP functions being tested, of course). A free C99 specification can be found at the http://www.open-std.org/JTC1/sc22/wg14/www/docs/n1256.pdf[open-std.org] web site. No other languages (like scripting) are allowed as their usage would make assumptions on the platform capability. This area is located at: `test/common_plat/` In this directory, tests are grouped by category: * validation : groups of test defining the ODP compliance * performance : tests checking system responsiveness * miscellaneous Each ODP interface contains modules, where each module groups the set of ODP functions related to the same "topic" for the given interface. Examples of modules for the application interface includes "classification" (API functions dealing with ingress packets classification), time (functions dealing with time, excluding timers which have their own module), timer, ... The complete module list can be seen at: https://www.opendataplane.org/api-documentation/master/api/modules.html[ODP Modules] + Within the platform agnostic area, the validation tests for a given interface are also grouped by modules, matching the ODP interface modules: for instance, `test/common_plat/validation/api` mainly contains a list of directories matching each module name (as defined by the doxygen `@defgroup` or `@ingroup` statement present in each API `.h` file). Within each of these directories, a library (called `libtest.la`) and its associated `.h` file (called `.h`) defines all the test functions for this module as well as few other functions to initialize, terminate, and group the tests. An executable called `_main*`, is also built. It is permissible to generate more than one executable to cover the functionality in the test library for the module. These executable(s) shall call all the tests for this module. See <> for more details. It is important to be aware that the tests defined for a given module (defined in `test/common_plat/validation/api/`) are focused to test the ODP functions belonging to this module, but are not limited to use this module's ODP functions only: many modules needs some interaction with some other module to be tested. The obvious illustration of this is for module "init" whose functions are required by all tests of all other modules (as ODP needs to be initialized to test anything else). There is a `Makefile.am` located at the top of the platform agnostic area. Its role is limited to the construction of the different test libraries and the `_main*` executables. No tests are run from this area when `make check` is performed. ==== CUnit Within a given test executable CUnit is used to run the different tests. The usage of CUnit implies the following structure: * Tests are simple C functions. * Tests are grouped in arrays called test suites. Each test suite can be associated with a suite initialization/termination function(s), called by CUnit before and after the whole suite is run. * An array of test suites (and associated init/term functions) defines the test registry run by the test executable. Moreover, two extra functions can be used to initialize/terminate the test executable (these are not part of CUnit). A test executable return success (0) if every test of each suite succeed. More details about CUnit can be found in the http://cunit.sourceforge.net/doc/index.html[CUnit User's Guide] [[anchor-1]] ==== Module test and naming convention * Tests, i.e. C functions which are used in CUnit test suites are named: *_test_+++*+++* where the suffix identifies the test. * Test arrays, _i.e.,_ arrays of `odp_testinfo_t`, listing the test functions belonging to a suite, are called: *_suite+++[_*]+++* where the possible suffix can be used if many suites are declared. * CUnit suite init and termination functions are called: *+++_suite[_*]_init()+++* and *+++_suite[_*]_term()+++* respectively, where the possible extra middle pattern can be used if many suites are declared. * Suite arrays, _i.e.,_ arrays of `odp_suiteinfo_t` used in executables (CUnit registry) are called *+++_suites[_*]+++* where the possible suffix identifies the executable using it, if any. * Main executable function(s), are called +++_main[_*]+++* where the *possible suffix identifies the executable, if any, using it. * Init/term functions for the whole executable are called *_init* and *_term* All the above symbols are part of the generated `libtest.la` libraries. The generated main executable(s) (named `_+++main[_*]+++`, where the optional suffix is used to distinguish the executables belonging to the same module, if many) simply call(s) the related `_main+++[_*]+++` from the library. === Platform specific These tests are located under `platform//`. There is one such area for each platform implementing ODP. This location will be referred as in the rest of this document. ==== The normal case If the considered platform needs no platform specific tests, this directory simply needs to contain a single `Makefile.am` listing each of the executables (named _main) built from the platform agnostic area. The executables are listed in the automake TEST variable and will therefore be run on "make check". For the linux-generic platform, most tested modules fall into this category: currently, the `test/linux-generic/test/Makefile.am` looks roughly as follows: [source,am] ---- include $(top_srcdir)/test/Makefile.inc TESTS_ENVIRONMENT += TEST_DIR=${top_builddir}/test/common_plat/validation ALL_API_VALIDATION = ${top_builddir}/test/common_plat/validation/api SUBDIRS = if test_vald TESTS = validation/api/pktio/pktio_run.sh \ validation/api/pktio/pktio_run_tap.sh \ $(ALL_API_VALIDATION)/atomic/atomic_main$(EXEEXT) \ $(ALL_API_VALIDATION)/barrier/barrier_main$(EXEEXT) \ $(ALL_API_VALIDATION)/buffer/buffer_main$(EXEEXT) \ $(ALL_API_VALIDATION)/classification/classification_main$(EXEEXT) \ $(ALL_API_VALIDATION)/cpumask/cpumask_main$(EXEEXT) \ $(ALL_API_VALIDATION)/crypto/crypto_main$(EXEEXT) \ $(ALL_API_VALIDATION)/errno/errno_main$(EXEEXT) \ $(ALL_API_VALIDATION)/hash/hash_main$(EXEEXT) \ $(ALL_API_VALIDATION)/init/init_main_ok$(EXEEXT) \ $(ALL_API_VALIDATION)/init/init_main_abort$(EXEEXT) \ $(ALL_API_VALIDATION)/init/init_main_log$(EXEEXT) \ $(ALL_API_VALIDATION)/lock/lock_main$(EXEEXT) \ $(ALL_API_VALIDATION)/packet/packet_main$(EXEEXT) \ $(ALL_API_VALIDATION)/pool/pool_main$(EXEEXT) \ $(ALL_API_VALIDATION)/queue/queue_main$(EXEEXT) \ $(ALL_API_VALIDATION)/random/random_main$(EXEEXT) \ $(ALL_API_VALIDATION)/scheduler/scheduler_main$(EXEEXT) \ $(ALL_API_VALIDATION)/std_clib/std_clib_main$(EXEEXT) \ $(ALL_API_VALIDATION)/thread/thread_main$(EXEEXT) \ $(ALL_API_VALIDATION)/time/time_main$(EXEEXT) \ $(ALL_API_VALIDATION)/timer/timer_main$(EXEEXT) \ $(ALL_API_VALIDATION)/traffic_mngr/traffic_mngr_main$(EXEEXT) \ $(ALL_API_VALIDATION)/shmem/shmem_main$(EXEEXT) \ $(ALL_API_VALIDATION)/system/system_main$(EXEEXT) \ ---- With the exception for module pktio, all other modules testing just involves calling the platform agnostic _main executables (in /test/common_plat/validation/api). ==== Using other languages The pktio module, above, is actually tested using a bash script. This script is needed to set up the interfaces used by the tests. The `pktio_run` script eventually calls the platform agnostic `test/common_plat/validation/api/pktio/pktio_main` after setting up the interfaces needed by the tests. Notice that the path to the script, `validation/api/pktio/pktio_run.sh`, is pointing to a file within the tree so is private to this platform. Any languages supported by the tested platform can be used there, as it will not impact other platforms. The platform "private" executables (such as this script), of course, must also return one of the return code expected by the automake test harness (0 for success, 77 for skipped, other values for errors). ==== Defining test wrappers The pktio case above is actually using a script as wrapper around the "standard" (platform independent) test executable. Wrappers can also be defined by using the `LOG_COMPILER` variable of automake. This is applicable in cases where the same wrapper should be used for more then one test, as the test name is passed has parameter to the wrapper. A wrapper is just a program expecting one argument: the test name. Automake also supports the usage different wrappers based of the executable filename suffix. See https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html[Parallel-Test-Harness] for more information. To add a wrapper around the executed test, just add the following `LOG_COMPILER` definition line in the `/Makefile.am`: [source,am] ---- ... if test_vald LOG_COMPILER = $(top_srcdir)/platform/linux-generic/test/wrapper-script TESTS = pktio/pktio_run \ ... ---- Here follows a dummy example of what wrapper-script could be: [source,bash] ---- #!/bin/bash # The parameter, $1, is the name of the test executable to run echo "WRAPPER!!!" echo "running $1!" # run the test: $1 # remember the test result: res=$? echo "Do something to clean up the mess here :-)" # return the test result. exit $res ---- Note how the above script stores the return code of the test executable to return it properly to the automake test harness. [[platform-specific-tests]] ==== Defining platform specific tests Sometimes, it may be necessary to call platform specific system calls to check some functionality: For instance, testing `odp_cpumask_*` could involve checking the underlying system CPU mask. On linux, such a test would require using the CPU_ISSET macro, which is linux specific. Such a test would be written in `///cpumask/...` The contents of this directory would be very similar to the contents of the platform agnostic side cpu_mask tests (including a `Makefile.am`...), but platform specific test would be written there. `/Makefile.am` would then trigger the building of the platform specific tests (by listing their module name in `SUBDIRS` and therefore calling the appropriate `Makefile.am`) and then it would call both the platform agnostic executable(s) and the platform specific test executable. The shm module of the linux-generic ODP API does have a validation test written this way. You can see it at: `test/linux-generic/validation/api/shmem` ==== Marking validation tests as inactive The general policy is that a full run of the validation suite (a `make check`) must pass at all times. However a particular platform may have one or more test cases that are known to be unimplemented either during development or permanently, so to avoid these test cases being reported as failures it's useful to be able to skip them. This can be achieved by creating a new test executable (still on the platform side), giving the platform specific initialization code the opportunity to modify the registered tests in order to mark unwanted tests as inactive while leaving the remaining tests active. It's important that the unwanted tests are still registered with the test framework to allow the fact that they're not being tested to be recorded. The `odp_cunit_update()` function is intended for this purpose, it is used to modify the properties of previously registered tests, for example to mark them as inactive. Inactive tests are registered with the test framework but aren't executed and will be recorded as inactive in test reports. In `test/common_plat/validation/api/foo/foo.c`, define all validation tests for the 'foo' module: [source,c] ------------------ odp_testinfo_t foo_tests[] = { ODP_TEST_INFO(foo_test_a), ODP_TEST_INFO(foo_test_b), ODP_TEST_INFO_NULL }; odp_suiteinfo_t foo_suites[] = { {"Foo", foo_suite_init, foo_suite_term, foo_tests}, ODP_SUITE_INFO_NULL }; ------------------ In `/validation/api/foo/foo_main.c`, register all the tests defined in the `foo` module, then mark a single specific test case as inactive: [source,c] ------------------ static odp_testinfo_t foo_tests_updates[] = { ODP_TEST_INFO_INACTIVE(foo_test_b), ODP_TEST_INFO_NULL }; static odp_suiteinfo_t foo_suites_updates[] = { {"Foo", foo_suite_init, foo_suite_term, foo_tests_updates}, ODP_SUITE_INFO_NULL }; int foo_main(void) { int ret = odp_cunit_register(foo_suites); if (ret == 0) ret = odp_cunit_update(foo_suites_updates); if (ret == 0) ret = odp_cunit_run(); return ret; } ------------------ So `foo_test_a` will be executed and `foo_test_b` is inactive. It's expected that early in the development cycle of a new implementation the inactive list will be quite long, but it should shrink over time as more parts of the API are implemented. ==== Conditional Tests Some tests may require specific conditions to make sense: for instance, on pktio, checking that sending a packet larger than the MTU is rejected only makes sense if packets can indeed, on that ODP implementation, exceed the MTU. A test can be marked conditional as follows: [source,c] ------------------ odp_testinfo_t foo_tests[] = { ... ODP_TEST_INFO_CONDITIONAL(foo_test_x, foo_check_x), ... ODP_TEST_INFO_NULL }; odp_suiteinfo_t foo_suites[] = { {"Foo", foo_suite_init, foo_suite_term, foo_tests}, ODP_SUITE_INFO_NULL }; ------------------ Foo_test_x is the usual test function. Foo_check_x is the test precondition, i.e. a function returning a Boolean (int). It is called before the test suite is started. If it returns true, the test (foo_test_x) is run. If the precondition function (foo_check_x above) returns false, the test is not relevant (or impossible to perform) and it will be skipped. ================= *Note* Conditional tests can be marked as inactive, keeping the precondition function. Both the test and the precondition function will be skipped, but re-activating the test is then just a matter of changing back the macro from ODP_TEST_INFO_INACTIVE to ODP_TEST_INFO_CONDITIONAL: [source,c] ------------------ ... /* active conditional test */ ODP_TEST_INFO_CONDITIONAL(foo_test_x, foo_check_x), /* inactive conditional test */ ODP_TEST_INFO_INACTIVE(foo_test_y, foo_check_y), ... ------------------ ================= ==== helper usage ==== The tests (both platform agnostic and platform dependent tests) make use of a set of functions defined in a helper library. The helper library tries to abstract and regroup common actions that applications may perform but which are not part of the ODP API (i.e. mostly OS system calls). Using these functions is recommended, as running the tests on a different OS could (hopefully) be as simple as changing the OS related helper lib. In the linux helper, two functions are given to create and join ODP threads: `odph_odpthreads_create()` `odph_odpthreads_join()` These two functions abstract what an ODP thread really is and their usage is recommended as they would be implemented in other OS`s helper lib. Five older functions exist to tie and ODP thread to a specific implementation: `odph_linux_pthread_create()` `odph_linux_pthread_join()` `odph_linux_process_fork_n()` `odph_linux_process_fork()` `odph_linux_process_wait_n()` The usage of these functions should not occur within ODP examples nor tests. The usage of these functions in other application is not recommended. [[typedefs]] == ODP Abstract Types and Implementation Typedefs ODP APIs are defined to be abstract and operate on abstract types. For example, ODP APIs that perform packet manipulation manipulate objects of type `odp_packet_t`. Queues are represented by objects of type `odp_queue_t`, etc. Since the C compiler cannot compile code that has unresolved abstract types, the first task of each ODP implementation is to decide how it wishes to represent each of these abstract types and to supply appropriate `typedef` definitions for them to make ODP applications compilable on this platform. It is recommended that however a platform wishes to represent ODP abstract types, that it do so in a strongly typed manner. Using strong types means that an application that tries to pass a variable of type `odp_packet_t` to an API that expects an argument of type `odp_queue_t`, for example, will result in a compilation error rather than some difficult to debug runtime failure. The *odp-linux* reference implementation defines all ODP abstract types strongly using a set of utility macros contained in `platform/linux-generic/include/odp/api/plat/strong_types.h`. These macros can be used or modified as desired by other implementations to achieve strong typing of their typedefs. === Typedef approaches ODP abstract types serve two distinct purposes that each implementation must consider. First, they shield applications from implementation internals, thus facilitating ODP application portability. Equally important, however, is that implementations choose typdefs and representations that permit the implementation to realize ODP APIs efficiently. This typically means that the handles defined by typedefs are either a pointer to an implementation-defined struct or else an index into an implementation-defined resource table. The two LNG-provided ODP reference implementations illustrate both of these approaches. The *odp-dpdk* implementation follows the former approach (pointers) as this offers the highest performance. For example, in *odp-dpdk* an `odp_packet_t` is a pointer to an `rte_mbuf` struct, which is how DPDK represents packets. The *odp-linux* implementation, by contrast, uses indices as this permits more robust validation support while still being highly efficient. In general, software-based implementations will typically favor pointers while hardware-based implementations will typically favor indices. === ABI Considerations An _Application Binary Interface_ is a specification of the _representation_ of an API that guarantees that applications can move between implementations of an API without recompilation. ABIs thus go beyond the basic source-code portability guarantees provided by APIs to permit binary portability as well. It is important to note that ODP neither defines nor prohibits the specification of ABIs. This is because ODP itself is an _Abstract API Specification_. As noted earlier, abstract APIs cannot be compiled in the absence of completion by an implementation that instantiates them, so the question of ABIs is really a question of representation agreement between multiple ODP implementations. If two or more ODP implementations agree on things like typedefs, endianness, alignments, etc., then they are defining an ABI which would permit ODP applications compiled to that common set of instantiations to inter operate at a binary as well as source level. ==== Traditional ABI ABIs can be defined at two levels. The simplest ABI is within a specific Instruction Set Architecture (ISA). So, for example, an ABI might be defined among ODP implementations for the AArch64 or x86 architectures. This traditional approach is shown here: .Traditional ABI Structure image::abi_traditional.svg[align="center"] In the traditional approach, multiple target platforms agree on a common set of typedefs, etc. so that the resulting output from compilation is directly executable on any platform that subscribes to that ABI. Adding a new platform in this approach simply requires that platform to accept the existing ABI specification. Note that since the output of compilation in a traditional ABI is a ISA-specific binary that applications cannot offer binary compatibility across platforms that use different ISAs. ==== Bitcode based ABI An ABI an also be defined at a higher level by moving to a more sophisticated tool chain (such as is possible using LLVM) that implements a split compilation model. In this model, the output from a compiler is not directly executable. Rather it is a standardized intermediate representation called _bitcode_ that must be further processed to result in an executable image as shown here: .Bitcode ABI Structure image::abi_llvm.svg[align="center"] The key to this model is that the platform linking and optimization that is needed to create a platform executable is a system rather than a developer responsibility. The developer's output is a universal bitcode binary. From here, the library system creates a series of _managed binaries_ that result from performing link-time optimization against a set of platform definitions. When a universal application is to run on a specific target platform, the library system selects the appropriate managed binary for that target platform and loads and runs it. Adding a new platform in this approach involves adding the definition for that platform to the library system so that a managed binary for it can be created and deployed as needed. This occurs without developer involvement since the bitcode format that is input to this backend process is independent of the specific target platform. Note also that since bitcode is not tied to any ISA, applications using bitcode ABIs are binary portable between platforms that use different ISAs. This occurs without loss of efficiency because the process of creating a managed binary is itself a secondary compilation and optimization step. The difference is that performing this step is a system rather than a developer responsibility. == Configuration Each ODP implementation will choose various sizes, limits, and similar internal parameters that are well matched to its design and platform capabilities. However, it is often useful to expose at least some of these parameters and allow users to select specific values to use either at compile time or runtime. This section discusses options for doing this, using the configuration options offered in the `odp-linux` reference implementation as an example. === Static Configuration Static configuration requires the ODP implementation to be recompiled. The reasons for choosing static configuration vary but can involve both design simplicity (_e.g.,_ arrays can be statically configured) or performance considerations (_e.g.,_ including optional debug code). Two approaches to static configuration are `#define` statements and use of autotools. ==== `#define` Statements Certain implementation limits can best be represented by `#define` statements that are set at compile time. Examples of this can be seen in the `odp-linux` reference implementation in the file `platform/linux-generic/include/odp_config_internal.h`. .Compile-time implementation limits (excerpt) [source,c] ----- /* * Maximum number of CPUs supported. Maximum CPU ID is CONFIG_NUM_CPU - 1. */ #define CONFIG_NUM_CPU 256 /* * Maximum number of pools */ #define ODP_CONFIG_POOLS 64 ----- Here two fundamental limits, the number of CPUs supported and the maximum number of pools that can be created via the `odp_pool_create()` API are defined. By using `#define`, the implementation can configure supporting structures (bit strings and arrays) statically, and can also allow static compile-time validation/consistency checks to be done using facilities like `ODP_STATIC_ASSERT()`. This results in more efficient code since these limits need not be computed at runtime. Users are able to change these limits (potentially within documented absolute bounds) by changing the relevant source file and recompiling that ODP implementation. ==== Use of `autotools configure` The ODP reference implementation, like many open source projects, makes use of https://www.gnu.org/software/automake/faq/autotools-faq.html[autotools] to simplify project configuration and support for various build targets. These same tools permit compile-time configuration options to be specified without requiring changes to source files. In addition to the "standard" `configure` options for specifying prefixes, target install paths, etc., the `odp-linux` reference implementation supports a large number of static configuration options that control how ODP is built. Use the `./configure --help` command for a complete list. Here we discuss simply a few for illustrative purposes: `--enable-debug`:: The ODP API specification simply says that "results are undefined" when invalid parameters are passed to ODP APIs. This is done for performance reasons so that implementations don't need to insert extraneous parameter checking that would impact runtime performance in fast-path operations. While this is a reasonable trade off, it can complicate application debugging. To address this, the ODP implementation makes use of the `ODP_ASSERT()` macro that by default disappears at compile time unless the `--enable-debug` configuration option was specified. Running with a debug build of ODP trades off performance for improved parameter/bounds checking to make application debugging easier. `--enable-user-guides`:: By default, building ODP only builds the code. When this option is specified, the supporting user documentation (including this file) is also built. `--disable-abi-compat`:: By default ODP builds with support for the ODP ABI, which permits application binary portability across different ODP implementations targeting the same Instruction Set Architecture (ISA). While this is useful in cloud/host environments, it does involve some performance cost to provide binary compatibility. For embedded use of ODP, disabling ABI compatibility means tighter code can be generated by inlining more of the ODP implementation into the calling application code. When built without ABI compatibility, moving an application to another ODP implementation requires that the application be recompiled. For most embedded uses this is a reasonable trade off in exchange for better application performance on a specific target platform. === Dynamic Configuration While compile-time limits have the advantage of simplicity, they are also not very flexible since they require an ODP implementation to be regenerated to change them. The alternative is for implementations to support _dynamic configuration_ that enables ODP to change implementation behavior without source changes or recompilation. The options for dynamic configuration include: command line parameters, environment variables, and configuration files. ==== Command line parameters Applications that accept a command line passed to their `main()` function can use this to tailor how they use ODP. This may involve self-imposed limits driven by the application or these can specify arguments that are to be passed to ODP initialization via the `odp_init_global()` API. The syntax of that API is: [source,c] ----- int odp_init_global(odp_instance_t *instance, const odp_init_t *params, const odp_platform_init_t *platform_params); ----- and the `odp_init_t` struct is used to pass platform-independent parameters that control ODP behavior while the `odp_platform_init_t` struct is used to pass platform-specific parameters. The `odp-linux` reference platform does not make use of these platform-specific parameters, however the `odp-dpdk` reference implementation uses these to allow applications to pass DPDK initialization parameters to it via these params. ODP itself uses the `odp_init_t` parameters to allow applications to specify override logging and abort functions. These routines are called to perform these functions on behalf of the ODP implementation, thus better allowing ODP to interoperate with application-defined logging and error handling facilities. ==== Environment variables Linux environment variables set via the shell provide a convenient means of passing dynamic configuration values. Each ODP implementation defines which environment variables it looks for and how they are used. For example, the `odp-dpdk` implementation uses the variable `ODP_PLATFORM_PARAMS` as an alternate means of passing DPDK initialization parameters. Another important environment variable that ODP uses is `ODP_CONFIG_FILE` that is used to specify the file path of a _configuration override file_, as described in the next section. ==== Configuration files The https://hyperrealm.github.io/libconfig/[libconfig] library provides a standard set of APIs and tools for parsing configuration files. ODP uses this to provide a range of dynamic configuration options that users may wish to specify. ODP uses a _base configuration file_ that contains system-wide defaults, and is located in the `config/odp-linux-generic.conf` file within the ODP distribution. This specifies a range of overridable configuration options that control things like shared memory usage, queue and scheduler limits and tuning parameters, timer processing options, as well as I/O parameters for various pktio classes. While users of ODP may modify this base file before building it, users can also supply an override configuration file that sets specific values of interest while leaving other parameters set to their defaults as defined by the base configuration file. As noted earlier, the `ODP_CONFIG_FILE` environment variable is used to point to the override file to be used. === Summary There is a place for both static and dynamic configuration in any ODP implementation. This section described some of the most common and discussed how the ODP-supplied reference implementations make use of them. Other ODP implementations are free to copy and/or build on these, or use whatever other mechanisms are native to the platforms supported by those ODP implementations. include::../glossary.adoc[]