aboutsummaryrefslogtreecommitdiff
path: root/platform/linux-generic/include/odp_buffer_internal.h
AgeCommit message (Collapse)Author
2024-02-21linux-gen: pool: add buffer_subtype_set()Lifang Zhang
ML need to use the function to set buffer subtypes. Signed-off-by: Lifang Zhang <lifang.zhang@nokia.com> Reviewed-by: Jere Leppänen <jere.leppanen@nokia.com> Reviewed-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
2022-09-30linux-gen: pool: implement user areaPetri Savolainen
Added new user area parameters and capabilities to buffer, timeout and vector events. Make event header initialization simpler and more explicit that it covers type specific headers as well. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-by: Matias Elo <matias.elo@nokia.com>
2022-05-24linux-gen: event: inline odp_event_flow_id_set()Matias Elo
Inline odp_event_flow_id_set() function implementation. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
2022-05-24linux-gen: event: inline odp_event_flow_id()Matias Elo
Inline odp_event_flow_id() function implementation. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
2022-05-02linux-gen: macros: prefix implementation internal macro namesMatias Elo
Add _ODP_ prefixes to implementation internal helper macros and collect common macros into a single header file (odp_macros_internal.h). Unused macros (DIV_ROUND_UP, odp_container_of) have been removed. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
2021-11-23linux-gen: buffer: use common event headerMatias Elo
Use common _odp_event_hdr_t in odp_buffer_hdr_t. Cache line alignment constraint has been moved to odp_buffer_hdr_t. The implementation internal scheduler and queue interfaces have been modified to use event headers instead of buffer headers. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
2021-10-21linux-gen: fix zero-size array build errorsMatias Elo
Fix "ISO C forbids zero-size array" errors when building with 'pedantic' option. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
2021-08-20linux-gen: sched: limit interface header visibilityPetri Savolainen
Scheduler interface header should be included only when needed. Buffer internal header is included by many files, which do not use scheduler interface. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-by: Matias Elo <matias.elo@nokia.com>
2021-02-05linux-gen: packet: ensure header size with static assertPetri Savolainen
Ensure that buffer and packet header sizes do not accidentally grow to new cache lines when adding new struct fields. Removed comments about data alignment as those may not be always valid (e.g. 32 bit builds). Added debug prints for checking data alignment on the same offsets. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-by: Matias Elo <matias.elo@nokia.com>
2021-01-11linux-gen: packet: fix packet printPetri Savolainen
Packet print function used buffer_snprint, which assumed that event type is buffer and printed an error on packets. Re-implemented buffer and packet print functions to not use buffer_snprint. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-by: Jere Leppänen <jere.leppanen@nokia.com>
2019-11-05linux-gen: ipc: remove ipc_data_offset from buffer headerMatias Elo
Instead of storing IPC packet data offset in odp_buffer_hdr_t, calculate the offset on receive using odp_packet_hdr_t.seg_data pointer and remote pool base address. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
2019-10-23linux-gen: packet: reimplement segmentation using linked listMatias Elo
Packet segmentation has been reimplemented using a linked list. The new implementation is more simple and has the added benefit of reducing packet and buffer header sizes: odp_packet_hdr_t: 384B -> 256B odp_buffer_hdr_t: 256B -> 64B The dynamic packet reference implementation has been temporarily downgraded to a simple packet copy. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
2019-02-08linux-gen: fix internal header code style warningsMatias Elo
Fix checkpatch code style warnings. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2018-12-26linux-gen: sched: dummy flow aware implementationPetri Savolainen
Implement flow aware scheduling API with least possible changes. Scheduler does not care about flow IDs, but synchronizes still on queue level. This is functionally correct, but does provide parallelism between different flows of a queue. So, application does not benefit from using flows, but functions correctly. Maximum number of flows per queue is limited to 256 just to minimize number of bytes used in buffer header. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2018-08-20linux-gen: queue: remove enq/deq from queue interfacePetri Savolainen
Use inlined queue enqueue and dequeue functions instead of internal interface versions. Removed functions from internal interface. Original dequeue multi call remains for overriding dequeue calls (e.g. packet input overrides queue dequeue function but uses the original queue). Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2018-05-10linux-gen: buffer: remove buffer_inlines header filePetri Savolainen
Moved odp_buffer_inlines.h content to odp_buffer_internal.h and removed the file. Other xxx_inlines.h headers do not contain internal functions but only code for API function inlining. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Reviewed-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2018-02-28linux-gen: queue: ring based queue implementationPetri Savolainen
Change from linked list of bursts to a ring implementation. Queues have maximum size but code is simpler and performance is a bit better. This step helps in a potential future step to implement queues with a lockless ring. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2018-02-26linux-gen: packet: single user ptr fieldPetri Savolainen
Remove unnecessary union of user context u64/pointer. Rename the field for better code readability. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2018-02-26update Linaro Copyrights to 2018 yearMaxim Uvarov
Using simple commands: find . -not -path .git -type f -exec \ sed -i -E 's/ (201[0-9]), Linaro/ \1-2018, Linaro/g' {} + find . -not -path .git -type f -exec \ sed -i -E 's/ (201[0-9])(-201[0-9]), Linaro/ \1-2018, Linaro/g' {} + Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org>
2018-01-18linux-gen: align: Fix alignment for typedef definitionsIlias Apalodimas
* typedef struct { } ODP_ALIGN() test_t; is correct * typedef struct ODP_ALIGN() {} test_t; is correct and preferred from gcc standards * typedef struct {} test_t ODP_ALIGN() discards align Had segfauls on gcc-7 using that Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-12-27linux-gen: event: move event subtype to packet headerMatias Elo
Event subtype is only used by packets, so move subtype to packet header to optimize cache usage. This change fixes the ~30% performance penalty in l2fwd (zero-copy dpdk pktio) caused by the initialization of event subtype. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-12-27linux-generic: events subtype implementationDmitry Eremin-Solenikov
Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov@linaro.org> Reviewed-by: Petri Savolainen <petri.savolainen@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-11-16linux-gen: pool: add inlined accessors for selected pool parametersMatias Elo
Add inlined accessors for pool_t members pool_hdl and uarea_size. This enables removing the matching members from odp_buffer_hdr_t, and thus reduces per buffer overhead. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-11-16linux-gen: buffer: remove data size member from odp_buffer_hdr_tMatias Elo
Data size is constant for all buffers from the same pool, so there is no need to store the value in the buffer header. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-10-19linux-gen: pool: modify packet pool config definesMatias Elo
Use separate defines for the maximum number of segments per packet (CONFIG_PACKET_MAX_SEGS) and the number of segments stored in a packet header (CONFIG_PACKET_SEGS_PER_HDR). A separate define is also added for the maximum packet length. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-09-14linux-gen: packet: optimize header layoutPetri Savolainen
Pack most often used data into the first and the last cache line of buffer header, and the first cache line of packet header. Reduce offsets and head-/tailroom fields to 16 bits, since those are in maximum as large as a segment, usually much less. Pack header fields into correct alignments (when possible), so that holes are avoided. These changes reduce packet header size by one cache line. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-09-14linux-gen: packet: implement static referencesPetri Savolainen
Implemented static references with a reference counter. Counter is zero when not used (reference API not used). Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-09-14linux-gen: buffer: optimize header layoutPetri Savolainen
Pack most often used buffer header fields (including seg[0]) into the first cache line. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-09-14linux-gen: packet: re-implement packet segmentationPetri Savolainen
Removed limitation to fixed number of segments (CONFIG_PACKET_MAX_SEGS) per packet. Packet headers carrying segment tables are now linked together with 'next_seg' pointer. Last header containing a segment table is pointed by 'last_seg' pointer. Other than the first segment are not referred directly but with seg_entry_ functions. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-08-25linux-gen: drop _ODP_PKTIO_IPC defineDmitry Eremin-Solenikov
Since 39b3a1681097638 ("linux-gen: remove pktio ipc option from configure") IPC pktio is enabled unconditionally. Drop define guarding conditional compilation of several remaining bits. Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-08-08linux-gen: pool: drop two unused functionsDmitry Eremin-Solenikov
Signed-off-by: Dmitry Eremin-Solenikov <dmitry.ereminsolenikov@linaro.org> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-07-18linux-gen: dpdk: enable zero-copy operationMatias Elo
Implements experimental zero-copy mode for DPDK pktio. This can be enabled with additional '--enable-dpdk-zero-copy' configure flag. This feature has been put behind an extra configure flag as it doesn't entirely adhere to the DPDK API and may behave unexpectedly with untested DPDK NIC drivers. Zero-copy operation has been tested with pcap, ixgbe, and i40e drivers. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-07-13linux-gen: buffer: change buffer handle to pointerPetri Savolainen
Changed buffer, event and timeout handles to be pointers. Packet handles are already pointers. This enabled code optimization as some conversions may be removed and remaining once are just type casts. Signed-off-by: Petri Savolainen <petri.savolainen@linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: packet: replace base_len with constantPetri Savolainen
Only packets used base_len of buffer header. Replace the struct field with constant. This improves performance as the constant data is not read any more and buffer header size is smaller. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: packet: clean and pack packet header structPetri Savolainen
Optimized buffer and packet header struct cache usage by: * removing unused fields * packed remaining fields * arrange fields for more optimal cache line usage Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: packet: optimize concatPetri Savolainen
Optimized concat operation to avoid packet copy when destination packet has room to link source packet segments. Since concat uses extend tail it was also modified to handle variable segment sizes. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: pktio ipc: fix clang buildMaxim Uvarov
clang is more clever on setting and not using variables, so it traps compilation. Also buffers header almost everywhere reference by pointer so size of it should not impact on performance. Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org>
2017-01-12linux-gen: pktio ipc: make it work againMaxim Uvarov
Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org>
2017-01-12linux-gen: sched: remove old ordered queue implementationMatias Elo
Remove old ordered queue code. Replaced temporarily by atomic handling. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: packet: added support for segmented packetsPetri Savolainen
Added support for multi-segmented packets. The first segments is the packet descriptor, which contains all metadata and pointers to other segments. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: pool: ptr instead of hdl in buffer_alloc_multiPetri Savolainen
Improve performance by changing the first parameter of buffer_alloc_multi() to pool pointer (from handle), to avoid double lookup of the pool pointer. Pointer is available for packet alloc calls already. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: pool: optimize buffer allocPetri Savolainen
Round up global pool allocations to a burst size. Cache any extra buffers for future use. Prefetch buffers header which very newly allocated from global pool and will be returned to the caller. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2017-01-12linux-gen: pool: reimplement pool with ringPetri Savolainen
Used the ring data structure to implement pool. Also buffer structure was simplified to enable future driver interface. Every buffer includes a packet header, so each buffer can be used as a packet head or segment. Segmentation was disabled and segment size was fixed to a large number (64kB) to limit the number of modification in the commit. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-09-22linux-gen: config: increase burst sizesPetri Savolainen
Added main burst size configuration option (CONFIG_BURST_SIZE), which helps to keep various burst sizes in sync. Increased common burst size from 8 to 16. This increases scheduled queue throughput about 30-40%. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-09-22linux-gen: queue: burst enq and deqPetri Savolainen
Added support for a buffer header to carry a burst of buffer pointers. The buffer itself is the last one a burst. Burst are built with a enq_multi call, so single enq operations do not benefit from it. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-07-26linux-gen: pool: optimize thread local buffer cacheMatias Elo
Optimize local buffer cache performance which is critical to many use cases - including packet IO. Main parts of the optimization are: * Local cache implemented as an array of buf_hdr pointers, instead of a linked list (which causes a lot of cache misses) * Alloc and free N buffers per operation All above steps are needed to demonstrate the performance upgrade. Some related pool functions (get_buf(), ret_buf(), etc) were moved from pool header to c source file, since those were actual local to the c source file. Also some unused pool variables are removed also. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-07-05linux-gen: std_types: remove extra c headersPetri Savolainen
Removed C header includes which are not needed for API definitions. ODP API depends on uint64_t, etc types in stdint.h, but not other C headers. As an exception, stdbool.h remains since 'true' and 'false' definitions may be used with odp_bool_t. It could be also removed later since true/false are not part of the API (where as uint64_t, etc types). Application needs to include other C library headers directly. ODP API does not specify which headers are included by odp_api.h. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-05-26linux-generic: schedule: rename schedule interface headerPetri Savolainen
Renamed odp_schedule_internal.h to odp_schedule_if.h, since it defines the interface. A new, truly internal header is needed for the scheduler. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-05-26linux-generic: schedule: clean up dependency to config interfacePetri Savolainen
Changed global config #defines into function calls and a scheduler specific define (ORDERED_LOCKS_PER_QUEUE), which needs more work to get rid off. Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
2016-05-19linux-generic: buffer: ifdef ipc_addr_offset member from odp_buffer_hdr_tMatias Elo
Define ipc_addr_offset member of struct odp_buffer_hdr_t only if ipc pktio is enabled to reduce struct size. Signed-off-by: Matias Elo <matias.elo@nokia.com> Reviewed-and-tested-by: Bill Fischofer <bill.fischofer@linaro.org> Signed-off-by: Maxim Uvarov <maxim.uvarov@linaro.org>
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-01 08:48:48 +0000