The Linux RapidIO Subsystem ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The RapidIO standard is a packet-based fabric interconnect standard designed for use in embedded systems. Development of the RapidIO standard is directed by the RapidIO Trade Association (RTA). The current version of the RapidIO specification is publicly available for download from the RTA web-site [1]. This document describes the basics of the Linux RapidIO subsystem and provides information on its major components. 1 Overview ---------- Because the RapidIO subsystem follows the Linux device model it is integrated into the kernel similarly to other buses by defining RapidIO-specific device and bus types and registering them within the device model. The Linux RapidIO subsystem is architecture independent and therefore defines architecture-specific interfaces that provide support for common RapidIO subsystem operations. 2. Core Components ------------------ A typical RapidIO network is a combination of endpoints and switches. Each of these components is represented in the subsystem by an associated data structure. The core logical components of the RapidIO subsystem are defined in include/linux/rio.h file. 2.1 Master Port A master port (or mport) is a RapidIO interface controller that is local to the processor executing the Linux code. A master port generates and receives RapidIO packets (transactions). In the RapidIO subsystem each master port is represented by a rio_mport data structure. This structure contains master port specific resources such as mailboxes and doorbells. The rio_mport also includes a unique host device ID that is valid when a master port is configured as an enumerating host. RapidIO master ports are serviced by subsystem specific mport device drivers that provide functionality defined for this subsystem. To provide a hardware independent interface for RapidIO subsystem operations, rio_mport structure includes rio_ops data structure which contains pointers to hardware specific implementations of RapidIO functions. 2.2 Device A RapidIO device is any endpoint (other than mport) or switch in the network. All devices are presented in the RapidIO subsystem by corresponding rio_dev data structure. Devices form one global device list and per-network device lists (depending on number of available mports and networks). 2.3 Switch A RapidIO switch is a special class of device that routes packets between its ports towards their final destination. The packet destination port within a switch is defined by an internal routing table. A switch is presented in the RapidIO subsystem by rio_dev data structure expanded by additional rio_switch data structure, which contains switch specific information such as copy of the routing table and pointers to switch specific functions. The RapidIO subsystem defines the format and initialization method for subsystem specific switch drivers that are designed to provide hardware-specific implementation of common switch management routines. 2.4 Network A RapidIO network is a combination of interconnected endpoint and switch devices. Each RapidIO network known to the system is represented by corresponding rio_net data structure. This structure includes lists of all devices and local master ports that form the same network. It also contains a pointer to the default master port that is used to communicate with devices within the network. 3. Subsystem Initialization --------------------------- In order to initialize the RapidIO subsystem, a platform must initialize and register at least one master port within the RapidIO network. To register mport within the subsystem controller driver initialization code calls function rio_register_mport() for each available master port. RapidIO subsystem uses subsys_initcall() or device_initcall() to perform controller initialization (depending on controller device type). After all active master ports are registered with a RapidIO subsystem, an enumeration and/or discovery routine may be called automatically or by user-space command. 4. Enumeration and Discovery ---------------------------- 4.1 Overview ------------ RapidIO subsystem configuration options allow users to specify enumeration and discovery methods as statically linked components or loadable modules. An enumeration/discovery method implementation and available input parameters define how any given method can be attached to available RapidIO mports: simply to all available mports OR individually to the specified mport device. Depending on selected enumeration/discovery build configuration, there are several methods to initiate an enumeration and/or discovery process: (a) Statically linked enumeration and discovery process can be started automatically during kernel initialization time using corresponding module parameters. This was the original method used since introduction of RapidIO subsystem. Now this method relies on enumerator module parameter which is 'rio-scan.scan' for existing basic enumeration/discovery method. When automatic start of enumeration/discovery is used a user has to ensure that all discovering endpoints are started before the enumerating endpoint and are waiting for enumeration to be completed. Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering endpoint waits for enumeration to be completed. If the specified timeout expires the discovery process is terminated without obtaining RapidIO network information. NOTE: a timed out discovery process may be restarted later using a user-space command as it is described later if the given endpoint was enumerated successfully. (b) Statically linked enumeration and discovery process can be started by a command from user space. This initiation method provides more flexibility for a system startup compared to the option (a) above. After all participating endpoints have been successfully booted, an enumeration process shall be started first by issuing a user-space command, after an enumeration is completed a discovery process can be started on all remaining endpoints. (c) Modular enumeration and discovery process can be started by a command from user space. After an enumeration/discovery module is loaded, a network scan process can be started by issuing a user-space command. Similar to the option (b) above, an enumerator has to be started first. (d) Modular enumeration and discovery process can be started by a module initialization routine. In this case an enumerating module shall be loaded first. When a network scan process is started it calls an enumeration or discovery routine depending on the configured role of a master port: host or agent. Enumeration is performed by a master port if it is configured as a host port by assigning a host device ID greater than or equal to zero. A host device ID is assigned to a master port through the kernel command line parameter "riohdid=", or can be configured in a platform-specific manner. If the host device ID for a specific master port is set to -1, the discovery process will be performed for it. The enumeration and discovery routines use RapidIO maintenance transactions to access the configuration space of devices. 4.2 Automatic Start of Enumeration and Discovery ------------------------------------------------ Automatic enumeration/discovery start method is applicable only to built-in enumeration/discovery RapidIO configuration selection. To enable automatic enumeration/discovery start by existing basic enumerator method set use boot command line parameter "rio-scan.scan=1". This configuration requires synchronized start of all RapidIO endpoints that form a network which will be enumerated/discovered. Discovering endpoints have to be started before an enumeration starts to ensure that all RapidIO controllers have been initialized and are ready to be discovered. Configuration parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which a discovering endpoint will wait for enumeration to be completed. When automatic enumeration/discovery start is selected, basic method's initialization routine calls rio_init_mports() to perform enumeration or discovery for all known mport devices. Depending on RapidIO network size and configuration this automatic enumeration/discovery start method may be difficult to use due to the requirement for synchronized start of all endpoints. 4.3 User-space Start of Enumeration and Discovery ------------------------------------------------- User-space start of enumeration and discovery can be used with built-in and modular build configurations. For user-space controlled start RapidIO subsystem creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate an enumeration or discovery process on specific mport device, a user needs to write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device registration. For example for machine with single RapidIO controller, mport_ID for that controller always will be 0. To initiate RapidIO enumeration/discovery on all available mports a user may write '-1' (or RIO_MPORT_ANY) into the scan attribute file. 4.4 Basic Enumeration Method ---------------------------- This is an original enumeration/discovery method which is available since first release of RapidIO subsystem code. The enumeration process is implemented according to the enumeration algorithm outlined in the RapidIO Interconnect Specification: Annex I [1]. This method can be configured as statically linked or loadable module. The method's single parameter "scan" allows to trigger the enumeration/discovery process from module initialization routine. This enumeration/discovery method can be started only once and does not support unloading if it is built as a module. The enumeration process traverses the network using a recursive depth-first algorithm. When a new device is found, the enumerator takes ownership of that device by writing into the Host Device ID Lock CSR. It does this to ensure that the enumerator has exclusive right to enumerate the device. If device ownership is successfully acquired, the enumerator allocates a new rio_dev structure and initializes it according to device capabilities. If the device is an endpoint, a unique device ID is assigned to it and its value is written into the device's Base Device ID CSR. If the device is a switch, the enumerator allocates an additional rio_switch structure to store switch specific information. Then the switch's vendor ID and device ID are queried against a table of known RapidIO switches. Each switch table entry contains a pointer to a switch-specific initialization routine that initializes pointers to the rest of switch specific operations, and performs hardware initialization if necessary. A RapidIO switch does not have a unique device ID; it relies on hopcount and routing for device ID of an attached endpoint if access to its configuration registers is required. If a switch (or chain of switches) does not have any endpoint (except enumerator) attached to it, a fake device ID will be assigned to configure a route to that switch. In the case of a chain of switches without endpoint, one fake device ID is used to configure a route through the entire chain and switches are differentiated by their hopcount value. For both endpoints and switches the enumerator writes a unique component tag into device's Component Tag CSR. That unique value is used by the error management notification mechanism to identify a device that is reporting an error management event. Enumeration beyond a switch is completed by iterating over each active egress port of that switch. For each active link, a route to a default device ID (0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written into the routing table. The algorithm recurs by calling itself with hopcount + 1 and the default device ID in order to access the device on the active port. After the host has completed enumeration of the entire network it releases devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint in the system, it sets the Discovered bit in the Port General Control CSR to indicate that enumeration is completed and agents are allowed to execute passive discovery of the network. The discovery process is performed by agents and is similar to the enumeration process that is described above. However, the discovery process is performed without changes to the existing routing because agents only gather information about RapidIO network structure and are building an internal map of discovered devices. This way each Linux-based component of the RapidIO subsystem has a complete view of the network. The discovery process can be performed simultaneously by several agents. After initializing its RapidIO master port each agent waits for enumeration completion by the host for the configured wait time period. If this wait time period expires before enumeration is completed, an agent skips RapidIO discovery and continues with remaining kernel initialization. 4.5 Adding New Enumeration/Discovery Method ------------------------------------------- RapidIO subsystem code organization allows addition of new enumeration/discovery methods as new configuration options without significant impact to to the core RapidIO code. A new enumeration/discovery method has to be attached to one or more mport devices before an enumeration/discovery process can be started. Normally, method's module initialization routine calls rio_register_scan() to attach an enumerator to a specified mport device (or devices). The basic enumerator implementation demonstrates this process. 5. References ------------- [1] RapidIO Trade Association. RapidIO Interconnect Specifications. http://www.rapidio.org. [2] Rapidio TA. Technology Comparisons. http://www.rapidio.org/education/technology_comparisons/ [3] RapidIO support for Linux. http://lwn.net/Articles/139118/ [4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005 http://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf