RTProc Platform Details
The conventional development of a product applying a new audio signal processing algorithm commonly can be divided into three development phases as depicted in Figure 1.
- Algorithm Exploration: First investigations on a new algorithm in general involve the usage of high level programming languages such as Matlab. Existing libraries are employed for algorithm exploration. Since the algorithm is operated with audio samples in offline manner, aspects such as computational complexity do not play a significant role yet.
- Real-Time Evaluation: In this phase, the algorithm developed in phase I is tested under real-time conditions. A highly efficient programming language such as C/C++ is chosen since processing efficiency is crucial to reach the required program execution speed. In order to identify a suitable realization of the new algorithm, further aspects have to be taken into consideration such as portability to fixed point arithmetic, algorithmic delay, and further optimization of the computational complexity.
- Product Integration: In the final development phase, the new algorithm is realized on the target product platform, e.g., a low power fixed point DSP or microcontroller unit (MCU). This phase can be well prepared in phases 1 and 2 and, in most cases, is not in the main focus of the algorithm developer.
Rapid real-time prototyping as the outcome of phase 2 is very important for the successful introduction of new algorithms: From the developer’s point of view, it enables efficient algorithm optimization, evaluation, and verification in real-world scenarios and customer field tests. In addition to that, a real-time prototype is a highly valuable instrument to market new solutions since potential customers can get in touch with prototypes of future products before entering development phase 3.
Development phase 1 is related to the development of high level programming language software only. A realization of the same algorithm to be tested and optimized under real-time conditions, however, involves a lot of additional programming effort in order to connect the new algorithm to audio hardware as well as to create a control user interface which allows to adapt algorithm parameters during runtime. In practice, during phases 1 and 2, due to the different programming languages, at least two versions of the algorithm exist. Each version is verified and optimized in each of the phases independently. The synchronization of both versions is difficult and time consuming. In summary, the conventional development is inefficient and prohibits short development intervals.
The goal of RTProc is to simplify the development process to reduce the time spent for the creation of real-time demonstrators. Due to a clear separation of functional components in RTProc, all hardware related programming aspects are hidden from the developer, and almost no additional effort is required to switch from the offline to the real-time development phase. In addition, easy-to-use mechanisms for the automatic generation of source code enable to, e.g., attach graphical control interfaces to prototypes or to realize common programming tasks in a very convenient way. Special attention has been payed to realize RTProc as an algorithm development system according to the needs of the algorithm developers: The integration of real-time audio processing functionality in
Matlab enables to smoothly make the transition from offline development to real-time processing by mixing Matlab and C/C++ code without significant additional programming effort due to RTProc code generation tools. Consequently, development phase 1 and 2 are merged such that problems related to different algorithm versions for offline and real-time processing are no longer present. Moreover, tools are provided to measure and optimize the timing behavior during real-time processing and algorithm efficiency in terms of computational complexity.
Principle of RTProc
RTProc is based on a software architecture realized in C/C++ that decomposes audio signal processing applications into three functional units, the driver, the algorithm and the host component, as depicted in Figure 2.
- The Driver Component: The driver component has access to the audio input/output device. In practice, the mechanism to interact with the hardware differs depending on the target platform and, on the PC, the chosen hardware access technology. In order to provide one unique interface for all applications, the driver component maps all basic hardware and system specific functionality to a simple interface, the RTProc driver application programmable interface (driver-API). During real-time processing, the driver component converts all hardware specific data types into hardware independent data types and grants access to the audio samples for further processing through the driver-API.
- The Algorithm Component: The algorithm component has access to the audio buffers prepared by the driver component. Its main purpose is to process the input audio samples according to the algorithm to be realized. The resulting output samples are written to the reserved output buffers. The algorithm component is completely independent from the rest of the audio processing application and the hardware. The runtime interaction block in Fig. 2 enables to modify processing parameters and therefore the algorithmic behavior during audio processing.
- The Host Component:The host component controls the driver and the algorithm component through the algorithm-and the driver-API. It is the key element of all audio processing applications and routes all user interaction to the other two components. During audio signal processing, it transfers all audio data between driver and algorithm component.
In RTProc aided development, the developer realizes the new algorithm as an algorithm component independently from any hardware aspects. Since driver and host components have been implemented as part of RTProc already, it is sufficient to focus on the algorithmic functionality only.
On the PC platform, RTProc is currently realized for the Windows Operating System and Mac OSX. Driver components
based on the ASIO and the CoreAudio technology are supported to enable multichannel audio processing with system latencies of less than 5 ms. Enhanced RTProc driver components to support functionality such as additional real-time audio file routing,
automatic resampling and asynchronous audio processing are available to cover the whole bandwidth of applications.
For development purposes, two different host components are available, one to integrate real-time processing in Matlab (the Matlab host), and a generic library to offer the opportunity to add RTProc to customized applications (the generic host). GUI host applications make use of the Matlab and the generic host component, respectively, to provide graphical user interfaces (GUI) for real-time audio processing control.
RTProc Aided Algorithm Development
RTProc aided algorithm development significantly benefits from the merge of the development phases 1 and 2. In that context, the different operation modes of the RTProc Matlab host application, the offline, the hook up, and the high efficiency mode, play an important role:
In general, the developer starts algorithm exploration in the offline operation mode of the host. In this mode, audio files stored on the harddrive or in the workspace are processed in a convenient way and stored back to harddrive or workspace. In order to realize a specific algorithm, the algorithm developer provides a callback function in analogy to the algorithm component main signal processing function. This callback function is called by the Matlab host application on a buffer-by-buffer basis. All other functionality, e.g., the decomposition of the input signal into buffers is managed by the Matlab host application.
If the buffer-by-buffer processing callback function execution time is sufficiently short, the Matlab implementation of the algorithm can also be executed in real-time. For this purpose, the RTProc Matlab host application can be operated in the hook up mode: The RTProc software architecture including the RTProc driver component is loaded in the background of Matlab and synchronizes Matlab to the soundcard I/O. Compared to the offline mode, the Matlab signal processing callback provided by the algorithm developer reads and writes audio samples from and to the audio device. Due to the required synchronization overhead, this purely Matlab based real-time audio signal processing approach is applicable only in case of algorithms with a low computational complexity and for applications with moderate latency constraints. In all other situations, the realization of an algorithm component in C/C++ according to the RTProc software architecture is the more efficient approach to follow. In order to enable a smooth transition from the offline version of the algorithm in Matlab to a C/C++ implementation, the RTProc Matlab host application enables to mix Matlab functions and functions which are part of the C/C++ algorithm component arbitrarily in the offline mode. Consequently, the code transition process of an algorithm from Matlab to C/C++ can be realized very efficiently as follows:
At the beginning, most algorithm functionality has been realized and verified in Matlab as shown in Fig. 3 a). In order to convert the Matlab code into a C/C++ based real-time version of the algorithm, all functional parts are slowly ported to C/C++ in a step-by-step procedure and from inside to outside. Two typical situations in the transition process are demonstrated by the examples in Fig. 3: In part b) of the figure, the first functions have been converted to C/C++ in an early development stage, whereas in part c), nearly the full algorithm has been converted to C/C++ in an advanced development stage. Since in this smooth transition process, each single function can be easily verified against its Matlab counterpart, implementation failures, which are often very time consuming to find in conventional development, are avoided. The effort to call C/C++ functions from within Matlab is only minimal since all required code is generated automatically by an RTProc tool based on simple meta information provided by the algorithm developer.
Once all functionality is completely implemented in C/C++ and verified, the Matlab Host GUI can be switched into the high efficiency operation mode. In this mode, the time-critical signal processing tasks are executed completely in the background of Matlab without interfering with any Matlab functionality. No modification of source code or recompilation of the algorithm component is required to run the algorithm in real-time in the high efficiency operation mode. And even though signal processing is done in the background, the algorithm developer benefits from the RTProc Matlab integration in the high efficiency mode: A non-time-critical bidirectional data link between Matlab and the C/C++ based algorithm component in the background enables to, e.g., compute processing parameters such as fixed filter coefficients in Matlab and to send those parameters to the algorithm component. Here, the parameters instantaneously have an impact on the signal processing algorithm.
In the last phase of the development, the real-time prototype should be prepared for deployment to customers. In this context, an operation independently from Matlab as a stand-alone real-time demonstrator can be realized based on the deployable host application. Purely C/C++ based algorithm components which have been developed with the Matlab host application can be loaded without any source code modification or recompilation.
RTProc and Embedded Development
Once the real-time prototype is complete (end of phase 2), in phase 3 of the development process the C/C++ based implementation of the new algorithm shall be ported to an embedded device. It is very useful to have optimized the code (e.g. the fixed point implementation of the new algorithm) based on the PC based C/C++ implementation as long as possible since optimization and functionality verification is much simpler in the PC environment. However, in order to start with phase 3, at first, the C code is taken as it is and compiled for the embedded digital signal processor (DSP) or microcontroller (MCU). In general, the new algorithm will function propperly on the DSP but will not be optimal, e.g. due to different arithmetics of the given hardware or sub optimal computational complexity. In order to optimize and verify optimal functionality, RTProc provides a mechanism to integrate the DSP or MCU processing routines following a hardware-in-the-loop approach: Fractions of the overall functionality of the new algorithm can be addressed by using so-called remote-calls to execute specific function on the DSP, in C/C++ or in Matlab in parallel. Input data is transferred from the RTProc system to the DSP or MCU based on a serial RS232 or TCP based cable link, the function is executed remotely and the output data is passed back to the RTProc system on the PC. Since output data is available from all implementations of the same functionality, verification, assessment as well as optimization of the DSP or MCU realization is very easy.
This methodology allows the same approach which was described here to port the PC based real-time version of a new algorithm to the embedded device following a module-by-module strategy which allows to shorten the development time and ensure propper functionality without a significant programming effort.
In order to connect different embedded boards to the RTProc system, the required low level functionality is available for serial links (RS232) and TCP based links. So far, the low level functionality has been developed for two embedded targets, the ADSP-21369 EZ KIT evaluation board and the OMAP-L137 evaluation kit.
This principle is shown exemplarily in Figure 4.
In that case, an embedded device is connected to the RTProc development system and allows to execute remote calls from within the C/C++ portion (shown in the figure) as well as from the Matlab portion of the algorithm realization.
Note that based on the integration of all development phases in one development platform it is also possible to make fundamental changes of the algorithm (should be done in phase 1) also in phase 3. This is often the case if the developer finds out that the algorithm concept is too complex to be successfully ported to the embedded hardware. If three different development systems would be used for each phase (as it would the case if developing conventionally), the fundamental change would cause a lot of code modifications and cause a development delay.
RTProc Runtime User Controls, Tools and Extensions
Once an algorithm component has the desired functionality, a very important task is to create a runtime user control in order to modify signal processing parameters while the algorithm operates (runtime user interaction). Different mechanisms are supported by RTProc to reduce the programming effort in different development phases:
- Generic User Controls: Simple user controls are sufficient for real-time algorithm exploration, verification and optimization. The RTProc generic runtime configuration mechanism is based on a description of parameters to be adapted during runtime in a software construct, in Fig. 2 shown as the runtime interaction block. Based on this description, graphical user interfaces are generated dynamically by the host. Consequently, the specification of the runtime control is independent from the host and no knowledge about GUI programming is required. All source code required to manage the generic runtime configuration is generated automatically by the RTProc Generic Runtime Compiler.
- QT based User Controls: More sophisticated user controls can be realized based on QT and the RTProc QT Extension. A framework involving code generation and the QT designer enables a drag-and-drop design methodology in which only very little portions of source code have to beadded by hand. Due to the powerful GUI features of QT, very attractive graphical userinterfaces can be designed.
In addition to the basic RTProc functionality, tools and extensions are provided with RTProc to support the algorithm developer in optimizing the real-time operation of newly created prototypes, e.g.,
- the RTProc Full Speed Data Logger to store data produced during real-time processing on hard drive without interfering with real-time processing.
- the RTProc Matlab Data Reader to read the stored data for timing accurate algorithm analysis afterwards in Matlab.
- Example 1: An embedded speech and audio codec was developed at the institute of communication systems which is suitable for ”super-wideband” (50 Hz – 14 kHz) audio communication over packet-switched networks. The codec had been successfully submitted to ITU-T by Huawei and ETRI as a candidate for the upcoming super-wideband extensions of Rec. G.729.1 and G.718. For the purpose of real-time evaluation and demonstration of this codec, an RTProc real-time demonstrator has been created which comprises the encoder and the decoder components. Using a graphical interface, the user can adjust the sent and the received bit rate in realtime. Furthermore, the instantaneous packet loss rate can be controlled to assess the codec performance under realistic channel conditions.
- Example 2: Based on the RTProc Matlab host, a tool was developed to measure room acoustics. For each measurement, a test signal is played back by a loudspeaker, recorded by a microphone and finally evaluated in order to compute the room impulse response. Signal generation and measurement evaluation are realized most efficiently in Matlab whereas simultaneous and reproducible playback and recording of audio signals must be realized by RTProc operating in the background since this functionality is not available in Matlab. The tool is an open platform in which signal generation and data evaluation functionality can be easily adapted to the needs of the user.
Figure 1 shows an exemplary screenshot of the RTProc graphical user interface.