Interview: Victor Yodaiken ... RTLinux
> m-station:
> I guess a good starting place is to say just exactly what RTLinux is
> and does. I didn't realize until I visited your web site for example
> that RTL is something that operates under an existing Linux installation
> rather than replacing the whole thing. Could you tell us a bit
> about the whole package?
Victor:
RTLinux is a small, quite simple, operating system that runs user
applications as something very much like POSIX threads. One of those
threads is Linux itself -- it runs as the lowest priority always
pre-emptible thread. The theory here is that code that needs tight
time constraints runs directly under the RT kernel and code that
does not can run as Linux kernel or user processes. For example, the
simplest data logging system under RTLinux may consist of a RT thread
that collects data from a device, say, every 200 microseconds, and
dumps that data down a "rt fifo" to the following Linux logging
shell program
cat > logfile < /dev/rtf0
We make all Linux services available to
RT apps and still allow them to have hard timing properties for
core code.
The programming approach in RTLinux is somewhat unusual. We ask
programmers to factor their applications into "real real time" and
"not realtime" and build applications by connecting these components.
My experience is that, especially for complex applications, this
model leads to a much more reliable design.
> What makes an RT app? Is there special code required to interface
> with RTLinux and if so, is converting an existing app trivial or
> ... not so trivial?
> The above seems to suggest that RT is either on or off. Is that an
> over-simplification?
On 2.3+ Linux kernels, you turn RTLinux on by putting the
rtlinux core module into the kernel:
insmod rtl.o
This module takes over interrupt control functions from Linux and
provides the basic hooks for programs. RTLinux applications are
usually constructed in two parts: A RT part in a kernel module
and the Linux part in applications. The RT part is somewhat analogous
to a driver, although it can be quite a bit more sophisticated
than a driver. As a simple synthetic example, suppose we need
to take data off one card, do a computation, and put it on a second
card. The app might start off in the form:
Module:
my_thread(){
read card1;
apply_function(table,read_data);
write output to card 2
}
module_init(){
pthread_create(&my_thread ...);
pthread_make_periodic(my_thread,period);
create shared memory region for users
put default table into shared memory
}
module_cleanup() ...
Then you might have Linux code that (in non-realtime) read the table
in shared memory and possibly changed parameters.
To run the app you would insert the needed RTLinux modules (or have the
system set up to do it for you) and then insmod your application module
and start your Linux code too.
> What happens if there is more than one app
> that wants RT access? Will they share under RTLinux with a slight
> slowdown or are there lockouts?
The CPU is 100% available to RT threads and can be divided up as you like
by setting scheduling. If you have two RTLinux task that each need
100Mflops on a machine that can only do 150Mflops, we have not yet
figured out how to solve the problem.
But, it is quite easy to have one RT task, for example that sends data down
one fifo and a second task that sends data down a second fifo and have two
different unix processes read the two fifos.
The RT kernel does not mandate any lockouts.
> Are there existing audio apps that will run under RTLinux? (and
> utilize it)
There are some simple ones, I joined the l-a-d list to see if there
was anything else.
We may write some examples, if needed to start things off.
> Ingo Molnar's Low Latency kernel patches offer a fairly spectacular
> speed up for some audio apps. Would you like to comment on how your
> approach differs and how the behaviour of apps might differ?
RTLinux offers low microsecond latencies. On a standard PC, we get under
20 microseconds _worst case_ interrupt latency and on a PowerPC we
seem to be closer to 10 -- depending on peripherals.
Ingo's patches are aimed at low millisecond latencies.
> Whoa! This is probably a good time to remind people who've
> forgotten what milli and micro mean - micro = 1 millionth and
> milli = 1 thousandth. We're getting right down to hardware
> latencies here. What are the main current uses for RTLinux?
Instrumentation: NASA uses it a lot, factory floor control, machine
tools, some telecom.
My favorite app is the Jim Henson muppet factory code for controlling
puppets and for generating graphics.
> Really? What form of control do they have with the puppets?
They run puppet motors using a laptop and a National Instruments PCMCIA
A/D card.
> Do you have any thoughts regarding Linus Torvald's rejection of
> the Low Latency patch for kernel inclusion?
I have decidely mixed feeling about this. On the one hand, RTLinux
would benefit a great deal from Linux with low millisecond latencies.
Having this in Linux would open up many applications for RTLinux.
One the other hand, one of the purposes of RTLinux was to keep the OS
simple and reliable by separating realtime and nonrealtime code. Linux
really needs to be optimized for average case and the RT side is
optimized for worst case. When you try to do both in the same
complex code, you often end up getting poor results. So, I would
be much happier if Ingo would do a whole lot more work in his spare
time and rework the slower algorithms in Linux, rather than
"simply" putting in more places where process switching can happen.
> Would reworking the algorithms result in such dramatic decreases in
> latency for a SCHED_FIFO app? Just intuitively, I would have thought
> that reworking the algorithms would give faster, "fairer" overall
> responce and the app in question would be quicker but not as quick as
> the current approach (which can affect useability adversely).
Here's a simple and perhaps not workable example. Write code
can do
copy a block
check needs_resched
send block to output device
loop
done
or perhaps it can do
create an abstract descriptor of the pages needed
ask the device for an optimal strategy
start a timer and feed the device driver every
timer interval
The first approach just puts opportunities to reschedule in the
code -- breaking a long, and perhaps un-needed copy into parts.
The second (with a whole lot of handwaving) makes the algorithm more
efficient and still removes the problematic long copy. The second
approach is much harder but might do things like avoiding bringing
paged out blocks into main memory at all ...
Of course, Ingo has working code and I'm proposing virtual ideas here
so the Linux tradition puts a lot more weight on Ingo's solutions.
> Thanks very much Victor.
Thanks for the interview and anyone interested in getting help starting a
RTLinux audio project, please contact me. We may even be able to arrange
some hardware loans.
Victor Yodaiken can be contacted by email at yodaiken @ fsmlabs.com
