Surprise!!!!: Linux is not Real-time

Ok, it's not a surprise, but I really wanted to feel it for myself.

I wrote a small C program to decode the PPM signal from an RC receiver, fed in via GPIO pin on the Overo. I'm using Dave Hylands' gpio-event kernel module to get interrupts in user-space. The driver allows a user-space program to subscribe to GPIO events and read them in via /dev/gpio-event. When an interrupt is received, the driver spits out the timestamp of the event, using do_gettimeofday() under the hood. 

To get an idea about how well this thing would keep up, I  hooked up the 'Cal' output from my trusty Tek 454 o-scope (1 kHz square wave @ 1V) to GPIO 114 on the overo, logged the timestamps, and then differenced the samples to get a jitter plot:

Of note:
- The plot is mislabeled but I'm too lazy to make another: replace "timestamp jitter ms" with "pulse interval".

- The jitter occurs in ~32us increments. I learned from this discussion that this increment corresponds to the 32kHz timer clock used by the stock kernel on the OMAP3503.

- Standard deviation of the jitter is 25us, but we frequently miss our 1ms deadline by as much as 190us. That's a lot for a PPM signal that varies between 0.5-1.5ms.

So, what to do? I have a vague sense that using a kernel with the PREEMPT_RT patches might help the situation, but I don't really understand enough to know if this is true.

I'm inclined to think that it's time to move my control code onto a dedicated microprocessor and stop trying to shoehorn it into a Linux-based system. I'd planned to do this someday anyway, and maybe this means it's time.

Rev 0 of Stick-and-rudder via Dedicated RC Link Complete

Up until today, stick-and-rudder control has been a very complicated kludge:

RC transmitter -> USB buddy-box-to-joystick-converter cable -> PC -> windows joystick driver  -> pygame ->  joystick-reading code (Python) -> TCP socket client (Python) -> wired ethernet -> dedicated wifi router -> 'copter flight computer wifi link -> TCP socket server and control loop (Python) -> servo controller board -> motor speed controllers.

Sounds like absolute madness, but implementing this took very little time. It's always kinda sucked, though, and controlling the quadcopter felt really spongy and difficult. Lot's of crashing into the bushes and whatnot.

I've now got throttle control working via the RC radio link:

RC transmitter -> RC reciever (on 'copter) PPM out -> 'copter flight computer -> ppm_decoder and TCP socket client (in C) ->  TCP socket server and control loop (Python) -> servo controller board -> motor speed controllers.

Much simpler.

Observations:

- Throttle response is way faster. This makes me happy (like "I've got a boner" happy). 

- Output from ppm_server is very noisy. At half-throttle with no stick movement, the throttle value jumps around as much as +-20% of max-throttle. Watching the PPM signal with the o-scope, I don't see any such jumpiness. I suspect that we're up against interrupt latency. Time for a RT_PREEMPT kernel! (I've always wanted and excuse to try this).

- Whenever the radio is powered, the motors twitch, even though the throttle commands from hl_controller.py are zeroed. I suspect RF interference with the servo controller board/wiring. Sorting this one out with require more quality time with the o-scope.

For Want of a Nail (or a bypass capacitor)

Indeed, noise on the IMU power line was the problem: while hooking up a bench power supply to the IMU (to isolate the IMU power from the Overo), I found that one of the leads of the bypass capacitor on the Pinto board had come loose. After fixing the solder joint (this time with a good, solid mechanical connection), the problem is gone. First, a plot of ten minutes of roll_rate after the repair (note how the plot is symmetrical around zero, and way smaller than the bad plot from the last post:

Updated stats:

	Factory	Quad	Quad, after bypass cap repair:
Roll Mean:	-0.78432	-0.63128	-0.77224
Roll STD:	0.14276	0.55705	-0.18555
Roll Max:	-0.4	1.04	-0.20874
Roll Min:	-1.4	-2.6	-1.3
Roll Rate Mean:	0.000353	-0.00224	0.000337
Roll Rate STD:	0.331451	1.89036	0.61565
Roll Rate Max:	2.635684	4.458878	2.8152
Roll Rate Min:	-2.92217	-8.58552	-3.637

Much better! While I was at it, I noticed that there's another problem: about 10% of our IMU readings have bad checksums. That's enough for concern, and might explain why our stdev numbers are still not as good as reading the IMU through a factory cable.

Shorter wires, ferrite beads?

Houston, we have a problem (IMU weirdness)

Notes from yesterday:

While looking into the direct use of the gyros for the 'D' term (as opposed to numerical differentiation), I noticed that the gyro output looked pretty crappy. Here's a plot of a ten-minute sample, vertical axis represents degress/s:

I'd expect the gyros to be a little noisy, but that's atrocious!!! There's also a weird bias toward negative values, and this isn't what gyro bias looks like. I took a look at the output using the factory software run from my laptop with the factory cable, and indeed, there's something amiss. The values obtained from the factory software are an order of magnitude less noisy. Here are the results of comparing 10-minute runs obtained via the PC (with factory cable) vs. from the Quad, with the Quad sitting on my bench, motionless:

Factory
Quad
Roll Mean:
-0.78432
-0.63128
Roll STD:
0.14276
0.55705
Roll Max:
-0.4
1.04
Roll Min:
-1.4
-2.6
Roll Rate Mean:
0.000353
-0.00224
Roll Rate STD:
0.331451
1.89036
Roll Rate Max:
2.635684
4.458878
Roll Rate Min:
-2.92217
-8.58552

On the roll axis, the standard deviation is 5x higher on the Quad. On the roll rate axis, the mean value is a nearly 10x higher, and the standard deviation is 6x higher. 

I carefully reviewed the IMU reading code and couldn't find anything, so I started thinking that there's some corruption on the serial line to the IMU, so I inserted some checksum code. Surprisingly, most of the time, there were no checksum errors (unless I wiggled the wires!...that's certainly a problem, but probably not the problem). The only thing I can think of is noise on the power lines to the IMU, since that's the only fundamental difference that i can think of. I'll spend some quality time with the o-scope.