Synthetic Air Data (an afternoon hack)

Motivation

Air France #447
Air France #447

On June 1, 2009 Air France flight #447 disappeared over the Atlantic Ocean.  The subsequent investigation concluded “that the aircraft crashed after temporary inconsistencies between the airspeed measurements – likely due to the aircraft’s pitot tubes being obstructed by ice crystals – caused the autopilot to disconnect, after which the crew reacted incorrectly and ultimately caused the aircraft to enter an aerodynamic stall from which it did not recover.”  https://en.wikipedia.org/wiki/Air_France_Flight_447

This incident along with a wide variety of in-flight pitot tube problems across the aviation world have led the industry to be interested in so called “synthetic airspeed” sensors.  In other words, is it possible to estimate the aircraft’s airspeed by using a combination of other sensors and techniques when we are unable to directly measure airspeed with a pitot tube?

Basic Aerodynamics

Next, I need to say a quick word about basic aerodynamics with much hand waving.  If we fix the elevator position of an aircraft, fix the throttle position, and hold a level (or fixed) bank angle, the aircraft will typically respond with a slowly damped phugoid and eventually settle out at some ‘trimmed’ airspeed.

If the current airspeed is faster than the trimmed airspeed, the aircraft will have a positive pitch up rate which will lead to a reduction in airspeed.  If the airspeed is slower than the trimmed airspeed, the aircraft will have a negative pitch rate which will lead to an acceleration.  https://en.wikipedia.org/wiki/Phugoid

The important point is that these variables are somehow all interrelated.  If you hold everything else fixed, there is a distinct relationship between airspeed and pitch rate, but this relationship is highly dependent on the current position of elevator, possibly throttle, and bank angle.

Measurement Variables and Sensors

In a small UAS under normal operating conditions, we can measure a variety of variables with fairly good accuracy.  The variables that I wish to consider for this synthetic airspeed experiment are: bank angle, throttle position, elevator position, pitch rate, and indicated airspeed.

We can conduct a flight and record a time history of all these variables.  We presume that they have some fixed relationship based on the physics and flight qualities of the specific aircraft in it’s current configuration.

It would be possible to imagine some well crafted physics based equation that expressed the true relationship between these variables …. but this is a quick afternoon hack and that would require too much time and too much thinking!

Radial Basis Functions

Enter radial basis functions.  You can read all about them here:  https://en.wikipedia.org/wiki/Radial_basis_function

From a practical perspective, I don’t really need to understand how radial basis functions work.  I can simply write a python script that imports the scipy.interpolate.Rbf module and just use it like a black box.  After that, I might be tempted to write a blog post, reference radial basis functions, link to wikipedia, and try to sound really smart!

Training the Interpolater

Step one is to dump the time history of these 5 selected variables into the Rbf module so it can do it’s magic.  There is a slight catch, however.  Internally the rbf module creates an x by x matrix where x is the number of samples you provide.   With just a few minutes of data you can quickly blow up all the memory on your PC.  As a work around I split the entire range of all the variables into bins of size n.  In this case I have 4 independent variables (bank angle, throttle position, elevator position, and pitch rate) which leads to an nnnn matrix.  For dimensions in the range of 10-25 this is quite manageable.

Each element of the 4 dimensional matrix becomes a bin that holds he average airspeed for all the measurements that fall within that bin.  This matrix is sparse, so I can extract just the non-zero bins (where we have measurement data) and pass that to the Rbf module.  This accomplishes two nice results: (1) reduces the memory requirements to something that is manageable, and (2) averages out the individual noisy airspeed measurements.

Testing the Interpolater

Now comes the big moment!  In flight we can still sense bank angle, throttle position, elevator position, and pitch rate.  Can we feed these into the Rbf interpolater and get back out an accurate estimate of the airspeed?

Here is an example of one flight that shows this technique actually can produce some sensible results.  Would this be close enough (with some smoothing) to safely navigate an aircraft through the sky in the event of a pitot tube failure?  Could this be used to detect pitot tube failures?  Would this allow the pitot tube to be completely removed (after the interpolater is trained of course)?

Synthetic airspeed estimate versus measured airspeed.
Zoom on one portion of the flight.
Zoom in on another portion of the flight.

Source Code

The source code for this experimental afternoon hack can be found here (along with quite a bit of companion code to estimate aircraft attitude and winds via a variety of algorithms.)

https://github.com/AuraUAS/navigation/blob/master/scripts/synth_asi.py

Conclusions

This is the results of a quick afternoon experiment.  Hopefully I have showed that creating a useful synthetic airspeed sensor is possible.  There are many other (probably better) ways a synthetic air speed sensor could be derived and implemented.  Are there other important flight variables that should be considered?  How would you create an equation that models the physical relationship between these sensor variables?  What are your thoughts?

Howto: Action Cam HUD overlay

NOTICE: This is a draft document and in the process of being written.  It is incomplete and subject to change at any time without notice.  

In this post I share my process and tools for creating HUD overlays on a flight video.  Here is a quick overview of the process: (1) Calibrate your camera, (2) Extract the roll rate information from the flight video, (3) Automatically and perfectly time correlate the video with the flight data, (4) Render a new video with the HUD overlay.

Source Code

Please be aware that this code has not yet gone through a v1.0 release process or outside review, so you are likely to run into missing python packages or other unanticipated issues.  I hope that a few brave souls will plow through this for themselves and help me resolve poor documentation or gaps in the package requirements.  All the code is open-source (MIT license) and available here:

https://github.com/UASLab/ImageAnalysis

Camera Calibration

Let’s jump right into it.  The very first thing that needs to be done is calibrate your specific camera.  This might sound difficult and mysterious, but have no fear, there is a script for everything!  Every camera (especially cheaper action cameras) has unique lens imperfections.  It is best to do your own calibration for each of your cameras rather than trying save some time and copy someone else’s configuration.

The camera calibration process involves feeding several images of a checkerboard calibration pattern to a calibration script.  Each image is analyzed to locate the checkerboard pattern.  This information is then passed to a solver that will find your camera’s specific calibration matrix and lens distortion parameters.

To make this process easier, the calibration script expects a short 1-2 minute movie.  It processes each frame of the movie, locates the checkboard pattern and stashes that information away.  After all frames are processed, the script samples ‘n’ frames from the movie (where ‘n’ is a number around 100-200) and uses those frames to solve for your camera’s calibration.  The reason that all frames are not used is because when ‘n’ starts pushing 250-300+, the solver begins to take a long time where long is measured in hours not minutes.

Here is a sample calibration video.  The goal is to always keep the checkerboard pattern fully in view while moving it around, closer, further, to different parts of the frame, and from different offset angles.  It is also important to hold the camera steady and move it slowly to avoid the effects of blurring and rolling shutter.

Now save your movie to your computer and run:

calibrate_movie.py --movie <name>

The script will run for quite some time (be patient!) but will eventually spit out a camera calibration matrix and set of lens distortion parameters. Save these somewhere (copy/paste is your friend.)

Extract the Roll Rate Information from the Flight Video

First take the camera calibration and distortion parameters derived in step one, and copy them into the gyro rate estimation script.

Next, run the script.  This script will detect features in each frame, match the found features with the previous frame, and compute the amount of rotation and translation from each frame to the next.  (It also is a hacky image stabilization toy).

1-est-gyro-rates.py --scale 0.4 --movie <name> --no-equalize

Here is an example of the script running and detecting features.  Notice that a very significant portion of the frame is covered by the aircraft nose and the prop covers most of the remaining area.  That’s ok!  The gyro estimation is still good enough to find the correlation with the flight data.

Correlate the Video with the Flight Data

The next script actually performs both of the last two steps (correlation and rendering.)

2-gen-hud-overlay.py --movie <name> --aura-dir <flight data dir> --resample-hz 30 --scale 0.45

The script loads the flight data log and the movie data log (created by the previous script).  It resamples them both at a common fixed rate (30hz in the above example.)  Then it computes the best correlation (or time offset) between the two.

Here is a plot of the roll rate estimate from the video overlaid with the actual roll gyro.  You can see there are many differences, but the overall trend and pattern leads to only one possible correct time correlation.

video-flight-log-correlation

This graph is especially noisy because only a large portion of the outside view is obscured by the nose and the visible portion is further obscured by the propeller.  But it is ok, the correlation process is magic and is really good at finding the best true correlation.  The next plot shows the results we can attain when we have more idea conditions with an unobstructed view.  Here is a plot that shows the video roll estimate and the actual roll gyro are almost perfectly in agreement.

overlay

Taking a step back, what did we just do there?  Essentially, we have created an automated way to align the video frames with the flight data log.  In other words, for any video frame number, I can compute the exact time in the flight log, and for any point in the flight log, I can compute the corresponding video frame number.  Now all that is left is to draw the exact current flight data (that we now have a way to find) on top of the video.

I look forward to your comments and questions!

This tutorial is far from complete and I know there are some built in assumptions about my own system and aircraft cooked into the scripts.  Please let me know your questions or experiences and I will do my best to answer or improve the code as needed.