Image Stitching Tutorial Part #1: Introduction

features

Background

During the summer of 2014 I began investigating image stitching techniques and technologies for a NOAA sponsored UAS marine survey project.  In the summer of 2015 I was hired by the University of Minnesota Department of Aerospace Engineering and Mechanics to work on a Precision Agriculture project that also involves UAS’s and aerial image stitching.

Over the past few months I have developed a functional open-source image stitching pipeline written in python and opencv.  It is my intention with this series of blog postings to introduce this work and further explain our approach to aerial image processing and stitching.

Any software development project is a journey of discovery and education, so I would love to hear your thoughts, feedback, and questions in the comments area of any of these posts.  The python code described here will be released under the MIT open-source software license (one of my to-do list items is to publish this project code, so that will happen “soon.”)
feature-matching

 Why?

The world already has several high quality commercial image stitching tools as well as several cloud based systems that are free to use.  Why develop yet another image stitching pipeline?  There are several reasons we began putting this software tool chain together.

  • We are looking at the usefulness and benefits of ‘direct georeferencing.’  If we have accurate camera pose information (time, location, and camera orientation of each image), then how can this improve the image analysis, feature detection, feature matching, stitching, and rendering process?
  • One of the the core strengths of the UMN Aerospace Engineering Department is a high quality 15-state kalman filter attitude determination system.  This system uses inertial sensors (gyros and accelerometers) in combination with a GPS to accurately estimate an aircraft’s ‘true’ orientation and position.  Our department is uniquely positioned to provide a high quality camera pose estimate and thus examine ‘direct georeferencing’ based image processing.
  • Commercial software and closed source cloud solutions do not enable the research community to easily ask questions and test ideas and theories.
  • We hope to quantify the sensor quality required to perform useful direct georeferencing as well as the various sources of uncertainty that can influence the results.
  • We would love to involve the larger community in our work, and we anticipate there will be some wider interest in free/open image processing and stitching tools that anyone can modify and run on their own computer.

Outline

I will be posting new tutorials in this series as they are written.  Here is a quick look ahead at what topics I plan to cover:

  • Direct Georeferencing
  • Image stitching basics
  • Introduction to the open-source software tool chain
  • Aircraft vs. camera poses and directly visualizing your georeferenced data set
  • Feature detection
  • Feature matching
  • Sparse Bundle Adjustment
  • Seamless 3D and 2D image mosaics, DEM’s, Triangle meshes, etc.

Throughout the image collection and image stitching process there is art, science, engineering, math, software, hardware, aircraft, skill, and a maybe bit of luck once in a while (!) that all come together in order to produce a successful aerial imaging result.

Software Download

The software referenced in this tutorial series is licensed with the MIT license and available on the University of Minnestoa UAV Lab public github page under the ImageAnalysis repository.

Credits

Flight Milestones

Congratulations!

Congrats ATI Resolution 3, Hobby Lobby Senior Telemaster, Hobbyking Skywalker, and Avior-lite autopilot on your recent milestones!

IMG_20130923_183033 IMG_20150804_073522 IMG_20150804_093541 IMG_20150801_135249

Avior-lite (beaglebone + apm2 hybrid) autopilot:

  • 300th logged flight
  • 7000+ logged flight minutes (117.8 hours)
  • 6400+ fully autonomous flight minutes (107.2 hours)
  • 2895 nautical miles flown (3332 miles, 5362 km)

Hobby Lobby Senior Telemaster (8′ wing span)

  • Actively flight testing autopilot hardware and software changes since 2007!
  • 200th logged flight.
  • 5013 logged flight minutes (83.5 hours)
  • 4724 fully autonomous flight minutes (78.7 hours)
  • 2015 nautical miles flown (2319 miles, 3733 km)

Today (October 7, 2015) I logged the 300th avior-lite flight and simultaneously logged the 200th flight on my venerable Senior Telemaster.  I realize these are just numbers, and they wouldn’t be big numbers for a full scale aircraft operation or even a production uav operation, but it represents my personal effort in the UAV field.

I’m proud of a mishap-free 2015 flying season so far!  (Ok, err, well one mishap on the very first launch of the skywalker … grrr … pilot error … and fixable thankfully.)

Enjoy the fall colors and keep flying!

fall-colors-skywalker

APM2 Sensor Head

apm2

The ardupilot mega is a fairly capable complete autopilot from both the hardware and the software perspective.  But what if you projects needs all the sensors and not the full APM2 autopilot code?

Overview

The apm2-sensorhead project provides a quick, robust, and inexpensive way to add a full suite of inertial and position sensors to your larger robotics project.  This project is a replacement firmware for the ardupilot-mega hardware.  The stock arduplane firmware has been stripped down to just include the library code that interrogates the connected sensors, and also maintains the code that can read your RC transmitter stick positions through a stock RC receiver as well as drive up to 8 servo outputs.  It also includes manual override (safety pilot) code and code to communicate with a host computer.  The next upcoming feature will be onboard mixing for vtail, elevon, and flaperon aircraft configurations.  This allows you the option to fly complicated airframes (and have an autopilot fly complicated airframes) without needing a complicated transmitter or adding complicated mixing code to your autopilot application.

Why?

Speaking a bit defensively: I want to address the “why?” question.  First of all, I needed something like this for one of my own autopilot projects.  That’s really the primary motivation right there and I could just skip to the next section.  If you can answer this question for yourself, then you are my target audience!  Otherwise if your imagination is not already running off on it’s own, why should you or anyone else possibly be interested?

  • You are working in the context of a larger project and need to incorporate an IMU and GPS and possibly other sensors.  Do you design your own board?  Do you shoehorn your code onto the ardupilot?  Do you look at some of the other emerging boards (pixhawk, etc.?)  What if you could integrate the sensors you need quickly and easily?
  • The ardupilot mega is a relatively inexpensive board.  There are some APM clones available that are even less expensive.  It would be hard to put together the same collection of sensors for a lower price by any other means.
  • The ardupilot mega is proven and popular and has seen very wide scale testing.  It is hard to find any other similar device on the market that has been tested as thoroughly and under as wide a variety of applications and environments than the ardupilot.
  • The ardupilot mega code (especially the sensor I/O and RC code) is also tremendously well tested and ironed out.
  • By stripping all the extra functionality out of the firmware and concentrating simply on sensor IO and communication with a host computer, the new firmware is especially lean, mean, fast, and simple.  Where as the full arduplane code is bursting the atmega2560 cpu at the seams with no more room to add anything, compiling the amp2-sensorhead code reports: “Binary sketch size: 36,132 bytes (of a 258,048 byte maximum)”
  • Along with plenty of space for code, removing all the extraneous code allows to CPU to run fast and service all it’s required work without missing interrupts and without dropping frames.
  • There is a design philosophy the prefers splitting the hard real-time work of low level sensor interrogation from the higher level intelligence of the application.  This can lead to two simpler applications that each do their own tasks efficiently and well, versus a single monolithic conglomerations of everything which can grow to be quite complex.  With all the hard real time work taken care of by the apm2, the host computer application has far less need for a complicated and hard-to-debug thread based architecture.
  • The APM2 board can connect to a host computer trivially via a standard USB cable.  This provides power and a UART-based communication channel.  That’s really all you need to graft a full suite of sensors to your existing computer and existing application.  Some people like to solder chips onto boards, and some people don’t.  Some people like to write SPI and I2C drivers, and some people don’t.  Personally, I don’t mind if once in a while someone hands me an “easy” button. 🙂

Some Technical Details

  • The apm2-sensorhead firmware reports the internal sensor data @ 100hz over a 115,200 baud uart.
  • The apm2-sensorhead firmware can be loaded on any version of the ardupilot-mega from 2.0 to 2.5, 2.6, and even 2.7.2 from hobbyking.
  • The UART on the APM2 side is 5V TTL which you don’t have to worry about if you connect up with a USB cable.
  • This firmware has flown for extensively for 3 flying seasons at the time of this writing (2012, 2013, 2014) with no mishap attributable to a firmware problem.
  • The firmware communicates bi-directionally with a host computer using a simple binary 16-bit checksum protocol.  Sensor data is sent to the host computer and servo commands are read back from the host computer.
  • Released under the GPLv3 license (the same as the arduplane library code.)
  • Code available at the project site: https://github.com/clolsonus/apm2-sensorhead  This code requires a minor ‘fork’ of the APM2 libraries available here: https://github.com/clolsonus/libraries

Questions?

I am writing this article to coincide with the public release of the apm2-sensorhead firmware under the LGPL license.  I suspect there will not be wide interest, but if you stumble upon this page and have a question or see something important I have not addressed, please leave a comment.  I continue to actively develop and refine and fly (fixed wing aircraft) with this code as part of my larger system.

 

PBY Catalina

January 27, 2015

All shiny again!

IMG_20150127_174222 IMG_20150127_174150 IMG_20150127_174100

January 27, 2015

Today my replacement plastic parts arrived … a new canopy and a new hull shield.  It wasn’t too hard to pry the old shield off, and the new part is already to slap on.

IMG_20150127_112053

January 26, 2015

IMG_20150126_102541

Here is a quick snapshot showing the new orange color on the engine cowls, the upper side wing tips, and also notice the tail stripes have been repainted orange (originally red.)  I’m just waiting on a replacement canopy which will hopefully arrive tomorrow and I should be able to completeley reassemble the model and re-maiden it after the crash.  Oh, I also have soldered up the led landing lights and will have those illuminated now just for fun.

January 23, 2015

Repair status update:  The crash repairs continue at a slow but steady rate; I am hoping the result will be better than ever!

Here is a picture showing the motor nacelle damage.  This has now been reglued and is as strong as ever.

IMG_20150117_142805

The next picture shows the repair process of the wing center pylon. This got shredded when the wing ripped off.  2 pieces have already been attached in this picture (with a battery on top to weight it down) and I am holding the final piece in it’s approximate location.

IMG_20150117_165949

My repair plan includes adding some orange color to the engine nacelles, the top of the wing, and the tail for enhanced visibility.  In the next picture I have removed the cowls before painting them.

IMG_20150120_125311

Now we see the cowls painted orange and reattached.  I think the final result will look nice if I can get some orange stripes on the wing tops and tail and do it cleanly.  In this next picture you can also see a view of the repaired wing center pylon (with some white filler that still needs to be painted.)  I found some flat gray spray primer at the hardware store that is a pretty close match to the factory paint, but it is cheap paint and covers poorly and requires a couple coats.

IMG_20150123_140702

I have ordered a replacement canopy and replacement forward hull shield because these thin plastic parts were substantially damaged in the crash.  Grayson hobby had these parts in stock for $9.99.  If you are frustrated because nitroplanes.com is always out of stock of the airplane or part you want, check out grayson hobbies online.  They have a wide range of dynam stuff.

January 4, 2015

IMG_20150104_212950 IMG_20150104_210307 IMG_20150104_210229 IMG_20150104_210112

More detailed damage assessment:

  • The fuselage center wing mounting pylon is pretty shredded.  I’m hoping I can glue the parts back together to get the shape, and then do some reinforcement to get the necessary structural strength.
  • The right wing tip float broke off.  I’m strongly considering a mod to convert this model to retractable wing tip floats.  There are a few people on youtube that have done this and it turns out pretty cool.
  • The canopy got shattered into dozens of bits.  I need to either buy replacement plastic parts or carve a replacement from something.
  • The left engine nacelle got completely ripped off.  I think this will go back together again ok with enough surface area to be structurally sound.
  • The right front bottom of the nose took some damage I just now noticed.  That’s probably something I can just glue back together and it will be ok.
  • I am thinking about getting out my can of orange spray paint and doing an outboard section of the top of the wing in high visibility orange (and maybe a couple other bits while I’m at it.)  Part of the reason for the crash was that I completely lost orientation on the model, and part of the reason for that (I think) was flying an all gray model on an overcast day.

January 3, 2015

IMG_20150103_134506

I went out to fly this off the lake this afternoon (frozen this time of year.)  I did one good flight, switched batteries and midway through the 2nd flight some winds started to come up a bit. Then I did the unthinkable … I lost my perspective/orientation with the model, executed one of those dumb thumb moves, and corkscrewed it right in.  It’s been a long time since I’ve had a crash I couldn’t blame on mechanical, electrical, or weather factors.  This one was just dumb thumbs and it was over before I could figure it out.  Now I’m really bummed.  Here is a picture of the wreckage I carted home.  Major damage includes the center wing pylon sheared off (foam/structural), the canopy shattered to bits, one of the wing tip floats broke off (plastic part broke), one of the engine nacelles broke off, and probably a few other odds and ends I’ll find if I dig in further.

I think the engine nacelle and center wing pylon will be repairable.  I don’t know what I will do about the canopy … maybe I can buy a replacement?  Same with the wing tip float mount?  I suppose I’ll set it to the side for now.  Usually after a few days I start to get a bit curious and dig in and often a lot of the damage is quickly repairable.  Some of it might be a little harder though … guess I’ll post a repair log if I think I’ll be able to fix it up and fly it again.

June 21, 2013

This is the PBY from nitroplanes.com.  It is a relatively big airplane compared to most ‘foamies’ these days with a 57-7/8″ wing span.  It is also a twin and a seaplane.  I have nothing but good things to say about it.  For the cost and the effort to assemble it, it looks really great!  On the ground, in the water, in the air, sitting still, water taxiing, flying … it just looks great.

IMG_20120129_133409

I purchased the PBY in January of 2012, so my first flights were off a snow covered lake on a bitter cold January afternoon.  I fly it with a 2200 mah 3 cell battery and that provides tons of flight time … probably pushing 20 minutes or more of relaxed flying.  Relaxed is the key word. There is nothing white knuckle about flying this airplane.

IMG_20120129_133337

One of the fun things I did this day was to snow taxi downwind a bit, turn back around into the wind and give it about 3 notches of throttle.  It began to slide across the slick snow and pick up speed slowly.  Little by little it accelerated and before I thought it would be ready to fly, it slowly lifted off on it’s own.  It passed in front of me about eye level, still climbing out at a very slow, very scale looking flight speed.  There are moments that are so perfect you always remember them, and in the context of flying RC airplanes, this was one of them.2012-04-19

She handles water operations just as effortlessly as flying off snow.   There is a little park with a dock at one corner of this lake in walking distance from my house.  It’s a perfect place to go fly a park-flyer seaplane on a nice calm summer evening.  The twin engines, the big wing with slow flight characteristics, smooth water handling … it just makes for a calm, relaxing fun evening of flying.  At the park, the PBY always seems to attract an audience too.

If you are into aerobatics, the catalina can do all the basic loops, rolls, inverted flight, wing overs, etc.  It can, but for some reason, I’d rather watch it do slow, scale, near perfect fly-by’s all evening long, mix in some touch and goes off the water … it’s really pretty!IMG_20120811_111738

For whatever it’s worth, the Catalina has plenty of power to do dry grass takeoffs from the RC club field.  I try to be extra careful keeping the wings level on landing so I don’t catch one of those wing tip floats.  The ground is a little less forgiving than water.

Just to summarize.  This is a very scale looking airplane.  It has a big fat wing and is lightly wing loaded (just like the real thing.)   It can fly amazingly slow in the air.  If you just want to enjoy a beautiful scale looking/flying aircraft, this one is hard to beat!

Sonic 64

Hobbyking Sonic 64

This is 1230mm (48.4″) wing span flying wing.  It is powered by a 64mm electric ducted fan.  It is a simple build and flies great!

Here are a couple pictures before the maiden flight (temperature was about +18F, winds were calm, skies had a medium thin overcast.)

IMG_20150110_143003  IMG_20150110_143017 IMG_20150110_143010

My one gripe about the Sonic 64

Of course we buy airplanes like the Sonic 64 because they are inexpensive, quick to assemble, and fast to get out to the field and up in the air.  It doesn’t hurt that they look great and are a lot of fun to fly!  But in the case of the Sonic 64, perhaps it is a little too inexpensive.  The servos that came with the kit were utter garbage.  One of them was completely useless and unsafe to fly.  It couldn’t hold a position, and it had a lot of trouble with it’s potentiometer.  The other servo probably could have been able to fly safely, but not fly well.  I submitted an RMA to Hobbyking where I purchased this ARF, and to-date they have not responded with an “initial evaluation.”  Typically the stuff I get from Hobbyking works and meets expectations, but this is pretty poor customer service to not even acknowledge my RMA request after a week and a half of waiting.  So I gave up, dug out the original servos, and replaced them with Hitec HS-55 servos.  These are $10 servos and far better than the $0.50 servos that came installed with the kit.  Unfortunately the servo leads on the Hitec servos were shorter so add two $3 servo extensions.  And finally add the cost of some glue to install the servos properly … altogether that added about $31 to the cost of the airplane … and Hobbyking still has yet to respond to my RMA.

Servo Installation in foam wings

My new favorite glue for installing servos into foam wings is: E6000 (available at Michaels and probably lots of other places.)  It is not a quick dry cement (let it dry over night) but the result is solid, it seems to be foam safe, it dries clear, and according to the video it’s pretty easy to cut through with a knife to later separate the glue joint.
I ran up to Michaels last night and got a small tube to try on my Sonic 64 and I’m really pleased with the results this morning.  I’ve tried different techniques over the years, but I’ve had hot-glue and double sided tape be wiggly or just pop out.  Epoxy is a bit too permanent when you are talking about cheap servos … E6000 seems to be a really good balance between creating a solid glue joint between a servo and foam, but then being able to separate it later if needed.  Here is an in depth how-to video (not mine) with a bunch more explanation.

e6000

 

Variants

This same design is sold under the name “Neptune” by Nitroplanes.com (but was out of stock when I was interested in ordering it.)  The manual still has at least one reference to Neptune that hadn’t been updated. 🙂

Launching

The Sonic 64 is hand launched.  This can be a little tricky if you are flying by yourself or need to get help from someone who doesn’t have a lot of airplane hand launching experience … so be cautious, things can go wrong quickly.

The manual suggests running the throttle up to about 50% (that feels about right) and then running 2-3 steps and giving it a good firm level throw (wings level, nose level, throw direction straight and level.)  I had one good launch, one bad/failed lauch (no damage, whew!), and a 2nd good launch on my 3rd try.  I noticed the left wing would really drop quickly.  I made every effort to make a straight level through, but maybe my technique is flawed?  Maybe there is some torque or spiral thrust issue going on?  Hopefully I can learn to do this more reliably on my own.  Often I’m out flying by myself, and then no offense to any of my fellow RC club members, but many of us don’t have much hand launch experience.

Next time out I think I may try to run the throttle up a little higher (maybe 2/3rds) and try to release it on an upwards trajectory … maybe 15-20 degrees up?

Flying

My maiden flight was on a dead calm afternoon and the Sonic 64 flies beautifully.  It is stable, tracks nicely, and responds well to control inputs.  It has no rudder, but it can do all the bank and yank type aerobatic maneuvers you can think of.  I attempted a high altitude stall to see what kind of craziness would happen and it was boring … it just dropped the nose and kept flying.  I’m sure I could get it to spin if I stalled it out in a steep turn, but I didn’t want to push it too hard on my first time out.  One thing I did manage to do was a sequence of really graceful wingovers.  I could fly a fast eye level pass, climb out steeply, and then as speed bleeds off, roll into a bank (maybe 10-20 degrees) and hold that as you run out of airspeed.  The airplane naturally does a wing over and comes back flying right past you the other direction.  It’s really graceful.  If you hold the throttle fixed through the maneuver, it tends to kick you over just a bit faster when you are at the apex and have minimal aerodynamic control and the thrust starts to dominate.  If you pull the throttle back at the apex, then it’s a bit more lazy and graceful.

I was flying a cold day (+18F) so the air was relatively dense.  As a result, top speed didn’t seem too fast, even though the motor was really wound up.  I expect on a warm summer evening it will zip by much faster in hotter, less dense air.

Landing

The Sonic 64 is a flying wing and behaves exactly how you would expect.  It has a relatively long, flat glide, and if you carry any speed over the threshold, it will glide forever in ground effect.  So there really isn’t anything difficult or unexpected, just be ready for a fairly long, flat approach and be ready for it to carry during the flare as it wants to just keep flying forever in ground effect.

Conclusions

This is a great flying, inexpensive airplane that goes together quickly (probably an hour or two maximum.)  It is appropriate for a moderately experienced sport flyer.  It’s also a big bundle of fun and looks really cool in the air.  However, I am now aware that if I do have any problem with something ordered from Hobbyking, I shouldn’t have high hopes for their customer service and for their response times.  That’s unfortunate because I actually like Hobbyking otherwise and everything else I’ve bought from them has met expectations.  In this case the problem was easy to fix (yeah model making skills) but that was an extra chunk of money out of my own pocket.

IMG_20150110_143109

Polaris Ultra

Manufacturer: Model Aero

Design: Polaris Ultra

Web sitehttp://modelaero.com/product/POL-ULT.html

Summary: Pretty awesome airplane

Model Aviation also has a nice review of the Polaris: http://modelaviation.com/modelaeropolarisultra

My story is that I missed one of the model aero sales by a day or two so I emailed the owner to see if I could sneak in my order late.  He replied that he had a couple unpainted polaris arf’s and he’d be willing to give me a package deal on one of those.  So I figured it couldn’t be that hard to paint a foam model and off I went.

The aircraft arrived completely unpainted. The foam was white, and even the plastic parts were just clear.  I test fit a few of the pieces and it sure was looking pretty cool even without paint:

IMG_20141029_141228 IMG_20141029_141246

The assembly process is pretty straight forward and the instructions are good.  It’s perhaps a bit more work than some of the newer foam arfs.  It took me a couple evenings (including the extra work of painting.)  But, there was nothing too difficult.  I chose to mount my elevator servo externally to simplify the install.

Because I was painting the model from scratch, I got to make all the choices.  That is good and bad.  I decided to try to keep things simple and roughly follow the original scheme.  I happened to have flat gray primer in a can, and then I ran to the local hardware store and picked up a can of bright red.  So those were my two colors.  The cockpit and dorsal vent got painted flat gray.  The cockpit piece is removable for swapping flight batteries so it was easy to get a nice clean edge there.

IMG_20141101_162920 IMG_20141101_162930 IMG_20141103_134127

Painting the color was a bit more work.  I’d never done this before, not exactly, so I had to make things up a bit as I went along.  It turns out that if I hit the foam surface hard with the spray paint, the propellant would melt the foam. That wouldn’t be a good thing.  Same problem with the primer.  I did some googling and water based spray paint exists, but I didn’t have it in hand.  I found some test foam and it turns out that if you start by lightly dusting the surface from a good distance (maybe 18+ inches) from the surface.  This lets most of the propellant evaporate off before it hits the foam, and as the dust accumulates the surfaces is more and more protected.  So I did this trick with the primer and then later could paint normally over that with the color layer.

I decided to use electrical tape to mask the paint areas.  This was good in the sense that it didn’t seem to bleed too much, but bad in the sense that it was too sticky and pulled some of the shiny foam finish off when I removed it.  That’s hard to notice though so I guess it wasn’t a complete disaster.  I layed down my exact lines and then masked off with newspaper after that.  One thing I discovered though is that the spray goes everywhere.  It managed to sneak through some cracks I thought were sealed.  Again, not perfect, but not too noticeable on the final result so I guess it’s not the end of the world.

IMG_20141103_145529 IMG_20141103_193730IMG_20141104_131857

After painting the color areas, I went back with 1/2″ electrical tape and used that as edge trim to hide paint edge and tidy things up.  Again, not perfect and kind of crude, but I think it turned out pretty sharp in the end.

IMG_20141104_184237 IMG_20141103_202244 IMG_20141103_205632

I painted the bottom of the wings flat gray to differentiate them from the top, and with the 1/2″ black tape on the edges I think it set it off and made it look pretty sharp.  I’m happy with the result.

IMG_20141103_202407

Here is the end result.  The canopy is glued on, the cowl is painted red and attached.  3-bladed prop looks great.  The decals are applied in the final shot.

IMG_20141104_184329 IMG_20141104_184425 IMG_20141105_163631

I don’t have any flight videos or pictures yet, but I did do the maiden flight. I waited and waited (almost a month) for decent weather and finally got a day with almost dead calm winds here in MN.  The Saturday after Thanksgiving was the day.  There is a small lake across the street from my house and this time of year it is frozen.  Fortunately due to early extra cold weather, the ice on the lake was thick enough to walk on safely.  So I put on my boots and headed out to find sunshine, upper 30’s, and dead calm winds.  It was later in the afternoon, but the sun was still bright.  The first flight went perfect.  It needed a bunch of clicks of right trim and a couple clicks of up, but once I got her dialed in, then WOW!  She really flies spectacularly.  I love how you can pull high alpha in the turns to make them extra tight.  She tracks and handles so nicely, like she is on rails.  She looks so sweet in the air!  Take offs from the snow were amazingly smooth.  I was flying on some tired old 3S batteries, but honestly that was plenty of power and plenty fast for the area I was flying.  It is a relatively long model so it is extremely stable in pitch.  It is all around really wonderful … looks and flies spectacularly!

 

 

Coding and Complexity

Norris Numbers

I recently stumbled on the following article about “Norris numbers”

http://www.teamten.com/lawrence/writings/norris-numbers.html

The quick summary is that for an untrained programmer, 1500 lines of code is about where they hit the wall before they succumb to complexity and organizational issues.  The next big barrier is 20,000 lines of code for those that have some training and a bit of experience.  The next big barriers are at 200,000 lines, 2 million lines, etc.  At each level, a programmer or team has to develop new techniques and strategies to overcome the inherent cognitive limits of our human brains.

SLOCCOUNT

Linux offers a neat little tool called “sloccount”.  You can run it on just about any source tree, and it will count up the lines of actual code and then estimate how long it might have taken to develop that code and what the expected costs might be.

For example, my core UAS autopilot project is about 41,000 lines of code currently.  Sloccount estimates this represents nearly 10 years of effort (I started the work in 1995) and at a modest yearly salary estimates the cost to develop this code at about $1.3 million dollars.

My first reaction is this is crazy talk, sloccount is just pulling numbers out of it’s rear aperture.  But after thinking about it, maybe these numbers aren’t so far off.  If you also consider the need for a project to mature over a period of time and consider bugs, testing, etc. then it’s not so much about how many lines of code can you crank out per hour, but more about how much effort is required to create a body of code that is functional, robust, and mature.  If you take a mature software package and work backwards, I suspect sloccount’s numbers would start to look much more reasonable.

Workload and Expectations?

I’ve spent my career in the trenches.  I never really was interested in the management track.  This clearly colors my perspective.  When I read generalized comments (like the article linked above) my first thought is to apply them to myself.

After reading the article about code complexity barriers, I immediately went out and evaluated several of my projects.  FlightGear = 264,000 lines of code.  My UAS autopilot is 41,000 lines of code.  My little summer side project is 6,000 lines of code (sloccount estimates that is 15.76 months worth of effort packed into about 2 months of time.)  My model airplane designer project that I’ve worked on during Christmas break the past 2 years is 5,600 lines of code.

I still don’t think we can put too much faith in sloccount’s exact numbers, but when I wonder about why I’m so overwhelmed and feel buried in complexity and deadlines and pressure, maybe this offers a little perspective, and maybe I’m doing ok, even if I feel like I’m coming up short of everyone’s expectations.  For the managers and entrepreneurs out there, maybe this can guide your expectations a bit more accurately.

Cutting the Cord

Update: 10/01/2014

Things just aren’t quite falling into place perfectly for this cord cutter.  Since cancelling our satellite, I’ve been monitoring 4 possible antenna DVR options.

  1. Tivo Roamio OTA.  This seems like exactly what I’m hoping for.  Maybe a little expensive on the monthly subscription, but I’m willing to give it a try.  The downside: not available anywhere, any target date has been scrubbed.  Rumors suggest best buy is removing all tivo products from their stores.  Oof dah.
  2. Tivo Roamio.  This is $110 more than the Tivo Roamio OTA for the initial purchase with the same monthly subscription.   On the plus side it is available and shipping now.
  3. Tablo DVR.  This one looks really interesting.  However, I don’t own an ipad and I want to watch TV on my TV, not on my tablet.  They say they have newly added Chromecast support, but everything I’ve seen suggests their android support (needed for chromecast) sucks, and it’s unclear if watching live TV would work through the chromecast and how sports type shows would look.  It’s fairly pricey on top of everything else.
  4. Smart TV.  I looked at this for the first time yesterday.  It seems plausible, but some of the basic features I would expect were in their premier package and not their “basic” package.  This one just didn’t quite have me jumping up and down either.

So what is my plan?  For now I guess I wait for the Tivo Roamio OTA which could be a month or two or more away from being available to me.  In the mean time maybe I give up trying to save $110 and just buy the original Tivo Roamio.  But for now I keep watching.

Dear Tablo: you almost had me, but you lost me on your poor android support.  Give me a reasonable chromecast experience on par with netflix so that I can use my phone or tablet as a remote control and I think you probably would have my money.

Update: 9/23/2014

As of today, the 9/14/2014 release date of the Tivo Roamio OTA has come and gone.  The latest availability estimates are late October or early November.  Will I even want a tivo by then?  I guess we’ll have to wait and see.  Are there other potential options?  Does anyone have any opinions on the “Tablo”?

The cost of having all the channels you are told you should want

$110 per month.  I’m almost embarrassed to admit that is how much we were paying our satellite company each and every month for HD + DVR service.  $1320 per year.  That didn’t include any of the extra pay channels we didn’t get — like HBO, Cinemax, Stars, pay per view, etc. etc . etc.  I can’t imagine what some people must pay every month.

Was it great?  Was it awesome?  Was it worth every penny?  Did I eagerly watch each and every second of every NFL game every week?  Did I suck up endless 24 hours cable news shows?  Was I current with all the reality shows?  Did I see all the movies?  Did my kids ever run out of Disney and Nickelodeon shows to watch?  What ever happened to Hannah Montana?  Are they still making new Spongebob episodes?  I’m sure I would know the answers to all these important questions if I actually watched $110 worth of TV each and every month.

Maybe I was getting my money’s worth at some point along the way?  Somewhere, somehow, I accepted all of this.  But then we signed up for netflix and discovered so many of the shows and movies we do or did watch are available at 1/10th the cost.

Cancelled!

So we did it, we called up our satellite company and cancelled.  They gave me several chances to reconsider and even found a way to get my bill down to $68 a month.  But we held firm, they accepted our decision and we parted ways.  It was a much more pleasant breakup than others have reported with certain cable companies.

What Next?

Rabbit ears?  Running to the refrigerator or bathroom during commercials?  Making sure you are home at specific times to watch your show?  Missing your show and never getting a chance to see it?  Programming your vcr and juggling vhs tapes?   Getting up to turn the dial and adjust the antenna to switch stations?  Is this my new life?  Not exactly.

Digital Antennas

It suddenly occurred to me I had never tested the tuner on my TV.  I didn’t even know if it had a digital tuner.  So I went to the store and spent $25 on a digital antenna.  It took some fiddling around to find a place that worked, but it worked!  For me, closer to the window helped.  It was also sensitive to height and direction — table top level didn’t work work, but eye level or higher worked much better.

I discovered some interesting things when I plugged into the local broadcast world.

  • The resolution and detail was so much sharper.  Even though I got all the major broadcast stations over satellite, it turns out they heavily compress the signal, even the HD signal.  The local broadcast signal is way sharper!  If you are a sports fanatic, you should check out your game on the local broadcast station and compare to your satellite or cable feed.
  • I found a myriad of stations I didn’t know existed.  If you are into classic TV shows or classic movies, there are stations out there that feed an endless supply over the airwaves.
  • Soccer is my sport and the local mini-professional team broadcasts their games.  Cool!  I played one summer with the guy who is now the head coach, and his dad (who was the coach of that team) is now one of the broadcast announcers.  Crazy!  (I was never close to good enough to play at higher levels, but one summer I played on a team where everyone was a lot better than me, and it was a good experience.) 😉
  • I do miss my DVR.  I miss being able to set and collect shows and watch them when I want.  I miss being able to pause, skip commercials, rewind to catch something I missed.

Tivo to the Rescue?

According to the Tivo web site, they released a new version of their tivo designed especially for me.  For $50 (plus a modest monthly fee) they offer their basic DVR for recording over the air channels.  It has no cable card slot which I don’t need or want.  It is to be sold exclusively at Best Buy, but none in my area seem to have it yet … I am impatiently tapping my foot right now Mr. Best Buy.  Please take my money!

Netflix

Of course Netflix, and other streaming services such as Hulu, Amazon, and Youtube are what make this endeavor possible.

Big media companies, big content providers, and big cable/satellite companies are working hard to lock up exclusive deals and shut out those that don’t buy in.  These companies have tremendous power.  It’s why so many people stick with their current expensive providers.  But I feel the tide is shifting.  They won’t be able to lock things down forever, and eventually will have to relent to the online streaming world.  Kids these days (I’m told) watch much less traditional TV that we did as kids, and instead are watching their shows online, on their phones and tablets.

I’m not the first nut case to cut the cord, but I still feel like a pretty early adopter.   Maybe?   Maybe not?  Is this a powerful movement that is starting?  The first one to do something is the crazy nut job, especially if no one joins in.  It is the first followers though that form the bridge, that show the crowd how to follow, that make it less scary, more normal, more fun.  Before too many years go by, the non-cord cutters will be on the outside.  That is my prediction.  On line, on demand streaming and the various evolution of this new form of entertainment will soon rule the entertainment industry and our lives.

What about the internet connection

Yeah, that’s a small detail.  You can’t really stream your content without a high speed internet connection.  So maybe as time goes on, our money will shift more to the bandwidth providers, and away from the content aggregators?  Don’t ask me how much I pay per month on my internet bill!  That’s one cord I can’t see myself cutting for a long time.

Somehow I imagine that the total cost for our entertainment and communication needs won’t change all that much in the long run, but hopefully we small consumers will have more choice and control to direct our spending on exactly what we need and want.

Adventures in Aerial Image Stitching

A small UAV + a camera = aerial pictures.

SAM_0079

SAM_0057

SAM_0053

This is pretty cool just by itself.  The above images are downsampled, but at full resolution you can pick out some pretty nice details.  (Click on the following image to see the full/raw pixel resolution of the area.)

SAM_0057-detail

The next logical step of course is to stitch all these individual images together into a larger map.  The questions are: What software is available to do image stitching?  How well does it work?  Are there free options?  Do I need to explore developing my own software tool set?

Expectations

Various aerial imaging sites have set the bar at near visual perfection.  When we look at google maps (for example), the edges of runways and roads are exactly straight, and it is almost impossible to find any visible seam or anomaly in their data set.  However, it is well known that google imagery can be several meters off from it’s true position, especially away from well travelled areas.  Also, their imagery can be a bit dated and is lower resolution than we can achieve with our own cameras … these are the reasons we might want to fly a camera and get more detailed, more current , and perhaps more accurately placed imagery.

Goals

Of course the first goal is to meet our expectations. 🙂  I am very adverse to grueling manual stitching processes, so the second goal is to develop a highly automated process with minimal manual intervention needed.  A third goal is to be able to present the data in a way that is useful and manageable to the end user.

Attempt #1: Hugin

Hugin is a free/open-source image stitching tool.  It appears to be well developed, very capable, and supports a wide variety of stitching and projection modes.  At it’s core it uses SIFT to identify features and create a set of keypoints.  It then builds a KD tree and uses fastest nearest neighbor to find matching features between image pairs.  This is pretty state of the art stuff as far as my investigation into this field has shown.

Unfortunately I could not find a way to make hugin deal with a set of pictures taken mostly straight down and from a moving camera position.  Hugin seems to be optimized for personal panormas … the sort of pictures you would take from the top of a mountain when just one shot can’t capture the full vista.  Stitching aerial images together involves a moving camera vantage point and this seems to confuse all of hugin’s built in assumptions.

I couldn’t find a way to coax hugin into doing the task.  If you know how to make this work with hugin, please let me know!  Send me an email or comment at the bottom of this post!

Attempt #2: flightriot.com + Visual SFM + CMPMVS

Someone suggested I checkout flightriot.com.  This looks like a great resource and they have outlined a processing path using a number of free or open-source tools.

Unfortunately I came up short with this tool path as well.  From the pictures and docs I could find on these software packages, it appears that the primary goal of this site (and referenced software packages) is to create a 3d surface model from the aerial pictures.  This is a really cool thing to see when it works, but it’s not the direction I am going with my work.   I’m more interested in building top down maps.

Am I missing something here?  Can this software be used to stitch photos together into larger seamless aerial maps?  Please let me know!

Attempt #3: Microsoft ICE (Image Composite Editor)

Ok, now we are getting somewhere.  MS ICE is a slick program.  It’s highly automated to the point of not even offering much ability for user intervention.  You simply throw a pile of pictures at it, and it finds keypoint matches, and tries to stitch a panorama together for you.  It’s easy to use, and does some nice work.  However, it does not take any geo information into consideration.  As it fits images together you can see evidence of progressively increased scale and orientation distortion.  It has trouble getting all the edges to line up just right, and occasionally it fits an image into a completely wrong spot.  But it does feather the edges of the seams so the final result has a nice look to it.  Here is an example.  (Click the image for a larger version.)

SAM_0125_stitch

The result is rotated about 180 degrees off, and the scale at the top is grossly exaggerated compared to the scale at the bottom of the image.  If you look closely, it has a lot of trouble matching up the straight line edges in the image.  So ICE does a commendable job for what I’ve given it, but I’m still way short of my goal.

Here is another image set stitched with ICE.  You can see it does a better job avoiding progressive scaling errors on this set.  However, linear features still are crooked, there are many visual discontinuities, and it one spot it has completely bungled the fit and inserted a fragment completely wrong.  So it still falls pretty short of my goal of making a perfectly scaled, positioned, and seamless map that would be useful for science.

SAM_0372_stitch-reduced

Attempt #4: Write my own stitching software

How hard could it be … ? 😉

  1. Find the features/keypoints in all the images.
  2. Compute a descriptor for each keypoint.
  3. Match keypoint descriptors between all possible pairs of images.
  4. Filter out bad matches.
  5. Transform each image so that it’s keypoint position matches exactly (maybe closely? maybe roughly on the same planet as ….?) that same keypoint as it is found in all other matching images.

I do have an advantage I haven’t mentioned until now:  I have pretty accurate knowledge of where the camera was when the image was taken, including the roll, pitch, and yaw (“true” heading).  I am running a 15-state kalman filter that estimates attitude from the gps + inertials.  Thus it converges to “true” heading, not magnetic heading, not ground track, but true orientation.  Knowing true heading is critically important for accurately projecting images into map space.

The following image shows the OpenCV “ORB” feature detector in action along with the feature matching between two images.

feature-pairs

Compare the following fit to the first ICE fit above.  You can see a myriad of tiny discrepancies.  I’ve made no attempt to feather the edges of the seams, and in fact I’m drawing every image in the data set using partial translucency.   But this fit does a pretty good job at preserving overall all geographically correct scale, position, and orientation of all the individual images.

fit1

Here is a second data set taken of the same area.  This corresponds to the second ICE picture above.  Hopefully you can see that straight line edges, orientations, and scaling is better preserved.

fit2

Perhaps you might also notice that because my own software tool set understands the camera location when the image is taken, the projection of the image into map space is more accurately warped (none of the images have straight edge lines.)

Do you have any thoughts, ideas, or suggestions?

This is my first adventure with image stitching and feature matching.  I am not pretending to be an expert here.  The purpose of this post is to hopefully get some feedback from people who have been down this road before and perhaps found a better or different path through.  I’m sure I’ve missed some good tools, and good ideas that would improve the final result.  I would love to hear your comments and suggestions and experiences.  Can I make one of these data sets available to experiment with?

To be continued….

Expect updates, edits, and additions to this posting as I continue to chew on this subject matter.

Smart Trainer Photo Shoot

Design Observations

I really like the looks of this aircraft.  It is a unique combination of design choices that I think should provide a nice balance of good flying qualities and lots of aerobatic capability.

It calls itself a trainer, but it is not a “learn to fly” trainer.  Instead it positions itself as an entry level pattern ship.  It is light weight which means it should maneuver effortlessly and not have bad habits.  It has long lines so it should fly like it is on rails.  It has thickened wing tips so it should have excellent slow speed characteristics and be adverse to tip stalling.  Oh, and it also has all the design elements you’d want for effortless knife edge and inverted flight.  The aircraft has a semi-symmetrical airfoil.  Most aerobatic planes pick a fully symmetrical airfoil, but in my opinion, across the flight regime that this aircraft covers, the choice of airfoil shouldn’t make much difference.

While I am grounded and waiting for the snow to melt around here I took a couple pictures:

IMG_20140312_142930 IMG_20140312_142902 IMG_20140312_142834 IMG_20140312_142844

I am hoping this will be a fun airplane to go out and cut up the sky with classic pattern style aerobatics.