3D Photo, Photosynth, and Ground Imagery

Ruthless Design

It must have been eight or ten years ago I attended a conference where a couple of the wizards from Media Lab within Microsoft Research briefed us on their internet very-large image management design then called SeaDragon and a second really cool digital imagery tool Photosynth.  These two integrated tools are incredibly unappreciated in my opinion.

SeaDragon has become Deep Zoom, the critical element of Microsoft's Silverlight performance for viewing images and graphics.  SeaDragon represent a sea-change (whoops) that allows you to make a gigapixel (1000 megapixels) panorama if you want, and it can be opened by a viewer on the Web or in a mobile app in just a couple of seconds. And despite Silverlight's uncertain future due to HTML5, the DeepZoom group within Bing have created an open JAVA tool set in case you wondered?

DeepZoom offers to images what Google Maps offers to a map layer.  The viewer only calls out the particular tiles necessary based on the view.  In DeepZoom more and more image detail is provided as image scale is shifted-in whereas in Maps nested layering's of increasing details are provied.  Very alike in their design although the construction and source of the tiles at scale are quite different.

But it was Photosynth that really captured my interest.  I have found the Photosynth tool one of the really incredible ideas for spatial media geo-photographers.  And it is that interest since I first saw what it could do back in 2007 that has fascinated me ever since, particularly the spatial 3D model point cloud.  And this point cloud is the magic as it can generate if from set of seemingly dis-united images having only a common inward or panographic viewshed. It is LIDAR-like at a fraction of the cost and offers not a few hundred thousand points but essentially millions via simple point-and-shooters or the ever-evolving smart-phone.

Synths are generated locally via the PhotoSynth app and uploaded to the PhotoSynth web site.  The Micorsoft desktop too will organize a set of images of a common object of interest and related them together such that they form a 3D point-cloud that stitches/related the image sets to their points of view and intersections of viewshed. Images will Synth together regardless of when or with what camera or point of focus.  Only requirement is that there is a common overlap...

"Photosynth was inspired by the breakthrough research on Photo Tourism from the University of Washington and Microsoft Research. This work pioneered the use of photogrammetry to power a cinematic and immersive experience.' 

"The first style, and the one that still uniquely defines the product, is what we call a “synth”. A synth is a collection of overlapping photographs that have been automatically reconstructed into a 3D model of the space. The synth-ing process solves the same problem our brains are confronted by when we look at the world: the slight differences between what our left and right eyes see gives us cues about how far away different parts of the scene are. In a similar way, the differences between two photos taken from nearby positions can be analyzed to determine which parts of the scene are close, and which are further away. Amazingly, the synth-ing algorithm can reconstruct a scene of 200 photos in just five or ten minutes on an average laptop."

My interest in "Synth-ing" has been in its application to creating a point cloud from geotagged images. This quest led me to Vexcel, a Microsoft company located in Boulder, Colorado, that was tapped to continue working on the science of PhotoSynth-ing.  Vexcel has gone through a number of iterations of purpose and are now some sort of a black-box.  But at one time they were working on the extension of the Photosynth concept to be a spatialist's tool named GeoSynth.  Red Hen worked with them for a bit on their use of spatial motion video on the promise Red Hen might be able to apply the stand-alone tool for spatial ground imagery users via a digital mapping reconnaissance toolkit. We were excited as GeoSynth evolved to include a multipurpose design that had great potential.

• Drive by a set of buildings collecting a fast series of high-resolution photos, and process the photos into map-accessible products in minutes.
• Photograph a room from various perspectives, magnifications, and times of day. Navigate the synthesized collection spatial for situational awareness or mission training.
• Quickly and automatically form a 3-D aerial view from a series of photos collected from an unmanned aerial vehicle (UAV).
• Combine scenes from digitized video or handheld camera stills such that all images receive a full geotagging relationship... with a positional accuracy of under a meter if only a few of the image set included a reasonable GPS geotagged position.  This particular magic would allow legacy imagery scanned to digital to then become part of the synth.

GeoSynth may still reside somewhere in Bing but we still have Photosyth and a number of derivatives and innovations based on the original concept of photo tourism.  One part of the implied value of any Synth are the derrived camera orientations.  These camera orienations  can still be gotten but not in the GeoSynth's KML totally cool form - so cool the Vexcel developer soon departed after Synth-clouds began to form over Google Earth-scapes? There is a small group of extensions that can further distill PhotoSynth's hidden bits as found via The best of the homebrew Photosynth-data exporting tools.

My Uluru Synth was derrived from clipping some 80+ images from Google Earth on the maybe I could get a point cloud?  Take gander at the example live Synth just below.



Another really great example of the 3D cloud of a landscape comes from Mokojo's visit to Giza



The evolution of digital images to create 3D models continues.  There is one product from an academic group in Switzerland that created a really elegantly simple personal UAV,  Another like solution but tuned for the 3D modeling of landscapes can be found in Pix4D.  The Swiss are going after these ideas in a significant way.  Disney has a group there that has created a photo3D tool for their magical purposes as well.



AutoCAD has been interested in the image to 3D model process for some time.  Some years ago they initially offered a free tool named Photofly.  Photofly initially received some great reviews as it was supported by the "illustrious AutoCAD software, [which could] stitch standard digital photos into accurate 3D models. You don’t need a fancy camera — a point-and-shoot is more than good enough — and by leveraging Autodesk’s cloud computing cluster, you don’t even need a powerful computer to use Photofly. 3D models can be created out of faces, static objects, interior rooms, and even external architecture. Best of all, though: Photofly 2 is free." as described by Extremetech back in earily 2011.  It is now known as 123d Catch and is still free as long as you own a legit license to AutoCad.

Tgi3D PhotoScan is camera calibration solution that with some effort can create a wire frame with image textures.  Photomodeler offers a good bit of easy explanation of the photo-3D process.

3Defy is a rather neat small tool that with a single image create cha-cha 3D model.  You can also get a free solution from Cornell that has some neat results.



Indigo-i from Hover, Inc. is a hybred of SketchUp, the Google tool for 3D modeling and insertion into Google Earth now owned by Trimble.

3D Software Object Modeller Pro is a commercial product that creates a fully textured 3D model based on in-ward images.  Its a rather convoluted process that actually makes quite good models... at a price.

Hyper3D is a GeoSynth-like sort of that has evolved into a 3D modelling solution that then can be printed.  The new outfit and business model can is knwn as Cubify  I have not put really much time into its evaluation in the 3D additive manufacturing design but it sounds like a process that is likely to evolve.  To make it work properly you are required to use their viewer though?

And lastly, there is another way that is a hybred of the whole process and can create 3D renderings with a Lidar process.  ASC has created a 3D Flash LIDAR cameras operate and appear very much like 2D digital cameras. 3D focal plane arrays have rows and columns of pixels, also similar to 2D digital cameras but with the additional capability of having the 3D "depth" and intensity. the LidarCam has a 128 by 128 sensor that creates a lidar frame - no scanning required.  Why is is a neat idea is that the LidarCam can operate at up to 30 frames per second so you have essentially a lidar video and its 3D effects and accurate instantaneous measurements.  It really gets neat when a 36 megapixel color image is draped over its 16,000 instantaneous measurements.