Gigapixel Imaging at the Acropolis of Athens - Parthenon

SIMULZINE#2-colver-parthenon-gigapixel-screenshot

My latest project, with orbitlab, was the creation of an interactive virtual tour at the Acropolis of Athens. Amongst others, it involved the acquisition of five gigapixel scale images from the Acropolis Monuments. The project was an initiative of the Acropolis Restoration Service, the organization responsible for the restoration of the tormented and partially destroyed 2500 years old Acropolis Monuments over the past 30 years.

The decision to adopt state of the art technological solutions to “bring online” the entirety of the Acropolis is not merely a statement for the preservation work undertaken by the organization, but also a great contribution to the institution’s mission to promote remote access to cultural heritage.

Choosing the Point of View

The monument facades to be photographed, were previously examined for proper lighting conditions, required for a long gigapixel shooting. The point of view was carefully chosen with respect to insidious distortions that come up when you are too close to the monument. About 30 m in front of the east facade would not only show details of restoration works, but also produce the “wow!” effect when zooming in the final gigapixel image. The sun should be high, minimizing shadows. The east facade of the Parthenon was photographed by our crew at noon 13.00 for about one hour in September of 2010.

horizontal field of view of gigapixel shoot

figure 1
horizontal field of view of gigapixel shoot
hfov: horizontal field of view, d: distance of camera from object, red lines: equal sides of isosceles triangle, blue dotted line: Parthenon East Facade, red circle: camera, image: Acropolis groundplan orthophoto

Things to consider before shooting

  • Weather : sunny, clear atmosphere, no clouds (bad for stitching)
  • Sun Position: The sun has to be in a convenient position (during the hole shooting!) to properly light the part of the monument to be photographed
  • Crowding : You don’t want people in your shots. When possible set up a perimeter. Prepare for delays, repeated shots, manual shooting with the robotic arm.
  • Focusing : You may want to switch to manual focusing at areas where foreground objects could disturb your depth of field
  • Know your gear: Estimate the time needed to complete the shots! You don’t want the night to come or run out of battery. Estimate file size, keep backup!
  • Take Notes: Especially about duplicate shots and downloaded photographs from your camera (saves a lot of boring work afterward!)

Available Gear

  • Camera : Nikon D700
  • Lens : Sigma 500mm f/4.5 EX HSM
  • robotic camera mount : EPIC Pro

Our crew programmed the robotic rig to capture an area of 55° horizontal and 40° vertical field of view with a focal length of 500mm, that means a 16 X 17 grid with 272 cells (camera positions) - in three different exposure levels - that makes us 816 photographs and about 8.5 GB of raw data on the D700.

computed grid based on shooting parameters and covered area

figure 2
computed grid based on shooting parameters and covered area

Post-processing

The challenge at gigapixel imaging is to capture a solid dynamic range over the whole scene, and keep the details in light & dark areas at varying lighting conditions, this makes blending afterward an easier task. With the sun being the only light source (and a moving one…) things can get a little tricky. With time we know that when taking care of a few basic things in situ (at the image acquisition process), stitching algorithms will make a good job (only a few control points had to be added manually), but what matters most in gigapixel imaging is blending. You’ll want to include every possible technique that makes it easier for blending algorithms to produce an acceptable and homogeneous result over your final gigapixel image.

Below, I will describe the post-processing workflow, from converting the RAW photographs, HDR processing, stitching, blending & tonemapping up until the web deployment of the final gigapixel image.

Raw Conversion

The first task of post processing this big amount of photographs is to set a uniform white balance throughout all of them. We usually use a gray card for this procedure but it can be done in many different ways at the raw conversion process or in situ. So from the camera’s raw format (most DSLRs today compress at 14bit of color information) we convert to 16bit .tif without loosing valuable dynamic range. The use of a full frame sensor (Nikon D700) often results to strong vignetting around the image, so this had to be also corrected during the raw conversion.

raw conversion of all photographs involves white balance & vignetiing correction

figure 3
raw conversion of all photographs involves white balance & vignetiing correction

Grid Alignment

After removing possible duplicate shots, we are ready to align our set of 816 photographs in our favorite stitching & blending software. We favor PTGui not just because it has a new functionality supporting robotic rigs but because it is pretty straightforward and supports the stitching process with a variety of other tools. It has also proven to handle large data sets quiet stable. Hugin, the open source alternative fairly straightfoward too and can be effectively used easy for small panoramas but lacks support for gigapixel images. Below you can see a screenshot of 272 positions with 3 exposure brackets aligned in the PTGui editor.

816 photographs automatic aligned in ptgui, 272 grid positions, 3 exposure brackets

figure 4
816 photographs automatic aligned in ptgui, 272 grid positions, 3 exposure brackets

HDR acquisition & processing

HDR Processing will contribute effectively to the results and is standard for professional work. For multiple exposures acquisition, a step of 1EV was sufficient for the well lighten east facade of the Parthenon and was chosen for the 3-exposures bracketing sequence. Basic panoramic imaging requires the camera to be in aperture priority, to keep a constant depth of field during the multiple exposures. So we let the camera do the metering and the lens to focus. Taking some test images will help to adjust the bracketing sequence. The goal is to capture as much of the dynamic range of the scene in varying lighting conditions. Images were taken with an aperture size of f11 (best performance of our lens) and exposure times from 1/25 to 1/2500 second.

Figure 5 showcases the detail accumulation of the bracketed sequence for a single grid position. Click the button below to view an HDR visualization of the resulting 32bit image after exposure blending. Figures 6,7 & 8 display the separated exposure brackets for the whole set of loaded images in PTGui. Click the button below to view an HDR visualization of the resulting gigapixel 32bit image after exposure blending. (To view the HDR visualizations you need a WebGL enabled browser)

gaining detail from bracketed set of 3 exposures results to a 32bit HDR image

figure 5
gaining detail from bracketed set of 3 exposures results to a 32bit HDR image

272 normal exposed photographs aligned

figure 6
272 normal exposed photographs aligned

272 over exposed photographs aligned

figure 7
272 over exposed photographs aligned

272 under exposed photographs aligned

figure 8
272 under exposed photographs aligned

Stitching and Blending

There is a variety of reliable software in the market, suitable for gigapixel processing, but it is worth to say that they all use the same stitching algorithms (panotools). Most stitching software are just graphical user interfaces for these algorithms packed with a lot of functionalities that make our lives easier. An overlapping of 25% for neighboring (horizontal & vertilcal) photographs was sufficient enough for the automatic detection of control points throughout the whole gigapixel image.

The following figure depicts the whole post-processing workflow after raw conversion. Although very few, there were some stitching errors that were of minor importance (like the one marked below), that we chose not to retouch. To preserve the empirical provenance of the final image no retouching should be applied on pixel information of the actual monument.

stitching, blending & tonemapping of overlapping areas

figure 9
stitching, blending & tonemapping of overlapping areas
red area: overlap of horizontal neighbor, multicolored labes: control points, green circle: stitching error

When we’ve done a good job at capturing the dynamic range then blending of the images on overlapping areas will not be a hard job. PTGui supports three blending algorithms, the internal one, smartblend and enblend. We found out that smartblend can’t deal with big image sizes, it simply crashes during the rendering process. Enblend was a good choice for this work. Our final render is a 32bit HDR image in gigapixel resolution (.hdr or .exr format). Tip: Render small images first to see if blending suits your setup before going for maximum resolution.

Next step will be tonemapping so we can display the HDR image on traditional displays.

Tonemapping

As we seen before HDR images include more color information then traditional bitmaps. More than display hardware can use. These digital images can’t be displayed in traditional mediums like paper or common monitors or LCDs, with limited luminance. There is an ongoing development of HDR displays that can make use of 32 bit color information. Dolby is making efforts to license a HDR display technology to LCD manufacturers like Sony.

So our next step will be the tonemapping of the HDR gigapixel image, so it can be displayed in traditional displays. Tone mapping is an image processing technique that compresses (or maps) large color information to limited dynamic range formats like .tif or .jpg, with the goal to maintain as much image detail as possible. It is particularly helpful in scenes where dark and bright areas coexist (eg. high contrast scenes). Although tone mapping is mostly used by digital “cinephotographers” to produce VFX, illustrative or aesthetically pleasing images, our goal was to enhance realism and depict the monuments objectively for display on regular computer monitors. Below you can see different tone mapped variations of the Parthenon East Facade:

tonemapping of 32bit image vivid version

figure 10
tonemapping of 32bit image (vivid version)

tonemapping of 32bit image surrealistic version

figure 11
tonemapping of 32bit image (surrealistic version)

tonemapping of 32bit image (exposure corrected version)

figure 12
tonemapping of 32bit image (exposure corrected version)

tonemapping of 32bit image, retouched sky, cropped (final version)

figure 13
tonemapping of 32bit image, retouched sky, cropped (final version)
width:: 44318 pixel, heigth:: 23914 pixel, file size: 8.5 GB

Deployment - Tiling

Well, creating the gigapixel image is one matter. Another important aspect is how to display this huge amount of pixels over the internet. So let’s talk about deployment. Current possibilities depend on browser plug-ins like flash or silverlight required to display rich media within web environments. There are numerous viewers like krpano (our choice) and zoomify for flash or deepzoom for silverlight. Unfortunately HTML5 and its Canvas element isn’t currently suitable or mature enough to handle gigapixel images (both krpano and zoomify support HTML5 rendering, but not for gigapixel scale images). Another drawback regarding deployment of gigapixel images is the currently weak support for mobile devices. The only working example I could test was Gigapan for iPad, a native coded gallery like app for gigapixel images of the gigapan gallery.

The method for displaying gigapixel images effectively is based on tiling, just like maps are displayed in browsers. Figures 14 & 15 describe the process of tiling and displaying gigapixel images over the internet: The final image is spitted into multiple levels of smaller tiles that load depending on the navigation path of the user (panning & zooming). All the have the same resolution over the different zoom levels. Every zoom level is a plane with a varying quantity of (equal sized) tiles and updates the viewport of the user by loading the corresponding tiles.

tiling setup of N zoom levels

figure 14
tiling setup of N zoom levels,
every levelN has tN=t04N quantity of tiles,
(on every level tile resolution is constant)
tN: quantity of tiles on levelN, t0: quantity of tiles on level0 (zoomed out image), fovN: field of view starting point for loading tiles out of levelN, max: maximum field of view D: parameter of fov variations depended on zoom and interaction settings

hypothetical 3d space arrangement of different zoom levels

figure 15
hypothetical 3d space arrangement of different zoom levels
red line: users navigation path through different zoom levels, green area: users viewport - displaying corresponding tiles based on zoom level “entered” by the user

Further thoughts

The goal in gigapixel imaging, especially for cultural heritage work. is to capture the scene, or the object of interest, as objective as it gets. There is no decissive moment like in traditional photography. It is best to perceive the camera as a sensor for color acquisition than as a moment-capturing-device. Good field work leads to better lab results ;)

To preserve the empirical provenance of the final image, at all stages, records, notes and project files should be kept for the different processes of the above described workflow. This way, the whole post-processing process can be traced back to the original raw photographs and the final gigapixel image acquires the status of a trusted representation of the “real-world” in digital form. Read more about empirical provenance and the concept of digital surrogates in the inspiring paper “A Digital Future for Cultural Heritage”.

Over the last few years there is an ongoing race for the biggest gigapixel image. Wide metropolitan areas or large scale events are captured in gigapixel scale and displayed over the internet with often poor image quality. In general the final pixel size of an image depends on photographic gear and processing power. In contrast, the final image quality depends on consistent applying of some of the above mentioned techniques. Gigapixel scale images can be printed in large dimensions, exploiting their full detail, for a museum viewing experience. In addition gigapixel images can be embedded in 3d space as backdrops in vfx shots or game engines.

Expanding the boundaries of photography”

Gigapixel images are explorable images, that provide the viewer with access to a variety of information otherwise hidden to the physical visitor of an archaeological or other site. They represent the virtual binoculars of the user. They can serve documentation and restoration purposes over a variety of disciplines. In addition gigapixel images are suitable for fulldome or large theater projections. Gigapixel imaging was taken out of scientific research and pioneered for the internet and the rest of us by the xRez team. It’s about “expanding the boundaries of photography” as they say. You can watch a recent revealing lecture about gigapixel imaging by Greg Downing and Eric Hanson of xRez here.

Thank you for reading.