Art-Repair Software May Also Aid Hollywood, Military

Bijal P. Trivedi
National Geographic Today

September 17, 2002

In many ancient churches in Europe, scaffolding stands before crumbling frescoes. Atop the steel skeletons, conservators using techniques as old as the frescoes themselves can spend months if not years painstakingly filling in chips and cracks to return a work to its former glory—a process called inpainting.

Now Guillermo Sapiro, a professor of electrical and computer engineering at the University of Minnesota at Minneapolis–St. Paul, has developed a computer program that can dramatically speed up the inpainting process.

The program has applications beyond old masterpieces: Hollywood and the U.S. Navy, among others, have expressed interest in a technology that can fill in blanks and restore missing pieces.

"The goal of inpainting is to restore an image in a way not detectable by the viewer," Sapiro says. "It just has to look natural."

Sapiro was inspired to work on the program after attending an imaging conference in France several years ago. On a field trip, he went to the Minneapolis Institute of Arts to watch professional conservators at work.

Translating Art to Math

Elizabeth Buschor, a conservator with the Upper Midwest Conservation Association who specializes in art on paper, "began inpainting by first establishing boundaries and lines and then went back to fill in the color," Sapiro recalls. "We tried to translate her work into a mathematical language and imitate what she was doing."

When done by hand, inpainting is laborious and subjective. One conservator's decisions will differ from another. But Sapiro's program takes a scientific approach.

Once a damaged or incomplete picture is in digital format, a numerical value is assigned to each pixel, the smallest unit of an image. A set of algorithms—step-by-step mathematical procedures—use colors, lines and brightness to calculate the exact value of pixels surrounding the damage or gap.

Like interpolating trends from a graph, the program extends the cut-off lines and borders, then fills in the missing areas with color, working from the edges inward.

"It looks like a wonderful tool," says Dan Kushel, distinguished teaching professor at the Art Conservation Department at Buffalo State College in New York. "It won't take over what we do, but it could certainly speed it up. We can work things out and do a virtual reconstruction first before applying real materials to the original works."

Removing Movie Bloopers

For the past decade, conservators have used computer programs like Adobe PhotoShop that allows them to manipulate the digital image and simulate the restoration. But the work is iffy and time-consuming.

"What Sapiro's new program does is automate the process," Kushel says.

The potential use for an inpainting program is as vast as the human capacity to make errors or to do damage.

Sapiro mentions Movie-Mistakes.com, where buffs gleefully point out flaws like the car that drives through the background of a scene in "The Lord of the Rings: The Fellowship of the Ring."

The film repairing or replacing process usually involves frame-by-frame restoration. Sapiro's program can automatically remove the blooper and inpaint all frames. Film and special-effects companies have already made inquiries.

The U.S. Navy got involved in backing Sapiro's research because of surveillance, which often depends on remotely transmitted images.

Inpainting Gaps In Military Surveillance

Sapiro has demonstrated that blurry, speckled or even partial surveillance images in which a proportion of the pixels are blank may still hold enough information to inpaint what's missing.

The advantages are twofold: Poor-quality images can be recovered, and images can be deliberately transmitted at lower quality, which is faster, because the lost information can be recovered.

The Navy also is funding research by Andrea Bertozzi, a professor of mathematics at Duke University, in Durham, N.C., who is collaborating with Sapiro.

Coincidentally, the mathematics that underlie Sapiro's inpainting program are similar to the Navier-Stokes equations that describe some motions of fluids like air and water, Bertozzi says.

Sapiro focuses on the propagation of images through space; Bertozzi, on fluids—air around an airplane wing, silicon oil on a wafer or the flow of viscous liquids.

Both Sapiro and Bertozzi anticipate that the mathematics of fluid dynamics should help Sapiro in his work—another example of how scientific convergence can inpaint all sorts of blanks.