Training-Free Neural Matte Extraction for Visual Effects: Results and Failure Cases

6 Jul 2024


(1) Sharif Elcott, equal contribution of Google Japan (Email:;

(2) J.P. Lewis, equal contribution of Google Research USA (Email:;

(3) Nori Kanazawa, Google Research USA (Email:;

(4) Christoph Bregler, Google Research USA (Email:


We use images, trimaps, and GT alpha from the dataset [Rhemann et al. 2009] to allow comparison to the ground truth. Fig. 1 shows a moderately challenging case with extensive hair, some of which is similar to the background color. Fig. 3 shows several additional examples. While the ground truth and estimated alpha maps appear identical at first glance, small differences can be seen upon enlargement. These may be due to imperfections in our algorithm, but also may reflect differences in color spaces between our algorithm and the ground-truth estimation process.

Fig. 2 shows the extrapolated foreground 𝐹ˆ and background 𝐡ˆ for a particular image. Note that the inpainted colors need to be correct only in the unconstrained region surrounding the object border in order to allow compositing over a new background, as is the case in this figure. The unrealistic extrapolated 𝐹ˆ over the pure background regions is ignored.

Figure 4: (Left) objects with holes can be a failurecase. (Middle) ground truth alpha map. (Right) estimated alpha map.

4.1 Failure cases

While objects with "holes" sometimes yield good mattes (e.g. the plant in Fig. 3), they are also a failure case (Fig. 4). In some cases the trimap can be adjusted to highlight the missing holes, though this would be laborious in cases such as the cup in Fig. 4.

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.