Final Project: TRIANGULATION MATTING AND COMPOSITING!

Niraek Jain-Sharma

Shooting the Pictures

This part of the project took me quite a while. First, I tried to take pictures of a glass and a cup of spatulas, behind backgrounds such as my cushion and my shirt. However, what ended up happening was the backgrounds/item would move too much. See below for some examples of the failed attempts.

Backgrounds (Failed)

Items (Failed)

Notice how the shirt background had shadows, and seemed to move from the frame. This is because there is some minor wind where I shot the photos, so the shirt wasn't to be trusted as a stationary background. Moreover, the lighting seems to change because of shadows.

Finally, I decided take a picture of my dishwasher soap (who doesn't like dishwasher soap!), with the wall as one background, and a black yoga mat from lululemon.

It also took me longer than I want to admit to figure out the order to take pictures. Initially I tried taking the picture of the wall background, and then hanging the mat, then adding the item and taking the picture in front of the wall and mat. However this meant I would have to guess where the mat was initially. The way to do it is
1) Take a picture of background 1
2) Take a picture of the object in front of background 1
3) Take a picture of the object in front of background 2
4) Finally take a picture of background 2.

See below for the images I ended up using

Backgrounds

Items

Triangular Matting Algorithm

Now, we had to figure out how to use these 4 images to solve the traingular matting equations and get the alpha values and complete image C0. From the slide 15 the project points to, these equations are as follows.

Using the hint in the project description ("it’s very useful here to consider the colors of Co to be premultiplied by the alpha"), we can relabel alpha_o*R_o as R_o, and likewise for the other colors. So, we want to solve for 4 unknowns, and to do this we can put them in a matrix, and fill out matrices that delineate the equations above. We get the following equation.

If we expand out the first row for example, we get 1*R0 + 0*G0 + 0*B0 -Rk1 * alpha = R1 - Rk1, or simplifying, R0 - Rk1*alpha = R1 - Rk1 or rewriting, R1 = R0 + (1-alpha)*Rk1 which is the same as the first equation in the image before. Note that we have these equations for every pixel color at p.

To solve this equation, we need to solve Ax = b, but since the A matrix isnt square, we use a pseudo inverse to get the solution x = pseudo(A) * b. We then construct the image C0 and the alpha values, shown below.

C0 final image


Alpha values

Finally, let's put this image somewhere! What about on a windowsill, since every windowsill needs soap. Mathematically, we take the windowsill image and multiply it by 1 - alpha for each pixel, then add it to the final c0 image, or written out, c0 + (window * (1-alpha)) for each pixel.

Window

Blended

Note how the bottle looks like it could actually be there! The alpha values blend it well, and we can see the background behind it while still making out the bottle.

Warping?

One thing that worked out nicely above is that the image could stay in the same spot and it looked realistic, since its placement in front of the window was perfect. But what if we wanted the placement to be elsewhere on the window image, say on the small sill? Then, we need to apply homography and warping. We choose 4 points around the soap bottle, and 4 points on the window image we want the bottle to go to, and then apply warping, see below!

Bottle Points

Window Points

Now for the warping!

Warped Backgrounds

Warped Bottles

Now we repeat the earlier algorithm on these 4 images.

Warped Final Bottle

Warped Final Alphas

For the finale, we blend the window and the image together using the c0 + (window * (1-alpha)) method!

Final Window + Warped Bottle Combined

Final Comments: Because I took the image of the bottle slightly above its bottom , it looks like it's half a dishwasher soap bottle. So, a better way to take the 4 images would be camera facing straight downward, and getting the entire bottle, since taking it upright usually ends up including the surface which the object is standing on. I really had fun doing this project and seeing how cool it is to blend images together in this way, using the triangular matting algorithm!.

Final Project: Poor Man's Augmented Reality!

This project really interested me because I love the idea of augmented reality, and implementing that, even if a poor man's version, was very enticing to me. The only box I had lying around was a macbook pro box, so I drew my points on that. I used matlab to label the first frame's points - see below.

Box + Points

Next, I took a video to get the frames I would need to get all the points at the intersections. See below for the video - I took it pretty slowly so that the points would be detected relatively easily.

Box Video

Now, I needed to use a method to get the points for all the 144 frames. This was a bit tricky - I chose to use an "Off the Shelf Tracker" in order to accomplish this. I used the MedianFlow tracker from the legacy cv2 library. First, I needed to get bounding boxes for each of the points in the initial frame. See below for an example of one of the bounding boxes, which uses a square of 8x8 around the point, the red rectangle (made the width higher for demonstration purposes.)

Bounding Box

Now, we need to calculate the bounding boxes around each point, which we got from the initial frame labeling, and apply the MedianFlow tracker to all of the frames on each point. Note that at each frame, the flowtracker updates the bounding box, so to get the point estimate, I needed to average the lower left and upper right corner of the bounding boxes. Finally, I got all the points, and used cv2.circle to add the points to all the frames and generate a video with the points labeled, below:

Box Video w/ points

Now, we use the world coordinates and the image coordinates to create a camera matrix in order to relate the two, to make our augmented reality. We use the following equation, and solve it using least squares *for each frame*. Note that we have the 2d image points from our previous exercise, and since the world coordinates are always the same, we solve for the matrix, and then multiply it by the cube axes to get our cube!



First, we put the cube on top of the box, by using the axes coordinates [1,1,1], [1,1,2], [2,1,2], [2,1,1], 1,2,1],[1,2,2], [2,2,2], [2,2,1]]. Then we loop through each frame, calculate the matrix, multiply it by the axes, and get the new cube placement! Putting the frames together gives us AUGMENTED REALITY! First we show the cube, then we show the movie.

First Frame Cube

Entire Video

As we can see, it does pretty well! One issue is that the cube ends up being a little slanted near the end, which probably means our points aren't great, so ifi we had moreaccurate points we'd probably get a more straight cube.

B&W - Weirder image, with cross pillar

For this part I decided to do a cube-like thing, except have crossing pillars. The goal was to see how the image would look. I also made the cube bigger. Below was the result! First Frame Cube Weird

Entire Video Cube Weird

Overall I had a great time working on these two projects, and was able to use a lot of things I leraned in class to better understand photography and images, and computer vision. Thank you!