Imaging Lab 3: Improved Pixel Scanning

EECS 16A: Designing Information Devices and Systems I, Spring 2015

Name 1:

Login: ee16a-

Name 2:

Login: ee16a-

Overview

This week, you will experiment with two methods to try to produce better images: scanning multiple pixels at a time, and using multiple scans to produce a better image.

As with the last lab, you will begin by checking that the sensing circuit works and that the projector is correctly connected to the computer. Next, you will write code to generate the multipixel pattern that the projector will use to scan through the image. Finally you will use your code and scanning matrix to image a card!

In [20]:
# Import Necessary Libraries
from  pylab import *
import struct
import numpy as np
from scipy import random
import time
import scipy
from scipy import linalg
import serial

Task 1a: Test the Imaging Setup

When dealing with a complicated system it is often useful to perform a "sanity check" to make sure the system performs as expected before trying to make improvments. We will begin this lab by quickly making sure that the projector works for the single pixel scan from Lab 2. The circuit diagram and setup instructions from previous labs are provided below for reference.

Use the oscilloscope to confirm that the sensor you built last week still responds to changes in light. You should be able to reproduce the same output you saw before.

Task 1b: Upload Arduino Code

Upload the AnalogReadSerial program to your Arduino.

Task 1c: Projector Setup

  • Place the breadboard, Arduino, and solar cell in the stand.
  • Connect the USB cable to the Arduino.
  • Connect the HDMI and power cables to the projector.
  • Turn on the projector and select the HDMI output.
In [ ]:
#TODO: Create the matrix with ones along the diagonal used to scan 30x40 images from Lab 3 (exact same as last week)

# Save the mask for use with the projector
np.save("imaging_mask.npy", H) 

Run the capture image.py program just like Lab 2 and make sure you are able to produce an image before moving on.

Task 2a: Multipixel Scanning

In the previous lab, each pixel was only illuminated once. This system is not very robust, since errors from various sources could cause a few pixels to have bad readings, leading to a poor quality images. These errors may manifest themselves in the form of detail being obscured or speckled dots in some regions of the image.

To address this, we will illuminate each pixel multiple times for a more robust reconstruction. To begin, we will only focus on a 2x2 pixel section of the image. As we did last week, we will create start by creating a small version and then scale this up to image a real picure.

Multipixel Scanning Matrix

Our goal is to illuminate certain pixels more than once. How can we do this without going through more than a total of 4 imaging masks? The solution is to illuminate more than one pixel per mask. But how do you choose which pixels to illuminate in each mask?

Begin by assigning each grayscale value in the 2x2 section to a variable, $iv_{ij}$, where i is the row and j is the column. Your matrix corresponding to the 2x2 image will look like this: $\begin{bmatrix} iv_{00} & iv_{01} \\ iv_{10} & iv_{11} \end{bmatrix}$.

Representing the sensor reading from the kth mask by $sr_k$, we write the following system of equations: $$iv_{00} + iv_{01} + iv_{10} = sr_1$$ $$iv_{01} + iv_{10} = sr_2$$ $$iv_{10} + iv_{11} = sr_3$$ $$iv_{11} = sr_4$$

How would you represent these equations in terms of a matrix multiplication? Remember, your image vector and sensor vector are both 1D vectors, but your imaging mask should be a matrix.

Create the matrix MH that represents the series of linear equations above.


**Example output for a similar matrix**
In [29]:
##TODO: Create imaging mask MH (dimensions should be 4x4)

plt.imshow(MH, cmap='gray', interpolation='nearest')

Iterate through the 4 individual masks and display them.
Hint: Reference your code from last week where you checked to make sure the scanning matrix was producing the correct pattern.


**Example output for a similar matrix**
In [14]:
# Iterate through different masks
plt.figure(figsize=(18,18))
for j in range(0,4):
    subplot(5,5,j+1)
    ##TODO: Use each 4x1 column to create a 2x2 image
    proj = 
    plt.imshow(proj,cmap='gray', interpolation='nearest');
    title('Mask ' + str(j+1))

Task 2b: Scaling up the Scanning Matrix

What we want to do is scan 2x2 sections of the image at a time rather than scanning single pixels. To do this we can use the same type of pattern from our 4x4 MH matrix created above. To scan a 6x8 image what dimensions must our square scanning matrix have?

Design the first 4 columns of the scanning matrix for the 2x2 pixel area at the top left of the image. Using the MH matrix, this means the first mask will illuminate 3 pixels, the second and third masks will illuminate 2 pixels each, and the last mask will illuminate 1 pixel.

** Design a 48x4 matrix that represents the first 4 columns of the imaging matrix. ** Name it Mtx_temp.

Draw out what Mtx_temp should look like and think about the goal in terms of matrix multiplication. There should be very few pixels in each column that are non-zero. Think about how you should index into Mtx_temp to change those values.

In [25]:
##TODO: Create Mtx_temp to create images of size M by N (in this case 6x8)
c=2
M = 3*c
N = 4*c

** Plot what the 4 different masks would look like (these should be 6x8 images). ** Does it make sense? Adjust Mtx_temp if needed.


**Example output for a similar matrix**
In [26]:
figure(figsize=(18,18))
for j in range(0,4):
    plt.subplot(5,5,j+1)
    ##TODO: Use each 48x1 column to create a 6x8 image
    proj = 
    plt.imshow(proj,cmap='gray', interpolation='nearest');
    title('Mask ' + str(j+1))

** Run the code below that will repeat the first four columns(Mtx_temp) across the matrix H and create the full 48x48 scanning matrix **

The code will also plot which pixels will be illuminated with each mask, and you can check if your Mtx_temp was correct!

In [23]:
# Create matrix H by extending the first 4 columns of the mask to a larger matrix
first = True
H=[]
for i in range(M//2):
    k=i*2*N
    for j in range(N//2):
        if first:
            H = Mtx_temp
            first = False
        else:
            H = np.hstack((H, np.roll(Mtx_temp, k+2*j, axis=0)))
            
plt.figure(figsize=(18,18))
for i in range(M*N):
    plt.subplot(M,N,i+1)
    plt.imshow(np.reshape(H[:,i], (M,N)), cmap='gray', interpolation='nearest')
    title('Mask ' + str(i+1))
    
plt.figure()
plt.title("All Masks")
plt.imshow(H, cmap='gray', interpolation='nearest')

Task 2c: Real Imaging

Finally, we will use our matrix to image a real picture. Because our picture is fairly large, we want each individual mask to have dimensions 30x40 to match the 3:4 aspect ratio of the projector.

** Create a matrix H that could be used to scan a 30x40 image.**

In [23]:
##TODO: M and N appropriately
M = 3*c
N = 4*c

##TODO: Create Mtx_temp to create images of size M by N (in this case 30x40)

# Create matrix H by extending the first 4 columns of the mask to a larger matrix
first = True
H=[]
for i in range(M//2):
    k=i*2*N
    for j in range(N//2):
        if first:
            H = Mtx_temp
            first = False
        else:
            H = np.hstack((H, np.roll(Mtx_temp, k+2*j, axis=0)))
            
plt.imshow(H, cmap='gray', interpolation='nearest')
            
# Save the mask for use with the projector
np.save("imaging_mask.npy", H) 

From the command line (in the current directory), run

python capture_image.py

The script projects patterns based on the masks you designed, imaging_mask.npy. The sensor readings will then be saved into an array named sensor_readings.npy.

** Recreate the image vector from the sensor readings. Remember, you cannot just use the reshape command this time! **
Hint: Think back to the representation of the imaging system as taking a matrix-vector product. How can we reconstruct the original image?

In [9]:
sr = np.load('data/sensor_readings0.npy')
print "Raw sensor readings:"
print sr[:20]

#TODO: Create the image vector from H and sr
#iv = 

# Display the result
plt.figure(figsize=(3,4))
plt.imshow(np.reshape(iv,(30,40)), cmap='gray', interpolation='nearest')

Warning: the reconstructed image may not look like the output you expect. Considering our imaging system differentiates between colors based on the deviation from some baseline value as seen in Lab 1 (i.e. the voltage from the solar cell increases or decreases with light intensity), is the amount of change between the raw sensor readings printed above obvious?

** Recreate the image vector from the sensor readings after centering the data about 0. **
Hint: What is the difference between the values of $y_0=sin(t)$ and $y_1=sin(t)+2$? What are the average values of $y_0$ and $y_1$?

In [ ]:
 

Did this change make a difference? How do the multipixel images compare to the single pixel images from last lab? Can you think of possible causes for these differences?

Task 3: Dealing with Noise

As you have seen throughout the labs in this module, measurements in the real world are never perfect and can be affected by a a variety of factors; instead of seeing perfect, flat lines corresponding to the voltage changes from our moving white squares, we instead see these ideal lines overlaid with small, random fluctuations. The term noise is used to refer to any of these unwanted or unknown modifications to our ideal signal. Noise can come from a variety of sources including the measurement instruments, poor connections, other instruments in the room, etc.

Task 3a: Properties of Noise

We have already seen one method of unwanted noise in Lab 1; the stored charge in the capacitor was able to smooth the output of the solar cell and eliminate the rapid voltage variations we know do not correspond to our changing projector image. After briefly exploring the properties of noise, we will take advantage of these properties to improve our image.

We can simulate this process of adding noise by taking a vector of measurements and adding a small random number to each measurement as shown below.

In [2]:
# Create an array of values for plotting the sine function
t = np.linspace(0,10*np.pi,100)

# Create a square wave based on the sign of a sine wave.
# This represents an example of the "true" signal we would like to sense.
signal = sign(sin(t))

# Create an array of the same length with small random numbers
noise = np.random.normal(0, np.sqrt(1.0/pow(10,(10/10.0))/2.0), 100)

# Plot the original signal
plt.figure(figsize=(18,4))
plt.subplot(1,3,1)
plt.plot(signal)
plt.title("Original Signal")
plt.ylim(-2,2)

# Plot the noise
plt.subplot(1,3,2)
plt.plot(noise)
plt.title("'Noise' (vector of random numbers)")

# Plot the signal + noise to simulate an actual measurement.
plt.subplot(1,3,3)
plt.plot(signal + noise)
plt.title("Original Signal + Noise = Vector of Actual Measurements\n")
Out[2]:
<matplotlib.text.Text at 0xa6d6dd8>

The simulated noisy output above looks somewhat similar to our original signal considering we can identify the same number of peaks as the original square wave, but is clearly not the ideal square wave we would hope to measure on something like our oscilloscope or Arduino.

In many cases it is impossible to have a noiseless measurement setup, as with our solar cell and projector setup, however we have the advantage of being able to make as many scans of these images as we want; assuming the setup inside our box remains the same, intuitively our measurements should be similar. Is there some way we could come up with an approximation to our original signal by combining many measurements? The plots below show the sum of the original signals and the sum of 100 random noise vectors.

In [24]:
plt.figure(figsize=(18,4))
plt.subplot(1,2,1)
signal = sign(sin(t))
for i in range(100):
    signal += np.sign(np.sin(t))
    if i%10==0:
        plt.plot(signal)
plt.title("Sum of 100 Signals")

plt.subplot(1,2,2)
noise = np.random.normal(0, np.sqrt(1.0/pow(10,(1/10.0))/2.0), 100)
for i in range(100):
    noise += np.random.normal(0, np.sqrt(1.0/pow(10,(1/10.0))/2.0), 100)
    plt.plot(noise)
plt.title("Sum of 100 Noise Vectors")
Out[24]:
<matplotlib.text.Text at 0x14531668>

Note the y-axis of these plots again. How do the amplitudes of these signals compare? Is there some way we could exploit this fact to reconstruct or approximate our original signal?

In [14]:
def reconstruct(noisy_measurements):
    '''Reconstructs a single image from multiple noisy measurements.
    
    Args:
    noisy_measurements (list): a list of noisy signals composed from the sum of a square wave and random noise as shown above
    
    Returns:
    reconstructed_signal (np.array): an approximation of the original signal created from the noisy measurements
    '''   
    reconstructed_signal = np.zeros(len(noisy_measurements[0]))
    #TODO: Use a combination of the noisy measurements to reconstruct an approximation of the original signal.
    
    return reconstructed_signal

Run the cell below to test the reconstruction method you wrote for different values of num_measurements (e.g. 10, 100, 1000, etc). How does your reconstructed output compare to the original signal?

In [23]:
num_measurements = 100
signal = np.sign(np.sin(t))

noisy_measurements = [signal + random.normal(0, np.sqrt(1.0/pow(10,(1/10.0))/2.0), 100) for i in range(num_measurements)]

plt.figure(figsize=(18,4))
plt.subplot(1,2,1)
plt.plot(signal)
plt.title("Original Signal")

plt.subplot(1,2,2)
# Plot 5 random measurements
for meas in [randint(num_measurements) for i in range(5)]:
    plt.plot(noisy_measurements[meas])
plt.title("Example Noisy Measurements")

# Plot the reconstructed measurements
plt.figure(figsize=(18,4))
plt.plot(reconstruct(noisy_measurements))
plt.title("Reconstructed Signal")

Task 3b: Improving Real Images

Run the single pixel scan from Task 1 multiple times, capture_image.py should save all outputs until the program closes as data/sensor_readings[n].npy. Enter the number of measurements you took as indicated and run the code below. How does this compare to your original image?

In [ ]:
#TODO: Enter the number of sensor readings you have
num_measurements = 

# Load sensor readings
noisy_measurements = [np.load("data/sensor_readings%d.npy"%i) for i in range(num_measurements)]

plt.figure(figsize=(3,4))
plt.imshow(np.reshape(load("data/sensor_readings0.npy"), (M,N)), cmap='gray')

plt.figure(figsize=(3,4))
plt.imshow(np.reshape(reconstruct(noisy_measurements), (M,N)), cmap='gray')
plt.savefig("final_image.png")

Congratulations on finishing the labs for the Imaging module!

Contest (Optional)

This section is not required, but if you have additional time after completing the lab and are interested in making a better or more creative image now is your chance! The "Most Realistic" and "Most Creative" submissions from each lab section will be presented in lecture. You're welcome to submit just one, both, or neither.

Most Realistic Image

Submit the image taken using the lab setup as well one from a phone camera for comparison. Running python capture_image.py -h will bring up a list of arguments that you can use to change the resolution, scan rate, etc.

Brief description of submission (parameters/method used, etc)
Resolution/pixel size-
Scan rate-
Solar cell angle (with stand, without, etc)-
Averaging techniques used-

Most Creative Image

This is very open ended and could include anything from imaging 3D objects to alternative visualizations (animations, colored plots, etc) of the raw data.

Brief description of submission (interesting features, etc)