Active Single Photon Camera

Single-photon cameras (SPCs) are gaining popularity as promising candidates for high-resolution 3D imaging when combined with an active illumination source [1] [2]. These systems leverage the time-of-flight (ToF) principle to estimate scene depth by measuring the time it takes for photons to travel from an active light source, reflect off objects in the scene, and return to the sensor. Unlike conventional ToF sensors, which accumulate continuous light intensity, active SPCs use Single-Photon Avalanche Diodes (SPADs) to detect and timestamp individual photons with picosecond-level precision.


ToF Imaging Model

In a typical setup, the scene is illuminated using short-duration laser pulses, and the camera records the time-varying distribution of light intensity reflected back from the scene, called the transient distribution.

../../../_images/ToF_Imaging_Model.svg ../../../_images/ToF_Imaging_Model_dark.svg

The figure illustrates how SPCs sample the transient distribution \(\Phi(t)\) by digitizing and recording the returning photon timestamps. \(\Phi_{sig}\) represents the average signal photons incident on the sensor per laser cycle, \(\Phi_{bkg}\) represents the average background photons which includes photons due to ambient light sources and photons detected due to sensor dark counts, and \(A\) is a scaling factor that encapsulates the effect of scene reflectance, photon detection efficiency and distance square fall-off.


Equi-width and Equi-depth Histogrammers

Conventional SPCs use equi-width histogrammers (EWH) to compress the timestamp data into equi-width histograms where each histogram bin represents an equal time interval. Recently, researchers have proposed more efficient active SPCs which use histogrammers that compress the data into equi-depth histograms. Equi-depth histogrammers (EDH) use variable-width bins such that each bin contains (approximately) equal photon counts.

Note

The term “depth” in equi-depth refers to the equal photon counts in each histogram bin and should not be confused with scene distance.

Warning

Current version of VisionSim uses SPCSim library [3] to simulate active SPCs. Future versions of VisionSim will integrate active SPC modules as part of the emulate package.

Currently, emulating SPC data is a three step process:

  1. Generate RGB-D frames using the blender animation functionalities,

  2. Use the TransientGenerator class to generate transients \(\Phi(t)\) from the RGB-D frames for desired sensor properties and illumination condition,

  3. Use the transients generated from SPCSim to emulate active SPC measurements for desired number of laser cycles.

The SPCSim library also contains post-processing module to generate distance estimates from the emulated SPC measurements as well as several active SPCs classes.


Emulating Transient Data

To emulate transient data, we first set up sensor properties and illumination conditions, then generate ground truth transients for each pixel like so:


Nr, Nc = [128, 128]  # SPC sensor resolution
N_tbins = 1024  # Number of discrete time bins for "ideal" transient
tmax = 100  # Laser period in nano seconds
FWHM = 1  # Laser full wave half maximum in terms of bin-width
N_pulses = 1000  # Number of laser cycles to use
alpha_sig = 1.0  # Average signal photons per laser cycle
alpha_bkg = 4.0  # Average background photons per laser cycle
albedo = intensity = rgb[..., 0]

tr_gen = TransientGenerator(Nr=Nr, Nc=Nc, N_tbins=N_tbins, tmax=tmax, FWHM=FWHM)

# Generate the ground-truth transient for each pixel
# given distance, albedo, intensity, and illumination condition
# NOTE: The true distance is in meters and depends on tmax
phi_bar = tr_gen.get_transient(
    depth,
    albedo,
    intensity,
    torch.tensor(alpha_sig),
    torch.tensor(alpha_bkg),

From these ground truth transients we can emulate equi-width histograms and estimate scene distance from these measurements:


N_bins = 64  # Number of EWH SPC bins

# Simulating 64-bin EWH SPC output
spc = BaseEWHSPC(Nr, Nc, N_pulses, device, N_tbins, int(N_bins))
captured_data = spc.capture(phi_bar)
ewh_data = captured_data["ewh"]

# Estimate distance from EWH SPC measurements
ewh_postproc = PostProcEWH(Nr, Nc, N_tbins, tmax, device)

We can do the same with equi-depth histograms like so:


N_bins = 16  # Number of EDH SPC bins

# Simulating 16-Bin EDH SPC output
spc = BaseEDHSPC(Nr, Nc, N_pulses, device, N_tbins, int(N_bins))
captured_data = spc.capture(phi_bar)
oedh_data = captured_data["oedh"]

# Estimate depth from equi-depth histogram
edh_postproc = PostProcEDH(Nr, Nc, N_tbins, tmax, device)

Plotting the results, we can see that equi-depth histograms can estimate more accurate scene distances with lower number of histogram bins than EWH SPCs.

../../../_images/Emulate_active_spcs.svg ../../../_images/Emulate_active_spcs-dark.svg

See also

Refer to this tutorial to design SPCs with custom histogrammers.