Passive Single Photon Camera

What’s a single photon camera?

Single photon cameras (SPCs) are an emerging class of sensors that offer extreme sensitivity and are manufactured at scale using existing CMOS techniques. These sensors can not only detect individual photons but also time them with picosecond accuracy, enabling applications that are not possible with traditional sensors. By sensing individual photons, SPCs provide us with the most fine-grained visual information possible and provides a lot of flexibility to the authors of downstream inference tasks.


Image formation model

While a conventional camera can capture many thousands of photons during a single exposure, a single photon detector can only detect one before it has to be reset:

Due to this, SPCs are fundamentally digital devices. They will output a binary one if one or more photons are received during it’s exposure time and a zero otherwise. For a static scene with a radiant flux (photons/second) of \(\phi\), the number of incident photons \(k\) on a pixel during an exposure time \(\tau\) follows a Poisson distribution given by:

\[P(k) = \frac{(\phi\tau)^ke^{-\phi\tau}}{k!} \,.\]

From this, we can infer that the binary pixel measurement \(B\) will follow a Bernoulli distribution given by [1]:

\[\begin{split}\begin{split} P(B=0) &= P(k=0) = e^{-\phi \tau},\\ P(B=1) &= P(k \ge 1)=1-e^{-\phi \tau}. \end{split}\end{split}\]

However, a SPC can run at extremely fast rates, so in practice while each measurement is extremely noisy, and quantized to a single bits-worth of information, we can acquire many thousands of measurements in the time a conventional camera takes for a single exposure.


Inverting the response function

Now that we understand the basics of a SPC, how can we recover a “normal” image from one?

The key observation is that the previous equation is invertible, so if we have a good photon detection probability estimate \(\widehat P\), we can also estimate \(\phi\). Namely, the maximum likelihood estimator of \(\widehat P\) for a static scene is simply the average of neighboring binary frames [2]:

\[\widehat{P} = \frac{1}{n} \sum_{i=1}^n B_i.\]

And by inverting the image formation model above we can get a good estimate of the scene flux:

\[\widehat{\phi}=-\frac{1}{\tau} \ln \left(1-\widehat{P}\right)\,.\]

Finally, bear in mind that \(\phi\) is the scene flux in photons/second, which is proportional to linear intensity, yet most images are typically tonemapped. For more information about this, consider working through this notebook.