visionsim.emulate package

Subpackages

• visionsim.emulate.dvs package
  • Module contents

Submodules

visionsim.emulate.imu module

visionsim.emulate.imu.imu_integration(acc_pos: Iterable[npt.ArrayLike], vel_ang: Iterable[npt.ArrayLike], dt: float, gravity: npt.NDArray | None = None, pose_init: npt.NDArray | None = None, vel_init: npt.NDArray | None = None) → Iterator[npt.NDArray]

Integrate IMU measurements and estimate trajectory using forward Euler integration [1]_.

Parameters:

• acc_pos (Iterable[npt.ArrayLike]) – Positional acceleration as measured by the IMU. Expects an iterable of [ax, ay, az] vectors in m/s^2 (in camera-coordinates).

• vel_ang (Iterable[npt.ArrayLike]) – Rotational velocity as measured by the gyro. Expects an iterable of [wx, wy, wz] vectors in Rad/s (in camera-coordinates).

• dt (float) – Sampling period in seconds. Typically equal to 1/fps.

• gravity (npt.NDArray, optional) – Gravity vector in m/s^2 (in world-coordinates). Defaults to -9.8 m/s^2 in Z.

• pose_init (npt.NDArray, optional) – Initial pose. Defaults to identity.

• vel_init (npt.NDArray, optional) – Initial positional velocity. Defaults to the zero vector.

Yields:

Iterator[npt.NDArray] – Estimated pose

References

[1]

Forster et al. (2015), “IMU Preintegration on Manifold for Efficient Visual-Inertial Maximum-a-Posteriori Estimation”.

visionsim.emulate.imu.imu_integration_step(pose: npt.NDArray, vel_pos: npt.NDArray, vel_ang: npt.NDArray, acc_pos: npt.NDArray, dt: float) → tuple[npt.NDArray, npt.NDArray]

Computes single Euler integration step [1]_.

While the integration is performed in world coordinates, this helper operates on velocities and accelerations in camera coordinates and perform the coordinate change internally. Poses remain in world coordinates.

Note

Angular acceleration handling is not implemented!

Parameters:

• pose (npt.NDArray) – Current camera pose in world coordinates.

• vel_pos_c (npt.NDArray) – Translational velocity in camera coordinates.

• vel_ang_c (npt.NDArray) – Angular velocity in camera coordinates.

• acc_pos_c (npt.NDArray) – Translational acceleration in camera coordinates.

• dt (float) – Sampling period in seconds.

Returns:

Camera pose at next time step (in world-coords), Translational velocity at next time step (in camera coords)

Return type:

tuple[npt.NDArray, npt.NDArray]

visionsim.emulate.imu.emulate_imu(poses: list[npt.NDArray] | npt.NDArray, *, dt: float = 0.00125, std_acc: float = 0.008, std_gyro: float = 0.0012, std_bias_acc: float = 5.5e-05, std_bias_gyro: float = 2e-05, init_bias_acc: npt.NDArray | None = None, init_bias_gyro: npt.NDArray | None = None, gravity: npt.NDArray | None = None, rng: np.random.Generator | None = None) → Iterator[dict[str, npt.ArrayLike]]

Emulate IMU measurements from a sequence of ground-truth poses.

Follows the Appendix in Crassidis (2006) [2], also see Sec. IV.B. in Leutenegger et al. (2015) [3], and Sec. IV in Forster et al. (2015) [1]_.

The default parameter values are taken from Table I in Leutenegger et al [3].

Parameters:

• poses (list[npt.NDArray] | npt.NDArray) – Sequence of ground-truth poses to emulate IMU from.

• dt (float, optional) – Sampling period in seconds. Defaults to 1/800.

• std_acc (float, optional) – Standard deviation for positional acceleration in m/(s^2 sqrt(Hz)). Defaults to 8e-3.

• std_gyro (float, optional) – Standard deviation for angular velocity in rad/(s sqrt(Hz)). Defaults to 1.2e-3.

• std_bias_acc (float, optional) – Bias for positional acceleration in m/(s^3 sqrt(Hz)). Defaults to 5.5e-5.

• std_bias_gyro (float, optional) – Bias for angular velocity in rad/(s^2 sqrt(Hz)). Defaults to 2e-5.

• init_bias_acc (npt.NDArray, optional) – Initial positional acceleration. Defaults to the zero vector.

• init_bias_gyro (npt.NDArray, optional) – Initial angular velocity. Defaults to the zero vector.

• gravity (npt.NDArray, optional) – Gravity vector in m/s^2 (in world-coordinates). Defaults to -9.8 m/s^2 in Z.

• rng (np.random.Generator, optional) – Random generator instance. Defaults to np.random.default_rng.

Yields:

Iterator[dict[str, npt.ArrayLike]] – Return “acc_reading”, “gyro_reading”, “acc_bias”, “gyro_bias”, and “t”.

References

[2]

Crassidis (2006), “Sigma-Point Kalman Filtering for Integrated GPS and Inertial Navigation”.

[3] (1,2)

Leutenegger et al. (2015), “Keyframe-based visual-inertial odometry using nonlinear optimization”.

visionsim.emulate.rgb module

visionsim.emulate.rgb.emulate_rgb_from_sequence(sequence: npt.ArrayLike, shutter_frac: float = 1.0, readout_std: float = 0.0, fwc: float = 10000.0, adc_bitdepth: int = 12, flux_gain: float = 4096, iso_gain: float = 1.0, mosaic: bool = False, demosaic: Literal['off', 'bilinear', 'MHC04'] = 'off', denoise_sigma: float = 0.0, sharpen_weight: float = 0.0, rng: np.random.Generator | None = None) → npt.NDArray

Emulates a conventional RGB camera [1]_ from a sequence of intensity frames.

For demosaicing details see raw_bayer_to_rgb.

Note

Motion-blur is approximated by averaging consecutive ground truth frames, this can be done more efficiently if optical flow is available. See emulate_rgb_from_flow for more.

Parameters:

• sequence (npt.ArrayLike) – Input sequence of linear-intensity frames, can be a collection of frames, or np/torch array with time as the first dimension.

• shutter_frac (float, optional) – fraction of inter-frame duration the shutter is active. Range [0, 1]

• readout_std (float, optional) – Standard deviation of zero mean Gaussian read noise. Defaults to 0.0.

• fwc (float, optional) – Full well capacity, used for normalization. Defaults to 10000.0.

• adc_bitdepth (int, optional) – Resolution of ADC in bits. Defaults to 12.

• flux_gain (float, optional) – factor to scale the input [0, 1] image _before_ Poisson sampling

• iso_gain (float, optional) – factor to scale the photo-electron reading _after_ Poisson sampling

• mosaic (bool, optional) – implement one array with mosaiced R-/G-/B-sensitive pixels or an innately 3-channel sensor

• demosaic (string, optional) – demosaicing method to use if “mosaic” is set (default “off”)

• denoise_sigma (float, optional) – Gaussian blur kernel sigma (disabled if 0.0)

• sharpen_weight (float, optional) – sharpening weight (disabled if 0.0)

• rng (np.random.Generator, optional) – Optional random number generator. Defaults to none.

Returns:

Quantized linear-intensity RGB patch as floating point array (range [0, 1])

Return type:

npt.NDArray

References

[1]

S. W. Hasinoff, F. Durand, and W. T. Freeman, “Noise-optimal capture for high dynamic range photography,” CVPR 2010.

visionsim.emulate.rgb.emulate_rgb_from_flow()

Not (Yet) Implemented

visionsim.emulate.spc module

visionsim.emulate.spc.emulate_spc(img: npt.NDArray[np.floating], flux_gain: float = 1.0, bitplanes: int = 1, rng: np.random.Generator | None = None) → npt.NDArray[np.integer]

Perform bernoulli sampling on linearized RGB frames to yield binary frames.

Parameters:

• img (npt.NDArray[np.floating]) – Linear intensity image to sample binary frame from.

• flux_gain (float, optional) – scale factor to convert img in [0, 1] range to other flux levels. Defaults to 1.0.

• bitplanes (int, optional) – when bitplanes > 1, this represents a binomial SPAD sensor that sums binary samples internally for each measurement, operating at framerate 1 / bitplanes of a binary SPAD sensor. Defaults to 1.

• rng (np.random.Generator, optional) – Optional random number generator. Defaults to none.

Returns:

binomial-distributed single-photon frame

Return type:

npt.NDArray[np.integer]

visionsim.emulate.spc.spc_avg_to_rgb(mean_binary_patch: torch.Tensor | npt.NDArray, factor: float = 1.0, epsilon: float = 1e-06, quantile: float | None = None) → torch.Tensor | npt.NDArray

Convert average of many single photon binary frames to conventional RGB image.

Invert the process by which binary frames are emulated. The result can be either linear RGB values or sRGB values depending on how the binary frames were constructed.

Assuming each binary patch was created by a Bernoulli process with p=1-exp(-factor*rgb), then the average of binary frames tends to p. We can therefore recover the original rgb values as -log(1-bin)/factor where bin is the average of many binary frames.

Parameters:

• mean_binary_patch (torch.Tensor | npt.NDArray) – Binary avg to convert to rgb, expects a np.ndarray or torch tensor.

• factor (float, optional) – Arbitrary corrective brightness factor. Defaults to 1.0.

• epsilon (float, optional) – Smallest allowed value before taking logarithm, needed for stability. Defaults to 1e-6.

• quantile (float, optional) – If supplied, normalize by this quantile. For instance, if set to 0.9 then the ninety-percent brightest value will be used as the new maximum, with brighter values being clipped. Defaults to None (no normalization/clipping).

Returns:

Conventional intensity image

Return type:

torch.Tensor | npt.NDArray

Module contents