pydistort.project_points#

pydistort.project_points(world_3dpoints: ndarray, rvec: ndarray | None, tvec: ndarray | None, K: ndarray | None, distortion: Distortion | None, transpose: bool = False, dx: bool = False, dp: bool = False, faster_dx: bool = True, **kwargs) → ProjectPointsResult[source]#

Project 3D points to 2D image points using the camera intrinsic and extrinsic matrix and distortion coefficients.

The process to correspond a 3D-world point to a 2D-image point is as follows:

  1. The world_3dpoints (\(X_W\)) are expressed in the camera coordinate system using the rotation and translation matrices to obtain the camera_3dpoints (\(X_C\)).

  2. The camera_3dpoints (\(X_C\)) are normalized by dividing by the third coordinate to obtain the normalized_points (\(x_N\)).

  3. The normalized_points (\(x_N\)) are distorted by the distortion model using the coefficients \(\{\lambda_1, \lambda_2, \lambda_3, \ldots\}\) to obtain the distorted_points (\(x_D\)).

  4. The distorted_points (\(x_D\)) are projected onto the image plane using the intrinsic matrix K to obtain the image_points (\(x_I\)).

\[\begin{split}\begin{align*} X_C &= R \cdot X_W + T \\ x_N &= \frac{X_C}{Z_C} \\ x_D &= \text{distort}(x_N, \lambda_1, \lambda_2, \lambda_3, \ldots) \\ x_I &= K \cdot x_D \end{align*}\end{split}\]

where \(R\) is the rotation matrix, \(T\) is the translation vector et \(K\) is the intrinsic camera matrix. The intrinsic camera matrix is defined as:

\[\begin{split}K = \begin{pmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{pmatrix}\end{split}\]

The given points world 3dpoints are assumed to be in the world coordinate system and expressed in 3D coordinates with shape (…, 3).

Note

... in the shape of the arrays means that the array can have any number of dimensions. Classically, the ... can be replaced by \(N\) which is the number of points.

Note

The rotation matrix and the rotation vector must be given in the convention 0 of py3dframe. (see Artezaru/py3dframe)

To compute the Jacobians of the image points with respect to the input 3D world points and the projection parameters, set the dx and dp parameters to True. The Jacobians are computed using the chain rule of differentiation and are returned in the result object.

Note

For the Jacobian with respect to the input 3D world points, a faster method than the full chain rule can be used by setting the faster_dx parameter to True. The method uses the fact that the Jacobian of the image points with respect to the input 3D world points can be computed directly as the matrix product of the jacobian with respect to the translation and the rotation matrix. This method is only used if the dp parameter is set to True.

Parameters:

  • world_3dpoints (numpy.ndarray) – The 3D points in the world coordinate system. Shape (…, 3).

  • rvec (Optional[numpy.ndarray]) – The rotation vector (or rotation matrix) of the camera. Shape (3,) or (3, 3). If None, the identity rotation is used.

  • tvec (Optional[numpy.ndarray]) – The translation vector of the camera. Shape (3,). If None, the zero translation is used.

  • K (Optional[numpy.ndarray]) – The intrinsic camera matrix (or vector). Shape (3, 3) or (4,). If None, the identity intrinsic matrix is used.

  • distortion (Optional[Distortion]) – The distortion model to be applied to the image points. If None, a zero distortion is applied.

  • transpose (bool, optional) – If True, the input points are assumed to be in the shape (3, …) instead of (…, 3). Default is False. In this case, the output points will be in the shape (2, …) and the jacobians will be in the shape (2, …, 3) and (2, …, 10 + Nparams).

  • dx (bool, optional) – If True, compute the Jacobian of the image points with respect to the input 3D world points with shape (…, 2, 3). If False, the Jacobian is not computed. default is False.

  • dp (bool, optional) – If True, compute the Jacobian of the image points with respect to the projection parameters with shape (…, 2, 10 + Nparams). If False, the Jacobian is not computed. Default is False.

  • faster_dx (bool, optional) – If True, use a faster method to compute the Jacobian of the image points with respect to the input 3D world points. Default is True. This method is only processed if the dp parameter is set to True.

  • **kwargs (dict) – Additional keyword arguments to be passed to the distortion model’s transform method.

Returns:

The result of the projection transformation.

Return type:

ProjectPointsResult

Examples

Create a simple example to project 3D points to 2D image points using the intrinsic and extrinsic parameters of the camera.

import numpy
from pydistort import project_points, Cv2Distortion

# Define the 3D points in the world coordinate system
world_3dpoints = numpy.array([[0.0, 0.0, 5.0],
                                [0.1, -0.1, 5.0],
                                [-0.1, 0.2, 5.0],
                                [0.2, 0.1, 5.0],
                                [-0.2, -0.2, 5.0]]) # shape (5, 3)

# Define the rotation vector and translation vector
rvec = numpy.array([0.01, 0.02, 0.03])  # small rotation
tvec = numpy.array([0.1, -0.1, 0.2])    # small translation

# Define the intrinsic camera matrix
K = numpy.array([[1000.0, 0.0, 320.0],
                [0.0, 1000.0, 240.0],
                [0.0, 0.0, 1.0]])

# Define the distortion model (optional)
distortion = Cv2Distortion([0.1, 0.2, 0.3, 0.4, 0.5])

# Project the 3D points to 2D image points
result = project_points(world_3dpoints, rvec=rvec, tvec=tvec, K=K, distortion=distortion)
print("Projected image points:")
print(result.image_points) # shape (5, 2)

You can also compute the Jacobians of the image points with respect to the input 3D world points and the projection parameters by setting the dx and dp parameters to True.

# Project the 3D points to 2D image points with Jacobians
result = project_points(world_3dpoints, rvec=rvec, tvec=tvec, K=K, distortion=distortion, dx=True, dp=True)

print("Jacobian with respect to 3D points:")
print(result.jacobian_dx) # shape (5, 2, 3)
print("Jacobian with respect to projection parameters:")
print(result.jacobian_dp) # shape (5, 2, 10 + Nparams)
class pydistort.ProjectPointsResult(transformed_points: ndarray, jacobian_dx: ndarray | None = None, jacobian_dp: ndarray | None = None)[source]#

Bases: TransformResult

Subclass of TransformResult to represent the result of the projection transformation from 3D world points to 2D image points.

This class is used to store the result of transforming the world_3dpoints to image_points, and the optional Jacobians.

  • transformed_points: The transformed image points in the camera pixel coordinate system.

  • jacobian_dx: The Jacobian of the image points with respect to the input 3D world points if dx is True. Otherwise None. Shape (…, 2, 3).

  • jacobian_dp: The Jacobian of the image points with respect to the parameters (rotation, translation, distortion, intrinsic) if dp is True. Otherwise None. Shape (…, 2, 10 + Nparams). [rx, ry, rz, tx, ty, tz, fx, fy, cx, cy, d1, d2, …, dNparams] where Nparams is the number of distortion parameters.

Some properties are provided for convenience:

  • image_points: Alias for transformed_points to represent the transformed image points. Shape (…, 2).

  • jacobian_dr: Part of the Jacobian with respect to the rotation vector. Shape (…, 2, 3).

  • jacobian_dt: Part of the Jacobian with respect to the translation vector. Shape (…, 2, 3).

  • jacobian_df: Part of the Jacobian with respect to the focal length parameters. Shape (…, 2, 2).

  • jacobian_dc: Part of the Jacobian with respect to the principal point parameters. Shape (…, 2, 2).

  • jacobian_ddis: Part of the Jacobian with respect to the distortion coefficients. Shape (…, 2, Nparams).

Warning

If transpose is set to True during the transformation, the output points will have shape (output_dim, …) instead of (…, output_dim), same for the Jacobian matrices.

property image_points: ndarray#

Get the transformed image points.

Returns:

The transformed image points in the camera pixel coordinate system. Shape (…, 2).

Return type:

numpy.ndarray

property jacobian_dc: ndarray | None#

Get the jacobian of the image points with respect to the principal point.

Returns:

The Jacobian with respect to principal point parameters (dc). Shape (…, 2, 2).

Return type:

Optional[numpy.ndarray]

property jacobian_ddis: ndarray | None#

Get the jacobian of the image points with respect to the distortion coefficients.

Returns:

The Jacobian with respect to distortion parameters (ddis). Shape (…, 2, Nparams).

Return type:

Optional[numpy.ndarray]

property jacobian_df: ndarray | None#

Get the jacobian of the image points with respect to the focal length.

Returns:

The Jacobian with respect to focal lenght parameters (df). Shape (…, 2, 2).

Return type:

Optional[numpy.ndarray]

property jacobian_dr: ndarray | None#

Get the jacobian of the image points with respect to the rotation vector.

Returns:

The Jacobian with respect to rotation (dr). Shape (…, 2, 3).

Return type:

Optional[numpy.ndarray]

property jacobian_dt: ndarray | None#

Get the jacobian of the image points with respect to the translation vector.

Returns:

The Jacobian with respect to translation (dt). Shape (…, 2, 3).

Return type:

Optional[numpy.ndarray]