Documentation¶
While this documentation aims to go beyond a simple listing of parameters and instead attempts to explain some of the principles behind the functions, please see the section “Usage” for more details and usage examples including code and flow field visualisations.
Using the Flow Class¶
This section documents the custom flow class and all its class methods. It is the recommended way of using
oflibnumpy
and makes the full range of functionality available to the user.
Flow Constructors and Operators¶
-
class
oflibnumpy.
Flow
(flow_vectors: numpy.ndarray, ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None)¶ -
__init__
(flow_vectors: numpy.ndarray, ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None) → Flow¶ Flow object constructor. For a more detailed explanation of the arguments, see the class attributes
vecs
,ref
, andmask
.- Parameters
flow_vectors – Numpy array of shape \((H, W, 2)\) containing the flow vector in OpenCV convention:
flow_vectors[..., 0]
are the horizontal,flow_vectors[..., 1]
are the vertical vector components, defined as positive when pointing to the right / down.ref – Flow reference, either
t
for “target”, ors
for “source”. Defaults tot
mask – Numpy array of shape \((H, W)\) containing a boolean mask indicating where the flow vectors are valid. Defaults to
True
everywhere.
-
property
vecs
¶ Flow vectors, a numpy array of shape \((H, W, 2)\). The last dimension contains the flow vectors. These are in the order horizontal component first, vertical component second (OpenCV convention). They are defined as positive towards the right and the bottom, meaning the origin is located in the left upper corner of the \(H \times W\) flow field area.
- Returns
Flow vectors as numpy array of shape \((H, W, 2)\) and type
float32
-
property
ref
¶ Flow reference, a string: either
s
for “source” ort
for “target”. This determines whether the regular grid of shape \((H, W)\) associated with the flow vectors should be understood as the source of the vectors (which then point to any other position), or the target of the vectors (whose start point can then be any other position). The flow referencet
is the default, meaning the regular grid refers to the coordinates the pixels whose motion is being recorded by the vectors end up at.Applying a flow with reference
s
is known as “forward” warping, while referencet
corresponds to what is termed “backward” or “reverse” warping.Caution
The
apply()
method for warping an image is significantly faster with a flow int
reference. The reason is that this requires interpolating unstructured points from a regular grid, while references
requires interpolating a regular grid from unstructured points. The former uses the fast OpenCVremap()
function, the latter is much more operationally complex and relies on the SciPygriddata()
function.Caution
The
track()
method for tracking points is significantly faster with a flow ins
reference, again due to not requiring a call to SciPy’sgriddata()
function.Tip
If some algorithm
get_flow()
is set up to calculate a flow field with referencet
(ors
) as inflow_one_ref = get_flow(img1, img2)
, it is very simple to obtain the flow in references
(ort
) instead: simply call the algorithm with the images in the reversed order, and multiply the resulting flow vectors by -1:flow_other_ref = -1 * get_flow(img2, img1)
- Returns
Flow reference, as string of value
t
ors
-
property
mask
¶ Flow mask as a numpy array of shape \((H, W)\) and type
bool
. This array indicates, for each flow vector, whether it is considered “valid”. As an example, this allows for masking of the flow based on object segmentations. It is also necessary to keep track of which flow vectors are valid when different flow fields are combined, as those operations often lead to undefined (partially or fully unknown) points in the given \(H \times W\) area where the flow vectors are either completely unknown, or will not have valid values.- Returns
Flow mask as numpy array of shape \((H, W)\) and type
bool
-
property
shape
¶ Shape (resolution) \((H, W)\) of the flow, corresponding to the first two dimensions of the flow vector array of shape \((H, W, 2)\)
- Returns
Tuple of the shape (resolution) \((H, W)\) of the flow object
-
classmethod
zero
(shape: Union[list, tuple], ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None) → Flow¶ Flow object constructor, zero everywhere
- Parameters
shape – List or tuple of the shape \((H, W)\) of the flow field
ref – Flow reference, string of value
t
(“target”) ors
(“source”). Defaults tot
mask – Numpy array of shape \((H, W)\) and type
bool
indicating where the flow vectors are valid. Defaults toTrue
everywhere
- Returns
Flow object
-
classmethod
from_matrix
(matrix: numpy.ndarray, shape: Union[list, tuple], ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None) → Flow¶ Flow object constructor, based on transformation matrix input
- Parameters
matrix – Transformation matrix to be turned into a flow field, as numpy array of shape \((3, 3)\)
shape – List or tuple of the shape \((H, W)\) of the flow field
ref – Flow reference, string of value
t
(“target”) ors
(“source”). Defaults tot
mask – Numpy array of shape \((H, W)\) and type
bool
indicating where the flow vectors are valid. Defaults toTrue
everywhere
- Returns
Flow object
-
classmethod
from_transforms
(transform_list: list, shape: Union[list, tuple], ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None) → Flow¶ Flow object constructor, based on list of transforms
- Parameters
transform_list –
List of transforms to be turned into a flow field, where each transform is expressed as a list of [
transform name
,transform value 1
, … ,transform value n
]. Supported options:Transform
translation
, with valueshorizontal shift in px
,vertical shift in px
Transform
rotation
, with valueshorizontal centre in px
,vertical centre in px
,angle in degrees, counter-clockwise
Transform
scaling
, with valueshorizontal centre in px
,vertical centre in px
,scaling fraction
shape – List or tuple of the shape \((H, W)\) of the flow field
ref – Flow reference, string of value
t
(“target”) ors
(“source”). Defaults tot
mask – Numpy array of shape \((H, W)\) and type
bool
indicating where the flow vectors are valid. Defaults toTrue
everywhere
- Returns
Flow object
-
classmethod
from_kitti
(path: str, load_valid: Optional[bool] = None) → Flow¶ Loads the flow field contained in KITTI
uint16
png images files, optionally including the valid pixels. Follows the official instructions on how to read the provided .png files on the KITTI optical flow dataset website.
-
classmethod
from_sintel
(path: str, inv_path: Optional[str] = None) → Flow¶ Loads the flow field contained in Sintel .flo byte files, including the invalid pixels if required. Follows the official instructions provided alongside the .flo data on the Sintel optical flow dataset website.
- Parameters
path – String containing the path to the Sintel flow data (.flo byte file, little Endian)
inv_path – String containing the path to the Sintel invalid pixel data (.png, black and white)
- Returns
A flow object corresponding to the Sintel flow data, with flow reference
ref
s
-
copy
() → Flow¶ Copy a flow object by constructing a new one with the same vectors
vecs
, referenceref
, and maskmask
- Returns
Copy of the flow object
-
__str__
() → str¶ Enhanced string representation of the flow object, containing the flow reference
ref
and shapeshape
- Returns
String representation
-
__getitem__
(item: Union[int, list, slice]) → Flow¶ Mimics
__getitem__
of a numpy array, returning a new flow object cut accordinglyWill throw an error if
mask.__getitem__(item)
orvecs.__getitem__(item)
(corresponding tomask[item]
andvecs[item]
) throw an error. Also throws an error if slicedvecs
ormask
are not suitable to construct a new flow object with, e.g. if the number of dimensions is too low.- Parameters
item – Slice used to select a part of the flow
- Returns
New flow object cut as a corresponding numpy array would be cut
-
__add__
(other: Union[numpy.ndarray, Flow]) → Flow¶ Adds a flow object or a numpy array to a flow object
Caution
This is not equal to applying the two flows sequentially. For that, use
combine_with()
withmode
set to3
.Caution
If this method is used to add two flow objects, there is no check on whether they have the same reference
ref
.- Parameters
other – Flow object or numpy array corresponding to the addend. Adding a flow object will adjust the mask of the resulting flow object to correspond to the logical union of the augend / addend masks
- Returns
New flow object corresponding to the sum
-
__sub__
(other: Union[numpy.ndarray, Flow]) → Flow¶ Subtracts a flow object or a numpy array from a flow object
Caution
This is not equal to subtracting the effects of applying flow fields to an image. For that, use
combine_with()
withmode
set to1
or2
.Caution
If this method is used to subtract two flow objects, there is no check on whether they have the same reference
ref
.- Parameters
other – Flow object or numpy array corresponding to the subtrahend. Subtracting a flow object will adjust the mask of the resulting flow object to correspond to the logical union of the minuend / subtrahend masks
- Returns
New flow object corresponding to the difference
-
__mul__
(other: Union[float, int, list, numpy.ndarray]) → Flow¶ Multiplies a flow object with a single number, a list, or a numpy array
- Parameters
other –
Multiplier, options:
can be converted to a float
a list of shape \((2)\)
an array of the same shape \((H, W)\) as the flow object
an array of the same shape \((H, W, 2)\) as the flow vectors
- Returns
New flow object corresponding to the product
-
__truediv__
(other: Union[float, int, list, numpy.ndarray]) → Flow¶ Divides a flow object by a single number, a list, or a numpy array
- Parameters
other –
Divisor, options:
can be converted to a float
a list of shape \((2)\)
an array of the same shape \((H, W)\) as the flow object
an array of the same shape \((H, W, 2)\) as the flow vectors
- Returns
New flow object corresponding to the quotient
-
__pow__
(other: Union[float, int, bool, list, numpy.ndarray]) → Flow¶ Exponentiates a flow object by a single number, a list, or a numpy array
- Parameters
other –
Exponent, options:
can be converted to a float
a list of shape \((2)\)
an array of the same shape \((H, W)\) as the flow object
an array of the same shape \((H, W, 2)\) as the flow vectors
- Returns
New flow object corresponding to the power
-
Manipulating the Flow¶
-
Flow.
resize
(scale: Union[float, int, list, tuple]) → Flow¶ Resize a flow field, scaling the flow vectors values
vecs
accordingly.- Parameters
scale –
Scale used for resizing, options:
Integer or float of value
scaling
applied both vertically and horizontallyList or tuple of shape \((2)\) with values
[vertical scaling, horizontal scaling]
- Returns
New flow object scaled as desired
-
Flow.
pad
(padding: Optional[Union[list, tuple]] = None, mode: Optional[str] = None) → Flow¶ Pad the flow with the given padding. Padded flow
vecs
values are either constant (set to0
), correspond to the edge values of the flow field, or are a symmetric mirroring of the existing flow values. Paddedmask
values are set toFalse
.- Parameters
padding – List or tuple of shape \((4)\) with padding values
[top, bot, left, right]
mode – String of the numpy padding mode for the flow vectors, with options
constant
(fill value0
),edge
,symmetric
(see documentation fornumpy.pad()
). Defaults toconstant
- Returns
New flow object with the padded flow field
-
Flow.
invert
(ref: Optional[str] = None) → Flow¶ Inverting a flow: img1 – f –> img2 becomes img1 <– f – img2. The smaller the input flow, the closer the inverse is to simply multiplying the flow by -1.
- Parameters
ref – Desired reference of the output field, defaults to the reference of original flow field
- Returns
New flow object, inverse of the original
-
Flow.
switch_ref
(mode: Optional[str] = None) → Flow¶ Switch the reference
ref
betweens
(“source”) andt
(“target”)Caution
Do not use
mode=invalid
if avoidable: it does not actually change any flow values, and the resulting flow object, when applied to an image, will no longer yield the correct result.- Parameters
mode –
Mode used for switching, available options:
invalid
: just the flow reference attribute is switched without any flow values being changed. This is functionally equivalent to simply assigningflow.ref = 't'
for a “source” flow orflow.ref = 's'
for a “target” flowvalid
(default): the flow field is switched to the other coordinate reference, with flow vectors recalculated accordingly
- Returns
New flow object with switched coordinate reference
-
Flow.
combine_with
(flow: Flow, mode: int, thresholded: Optional[bool] = None) → Flow¶ Function that returns the result of the combination of two flow objects of the same shape
shape
and referenceref
Tip
All of the flow field combinations in this function rely on some combination of the
apply()
,invert()
, andcombine_with()
methods, and can be very slow (several seconds) due to callingscipy.interpolate.griddata()
multiple times. The table below aids decision-making with regards to which reference a flow field should be provided in to obtain the fastest result.¶ mode
ref = 's'
ref = 't'
1
1
3
2
1
1
3
0
0
All formulas used in this function have been derived from first principles. The base formula is \(flow_1 ⊕ flow_2 = flow_3\), where \(⊕\) is a non-commutative flow composition operation. This can be visualised with the start / end points of the flows as follows:
S = Start point S1 = S3 ─────── f3 ────────────┐ E = End point │ │ f = flow f1 v └───> E1 = S2 ── f2 ──> E2 = E3
The main difficulty in combining flow fields is that it would be incorrect to simply add up or subtract flow vectors at one location in the flow field area \(H \times W\). This appears to work given e.g. a translation to the right, and a translation downwards: the result will be the linear combination of the two vectors, or a translation towards the bottom right. However, looking more closely, it becomes evident that this approach isn’t actually correct: A pixel that has been moved from S1 to E1 by the first flow field f1 is then moved from that location by the flow vector of the flow field f2 that corresponds to the new pixel location E1, not the original location S1. If the flow vectors are the same everywhere in the field, the difference will not be noticeable. However, if the flow vectors of f2 vary throughout the field, such as with a rotation around some point, it will!
In this case (corresponding to calling
f1.combine_with(f2, mode=3)
), and if the flow referenceref
iss
(“source”), the solution is to first apply the inverse of f1 to f2, essentially linking up each location E1 back to S1, and then to add up the flow vectors. Analogous observations apply for the other permutations of flow combinations and referenceref
values.Note
This is consistent with the observation that two translations are commutative in their application - the order does not matter, and the vectors can simply be added up at every pixel location -, while a translation followed by a rotation is not the same as a rotation followed by a translation: adding up vectors at each pixel cannot be the correct solution as there wouldn’t be a difference based on the order of vector addition.
- Parameters
flow – Flow object to combine with
mode –
Integer determining how the input flows are combined, where the number corresponds to the position in the formula \(flow_1 ⊕ flow_2 = flow_3\):
Mode
1
: self corresponds to \(flow_2\), flow corresponds to \(flow_3\), the result will be \(flow_1\)Mode
2
: self corresponds to \(flow_1\), flow corresponds to \(flow_3\), the result will be \(flow_2\)Mode
3
: self corresponds to \(flow_1\), flow corresponds to \(flow_2\), the result will be \(flow_3\)
thresholded – Boolean determining whether flows are thresholded during an internal call to
is_zero()
, defaults toFalse
- Returns
New flow object
Applying the Flow¶
-
Flow.
apply
(target: Union[numpy.ndarray, Flow], target_mask: Optional[numpy.ndarray] = None, return_valid_area: Optional[bool] = None, consider_mask: Optional[bool] = None, padding: Optional[Union[list, tuple]] = None, cut: Optional[bool] = None) → Union[numpy.ndarray, Flow, Tuple[Union[numpy.ndarray, Flow], numpy.ndarray]]¶ Apply the flow to a target, which can be a numpy array or a Flow object itself. If the flow shape \((H_{flow}, W_{flow})\) is smaller than the target shape \((H_{target}, W_{target})\), a list of padding values needs to be passed to localise the flow in the larger \(H_{target} \times W_{target}\) area.
The valid image area that can optionally be returned is
True
where the image values in the function output:have been affected by flow vectors. If the flow has a reference
ref
value oft
(“target”), this is alwaysTrue
as the target image by default has a corresponding flow vector at each pixel location in \(H \times W\). If the flow has a referenceref
value ofs
(“source”), this is onlyTrue
for some parts of the image: some target image pixel locations in \(H \times W\) would only be reachable by flow vectors originating outside of the source image area, which is impossible by definitionhave been affected by flow vectors that were themselves valid, as determined by the flow mask
Caution
The parameter consider_mask relates to whether the invalid flow vectors in a flow field with reference
s
are removed before application (default behaviour) or not. Doing so results in a smoother flow field, but can cause artefacts to arise where the outline of the area returned byvalid_target()
is not a convex hull. For a more detailed explanation with an illustrative example, see the section “Applying a Flow” in the usage documentation.- Parameters
target – Numpy array of shape \((H, W)\) or \((H, W, C)\), or a flow object of shape \((H, W)\) to which the flow should be applied, where \(H\) and \(W\) are equal or larger than the corresponding dimensions of the flow itself
target_mask – Optional numpy array of shape \((H, W)\) that indicates which part of the target is valid (only relevant if target is a numpy array). Only impacts the valid area returned when
return_valid_area = True
. Defaults toTrue
everywherereturn_valid_area – Boolean determining whether the valid image area is returned (only if the target is a numpy array), defaults to
False
. The valid image area is returned as a boolean numpy array of shape \((H, W)\).consider_mask – Boolean determining whether the flow vectors are masked before application (only relevant for flows with reference
ref = 's'
). Results in smoother outputs, but more artefacts. Defaults toTrue
padding – List or tuple of shape \((4)\) with padding values
[top, bottom, left, right]
. Required if the flow and the target don’t have the same shape. Defaults toNone
, which means no padding neededcut – Boolean determining whether the warped target is returned cut from \((H_{target}, W_{target})\) to \((H_{flow}, W_{flow})\), in the case that the shapes are not the same. Defaults to
True
- Returns
The warped target of the same shape \((C, H, W)\) and type as the input (rounded if necessary), and optionally the valid area of the flow as a boolean array of shape \((H, W)\)
-
Flow.
track
(pts: numpy.ndarray, int_out: Optional[bool] = None, get_valid_status: Optional[bool] = None, s_exact_mode: Optional[bool] = None) → numpy.ndarray¶ Warp input points with the flow field, returning the warped point coordinates as integers if required
Tip
Calling
track()
on a flow field with referenceref
s
(“source”) is significantly faster (as long as s_exact_mode is not set toTrue
), as this does not require a call toscipy.interpolate.griddata()
.- Parameters
pts – Numpy array of shape \((N, 2)\) containing the point coordinates.
pts[:, 0]
corresponds to the vertical coordinate,pts[:, 1]
to the horizontal coordinateint_out – Boolean determining whether output points are returned as rounded integers, defaults to
False
get_valid_status – Boolean determining whether an array of shape \((N, 2)\) is returned, which contains the status of each point. This corresponds to applying
valid_source()
to the point positions, and returnsTrue
for the points that 1) tracked by valid flow vectors, and 2) end up inside the flow area of \(H \times W\). Defaults toFalse
s_exact_mode – Boolean determining whether the necessary flow interpolation will be done using
scipy.interpolate.griddata()
, if the flow has the referenceref
value ofs
(“source”). Defaults toFalse
, which means a less exact, but around 2 orders of magnitude faster bilinear interpolation method will be used. This is recommended for normal point tracking applications.
- Returns
Numpy array of warped (‘tracked’) points, and optionally a numpy array of the point tracking status
Evaluating the Flow¶
-
Flow.
is_zero
(thresholded: Optional[bool] = None, masked: Optional[bool] = None) → bool¶ Check whether all flow vectors (where
mask
isTrue
) are zero. Optionally, a threshold flow magnitude value of1e-3
is used. This can be useful to filter out motions that are equal to very small fractions of a pixel, which might just be a computational artefact to begin with.- Parameters
thresholded – Boolean determining whether the flow is thresholded, defaults to
True
masked – Boolean determining whether the flow is masked with
mask
, defaults toTrue
- Returns
True
if the flow field is zero everywhere, otherwiseFalse
-
Flow.
matrix
(dof: Optional[int] = None, method: Optional[str] = None, masked: Optional[bool] = None) → numpy.ndarray¶ Fit a transformation matrix to the flow field using OpenCV functions
- Parameters
dof –
Integer describing the degrees of freedom in the transformation matrix to be fitted, defaults to
8
. Options are:4
: Partial affine transform with rotation, translation, scaling6
: Affine transform with rotation, translation, scaling, shearing8
: Projective transform, i.e estimation of a homography
method –
String describing the method used to fit the transformations matrix by OpenCV, defaults to
ransac
. Options are:lms
: Least mean squaresransac
: RANSAC-based robust methodlmeds
: Least-Median robust method
masked – Boolean determining whether the flow mask is used to ignore flow locations where the mask
mask
isFalse
. Defaults toTrue
- Returns
Numpy array of shape \((3, 3)\) containing the transformation matrix
-
Flow.
valid_target
(consider_mask: Optional[bool] = None) → numpy.ndarray¶ Find the valid area in the target domain
Given a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The valid area is then a boolean numpy array of shape \((H, W)\) that is
True
wherever the value in the target img stems from warping a value from the source, andFalse
where no valid information is known.Pixels that are
False
will often be black (or ‘empty’) in the warped target image - but not necessarily, due to warping artefacts etc. The valid area also allows a distinction between pixels that are black due to no actual information being available at this position (validityFalse
), and pixels that are black due to black pixel values having been warped to that (valid) location by the flow.- Parameters
consider_mask – Boolean determining whether the flow vectors are masked before application (only relevant for flows with reference
ref = 's'
, analogous toapply()
). Results in smoother outputs, but more artefacts. Defaults toTrue
- Returns
Boolean numpy array of the same shape \((H, W)\) as the flow
-
Flow.
valid_source
(consider_mask: Optional[bool] = None) → numpy.ndarray¶ Finds the area in the source domain that will end up being valid in the target domain (see
valid_target()
) after warpingGiven a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The source area is then a boolean numpy array of shape \((H, W)\) that is
True
wherever the value in the source will end up somewhere inside the valid target area, andFalse
where the value in the source will either be warped outside of the target image, or not be warped at all due to a lack of valid flow vectors connecting to this position.- Parameters
consider_mask – Boolean determining whether the flow vectors are masked before application (only relevant for flows with reference
ref = 't'
as their inverse flow will be applied, using the references
; analogous toapply()
). Results in smoother outputs, but more artefacts. Defaults toTrue
- Returns
Boolean numpy array of the same shape \((H, W)\) as the flow
-
Flow.
get_padding
() → list¶ Determine necessary padding from the flow field:
When the flow reference
ref
has the valuet
(“target”), this corresponds to the padding needed in a source image which ensures that every flow vector invecs
marked as valid by the maskmask
will find a value in the source domain to warp towards the target domain. I.e. any invalid locations in the area \(H \times W\) of the target domain (seevalid_target()
) are purely due to no valid flow vector being available to pull a source value to this target location, rather than no source value being available in the first place.When the flow reference
ref
has the values
(“source”), this corresponds to the padding needed for the flow itself, so that applying it to a source image will result in no input image information being lost in the warped output, i.e each input image pixel will come to lie inside the padded area.
- Returns
A list of shape \((4)\) with the values
[top, bottom, left, right]
Visualising the Flow¶
-
Flow.
visualise
(mode: str, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None, range_max: Optional[float] = None) → numpy.ndarray¶ Visualises the flow as an rgb / bgr / hsv image, optionally showing the outline of the flow mask
mask
as a black line, and the invalid areas greyed out.- Parameters
mode – Output mode, options:
rgb
,bgr
,hsv
show_mask – Boolean determining whether the flow mask is visualised, defaults to
False
show_mask_borders – Boolean determining whether the flow mask border is visualised, defaults to
False
range_max – Maximum vector magnitude expected, corresponding to the HSV maximum Value of 255 when scaling the flow magnitudes. Defaults to the 99th percentile of the flow field magnitudes
- Returns
Numpy array of shape \((H, W, 3)\) containing the flow visualisation
-
Flow.
visualise_arrows
(grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None, colour: Optional[tuple] = None, thickness: Optional[int] = None) → numpy.ndarray¶ Visualises the flow as arrowed lines, optionally showing the outline of the flow mask
mask
as a black line, and the invalid areas greyed out.- Parameters
grid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to
20
img – Numpy array with the background image to use (in BGR mode), defaults to white
scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)
show_mask – Boolean determining whether the flow mask is visualised, defaults to
False
show_mask_borders – Boolean determining whether the flow mask border is visualised, defaults to
False
colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in
visualise()
thickness – Integer of the flow arrow thickness, larger than zero. Defaults to
1
- Returns
Numpy array of shape \((H, W, 3)\) containing the flow visualisation, in
bgr
colour space
-
Flow.
show
(wait: Optional[int] = None, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None)¶ Shows the flow in an OpenCV window using
visualise()
- Parameters
wait – Integer determining how long to show the flow for, in milliseconds. Defaults to
0
, which means it will be shown until the window is closed, or the process is terminatedshow_mask – Boolean determining whether the flow mask is visualised, defaults to
False
show_mask_borders – Boolean determining whether flow mask border is visualised, defaults to
False
-
Flow.
show_arrows
(wait: Optional[int] = None, grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None, colour: Optional[tuple] = None)¶ Shows the flow in an OpenCV window using
visualise_arrows()
- Parameters
wait – Integer determining how long to show the flow for, in milliseconds. Defaults to
0
, which means it will be shown until the window is closed, or the process is terminatedgrid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to
20
img – Numpy array with the background image to use (in BGR colour space), defaults to black
scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)
show_mask – Boolean determining whether the flow mask is visualised, defaults to
False
show_mask_borders – Boolean determining whether the flow mask border is visualised, defaults to
False
colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in
visualise()
-
oflibnumpy.
visualise_definition
(mode: str, shape: Optional[Union[list, tuple]] = None, insert_text: Optional[bool] = None) → numpy.ndarray¶ Return an image that shows the definition of the flow visualisation.
- Parameters
mode – Desired output colour space:
rgb
,bgr
, orhsv
shape – List or tuple of shape \((2)\) containing the desired image shape as values
(H, W)
. Defaults to (601, 601) - do not change if you leave insert_text asTrue
as otherwise the text will appear in the wrong locationinsert_text – Boolean determining whether explanatory text is put on the image (using
cv2.putText()
), defaults toTrue
- Returns
Numpy array of shape \((H, W, 3)\) and type
uint8
showing the colour definition of the flow visualisation
Using NumPy Arrays¶
This section contains functions that take NumPy arrays as inputs, instead of making use of the custom flow class. On the one hand, this avoids having to define flow objects. On the other hand, it requires keeping track of flow attributes manually, and it does not avail itself of the full scope of functionality oflibnumpy can offer: most importantly, flow masks are not considered or tracked.
Flow Loading¶
-
oflibnumpy.
from_matrix
(matrix: numpy.ndarray, shape: Union[list, tuple], ref: str) → numpy.ndarray¶ Flow field array calculated from a transformation matrix
- Parameters
matrix – Transformation matrix to be turned into a flow field, as numpy array of shape \((3, 3)\)
shape – List or tuple of the shape \((H, W)\) of the flow field
ref – Flow reference, string of value
t
(“target”) ors
(“source”)
- Returns
Flow field, as numpy array
-
oflibnumpy.
from_transforms
(transform_list: list, shape: Union[list, tuple], ref: str) → numpy.ndarray¶ Flow field array calculated from a list of transforms
- Parameters
transform_list –
List of transforms to be turned into a flow field, where each transform is expressed as a list of [
transform name
,transform value 1
, … ,transform value n
]. Supported options:Transform
translation
, with valueshorizontal shift in px
,vertical shift in px
Transform
rotation
, with valueshorizontal centre in px
,vertical centre in px
,angle in degrees, counter-clockwise
Transform
scaling
, with valueshorizontal centre in px
,vertical centre in px
,scaling fraction
shape – List or tuple of the shape \((H, W)\) of the flow field
ref – Flow reference, string of value
t
(“target”) ors
(“source”)
- Returns
Flow field as a numpy array
-
oflibnumpy.
load_kitti
(path: str) → Union[List[numpy.ndarray], numpy.ndarray]¶ Loads the flow field contained in KITTI
uint16
png images files, including the valid pixels. Follows the official instructions on how to read the provided .png files on the KITTI optical flow dataset website.- Parameters
path – String containing the path to the KITTI flow data (
uint16
, .png file)- Returns
A numpy array with the KITTI flow data (including valid pixels)
-
oflibnumpy.
load_sintel
(path: str) → numpy.ndarray¶ Loads the flow field contained in Sintel .flo byte files. Follows the official instructions provided alongside the .flo data on the Sintel optical flow dataset website.
- Parameters
path – String containing the path to the Sintel flow data (.flo byte file, little Endian)
- Returns
A numpy array containing the Sintel flow data
-
oflibnumpy.
load_sintel_mask
(path: str) → numpy.ndarray¶ Loads the invalid pixels contained in Sintel .png mask files, as a boolean mask marking valid pixels with
True
. Follows the official instructions provided alongside the .flo data on the Sintel optical flow dataset website.- Parameters
path – String containing the path to the Sintel invalid pixel data (.png, black and white)
- Returns
A boolean numpy array containing the Sintel valid pixels (mask) data
Flow Manipulation¶
-
oflibnumpy.
resize_flow
(flow: numpy.ndarray, scale: Union[float, int, list, tuple]) → numpy.ndarray¶ Resize a flow field array, scaling the flow vectors values accordingly
- Parameters
flow – Numpy array containing the flow vectors to be resized, shape \((H, W, 2)\)
scale –
Scale used for resizing, options:
Integer or float of value
scaling
applied both vertically and horizontallyList or tuple of shape \((2)\) with values
[vertical scaling, horizontal scaling]
- Returns
Scaled flow field as a numpy array
-
oflibnumpy.
invert_flow
(flow: numpy.ndarray, input_ref: str, output_ref: Optional[str] = None) → numpy.ndarray¶ Inverting a flow: img1 – f –> img2 becomes img1 <– f – img2. The smaller the input flow, the closer the inverse is to simply multiplying the flow by -1.
- Parameters
flow – The flow field as a numpy array of shape \((H, W, 2)\)
input_ref – Reference of the input flow field, either
s
ort
output_ref – Desired reference of the output field, either
s
ort
. Defaults toinput_ref
- Returns
Flow field as a numpy array of shape \((H, W, 2)\)
-
oflibnumpy.
switch_flow_ref
(flow: numpy.ndarray, input_ref: str) → numpy.ndarray¶ Recalculate flow vectors to correspond to a switched flow reference (see Flow reference
ref
)- Parameters
flow – The flow field as a numpy array of shape \((H, W, 2)\)
input_ref – The reference of the input flow field, either
s
ort
- Returns
Flow field as a numpy array of shape \((H, W, 2)\)
-
oflibnumpy.
combine_flows
(input_1: Union[Flow, numpy.ndarray], input_2: Union[Flow, numpy.ndarray], mode: int, ref: Optional[str] = None, thresholded: Optional[bool] = None) → Union[Flow, numpy.ndarray]¶ Returns the result of the combination of two flow fields of the same shape and reference
Tip
All of the flow field combinations in this function rely on some combination of the
apply()
,invert()
, andcombine_with()
methods, and can be very slow (several seconds) due to callingscipy.interpolate.griddata()
multiple times. The table below aids decision-making with regards to which reference a flow field should be provided in to obtain the fastest result.¶ mode
ref = 's'
ref = 't'
1
1
3
2
1
1
3
0
0
All formulas used in this function have been derived from first principles. The base formula is \(flow_1 ⊕ flow_2 = flow_3\), where \(⊕\) is a non-commutative flow composition operation. This can be visualised with the start / end points of the flows as follows:
S = Start point S1 = S3 ─────── f3 ────────────┐ E = End point │ │ f = flow f1 v └───> E1 = S2 ── f2 ──> E2 = E3
The main difficulty in combining flow fields is that it would be incorrect to simply add up or subtract flow vectors at one location in the flow field area \(H \times W\). This appears to work given e.g. a translation to the right, and a translation downwards: the result will be the linear combination of the two vectors, or a translation towards the bottom right. However, looking more closely, it becomes evident that this approach isn’t actually correct: A pixel that has been moved from S1 to E1 by the first flow field f1 is then moved from that location by the flow vector of the flow field f2 that corresponds to the new pixel location E1, not the original location S1. If the flow vectors are the same everywhere in the field, the difference will not be noticeable. However, if the flow vectors of f2 vary throughout the field, such as with a rotation around some point, it will!
In this case (corresponding to calling
combine_flows(f1, f2, mode=3)
), and if the flow reference iss
(“source”), the solution is to first apply the inverse of f1 to f2, essentially linking up each location E1 back to S1, and then to add up the flow vectors. Analogous observations apply for the other permutations of flow combinations and references.Note
This is consistent with the observation that two translations are commutative in their application - the order does not matter, and the vectors can simply be added up at every pixel location -, while a translation followed by a rotation is not the same as a rotation followed by a translation: adding up vectors at each pixel cannot be the correct solution as there wouldn’t be a difference based on the order of vector addition.
- Parameters
input_1 – First input flow as a numpy array. Can also take a flow object, but this will be deprecated soon
input_2 – Second input flow, same type as
input_1
mode –
Integer determining how the input flows are combined, where the number corresponds to the position in the formula \(flow_1 ⊕ flow_2 = flow_3\):
Mode
1
: input_1 corresponds to \(flow_2\), input_2 corresponds to \(flow_3\), the result will be \(flow_1\)Mode
2
: input_1 corresponds to \(flow_1\), input_2 corresponds to \(flow_3\), the result will be \(flow_2\)Mode
3
: input_1 corresponds to \(flow_1\), input_2 corresponds to \(flow_2\), the result will be \(flow_3\)
ref – The reference of the input flow fields, either
s
ort
thresholded – Boolean determining whether flows are thresholded during an internal call to
is_zero()
, defaults toFalse
- Returns
Flow object if inputs are flow objects (deprecated in future, avoid), Numpy array as standard
Flow Application¶
-
oflibnumpy.
apply_flow
(flow: numpy.ndarray, target: numpy.ndarray, ref: str, mask: Optional[numpy.ndarray] = None) → numpy.ndarray¶ Uses a given flow to warp a target. The flow reference, if not given, is assumed to be
t
. Optionally, a mask can be passed which (only for flows ins
reference) masks undesired (e.g. undefined or invalid) flow vectors.- Parameters
flow – Numpy array containing the flow vectors in cv2 convention (1st channel hor, 2nd channel ver), with shape \((H, W, 2)\)
target – Numpy array containing the content to be warped, with shape \((H, W)\) or \((H, W, C)\)
ref – Reference of the flow,
t
ors
mask – Boolean numpy array containing the flow mask, with shape \((H, W)\). Only relevant for
s
flows. Defaults toTrue
everywhere
- Returns
Numpy array of the same shape \((H, W)\) as the target, with the content warped by the flow
-
oflibnumpy.
track_pts
(flow: numpy.ndarray, ref: str, pts: numpy.ndarray, int_out: Optional[bool] = None, s_exact_mode: Optional[bool] = None) → numpy.ndarray¶ Warp input points with the flow field, returning the warped point coordinates as integers if required
Tip
Calling
track_pts()
on a flow field with references
(“source”) is significantly faster (as long as s_exact_mode is not set toTrue
), as this does not require a call toscipy.interpolate.griddata()
.- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Flow field reference, either
s
ort
pts – Numpy array of shape \((N, 2)\) containing the point coordinates.
pts[:, 0]
corresponds to the vertical coordinate,pts[:, 1]
to the horizontal coordinateint_out – Boolean determining whether output points are returned as rounded integers, defaults to
False
s_exact_mode – Boolean determining whether the necessary flow interpolation will be done using
scipy.interpolate.griddata()
, if the flow has the referenceref
value ofs
(“source”). Defaults toFalse
, which means a less exact, but around 2 orders of magnitude faster bilinear interpolation method will be used. This is recommended for normal point tracking applications.
- Returns
Numpy array of warped (‘tracked’) points, and optionally a numpy array of the point tracking status
Flow Evaluation¶
-
oflibnumpy.
is_zero_flow
(flow: numpy.ndarray, thresholded: Optional[bool] = None) → bool¶ Check whether all flow vectors are zero. Optionally, a threshold flow magnitude value of
1e-3
is used. This can be useful to filter out motions that are equal to very small fractions of a pixel, which might just be a computational artefact to begin with.- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
thresholded – Boolean determining whether the flow is thresholded, defaults to
True
- Returns
True
if the flow field is zero everywhere, otherwiseFalse
-
oflibnumpy.
get_flow_matrix
(flow: numpy.ndarray, ref: str, dof: Optional[int] = None, method: Optional[str] = None) → numpy.ndarray¶ Fit a transformation matrix to the flow field using OpenCV functions
- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Reference of the flow field,
s
ort
dof –
Integer describing the degrees of freedom in the transformation matrix to be fitted, defaults to
8
. Options are:4
: Partial affine transform with rotation, translation, scaling6
: Affine transform with rotation, translation, scaling, shearing8
: Projective transform, i.e estimation of a homography
method –
String describing the method used to fit the transformations matrix by OpenCV, defaults to
ransac
. Options are:lms
: Least mean squaresransac
: RANSAC-based robust methodlmeds
: Least-Median robust method
- Returns
Numpy array of shape \((3, 3)\) containing the transformation matrix
-
oflibnumpy.
valid_target
(flow: numpy.ndarray, ref: str) → numpy.ndarray¶ Find the valid area in the target domain
Given a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The valid area is then a boolean numpy array of shape \((H, W)\) that is
True
wherever the value in the target img stems from warping a value from the source, andFalse
where no valid information is known.Pixels that are
False
will often be black (or ‘empty’) in the warped target image - but not necessarily, due to warping artefacts etc. The valid area also allows a distinction between pixels that are black due to no actual information being available at this position (validityFalse
), and pixels that are black due to black pixel values having been warped to that (valid) location by the flow.- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Reference of the flow field,
s
ort
- Returns
Boolean numpy array of the same shape \((H, W)\) as the flow
-
oflibnumpy.
valid_source
(flow: numpy.ndarray, ref: str) → numpy.ndarray¶ Finds the area in the source domain that will end up being valid in the target domain (see
valid_target()
) after warpingGiven a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The source area is then a boolean numpy array of shape \((H, W)\) that is
True
wherever the value in the source will end up somewhere inside the valid target area, andFalse
where the value in the source will either be warped outside of the target image, or not be warped at all due to a lack of valid flow vectors connecting to this position.- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Reference of the flow field,
s
ort
- Returns
Boolean numpy array of the same shape \((H, W)\) as the flow
-
oflibnumpy.
get_flow_padding
(flow: numpy.ndarray, ref: str) → list¶ Determine necessary padding from the flow field:
When the flow reference is
t
(“target”), this corresponds to the padding needed in a source image which ensures that every flow vector will find a value in the source domain to warp towards the target domain. I.e. any invalid locations in the area \(H \times W\) of the target domain (seevalid_target()
) are purely due to no valid flow vector being available to pull a source value to this target location, rather than no source value being available in the first place.When the flow reference is
s
(“source”), this corresponds to the padding needed for the flow itself, so that applying it to a source image will result in no input image information being lost in the warped output, i.e each input image pixel will come to lie inside the padded area.
- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Reference of the flow field,
s
ort
- Returns
A list of shape \((4)\) with the values
[top, bottom, left, right]
Flow Visualisation¶
-
oflibnumpy.
visualise_flow
(flow: numpy.ndarray, mode: str, range_max: Optional[float] = None) → numpy.ndarray¶ Visualises the flow as an rgb / bgr / hsv image
- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
mode – Output mode, options:
rgb
,bgr
,hsv
range_max – Maximum vector magnitude expected, corresponding to the HSV maximum Value of 255 when scaling the flow magnitudes. Defaults to the 99th percentile of the flow field magnitudes
- Returns
Numpy array of shape \((H, W, 3)\) containing the flow visualisation
-
oflibnumpy.
visualise_flow_arrows
(flow: numpy.ndarray, ref: str, grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, colour: Optional[tuple] = None, thickness: Optional[int] = None) → numpy.ndarray¶ Visualises the flow as arrowed lines
- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Reference of the flow field,
s
ort
grid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to
20
img – Numpy array with the background image to use (in BGR mode), defaults to white
scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)
colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in
visualise()
thickness – Integer of the flow arrow thickness, larger than zero. Defaults to
1
- Returns
Numpy array of shape \((H, W, 3)\) containing the flow visualisation, in
bgr
colour space
-
oflibnumpy.
show_flow
(flow: numpy.ndarray, wait: Optional[int] = None)¶ Shows the flow in an OpenCV window using
visualise()
- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
wait – Integer determining how long to show the flow for, in milliseconds. Defaults to
0
, which means it will be shown until the window is closed, or the process is terminated
-
oflibnumpy.
show_flow_arrows
(flow: numpy.ndarray, ref: str, wait: Optional[int] = None, grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, colour: Optional[tuple] = None)¶ Shows the flow in an OpenCV window using
visualise_arrows()
- Parameters
flow – Flow field as a numpy array of shape \((H, W, 2)\)
ref – Reference of the flow field,
s
ort
wait – Integer determining how long to show the flow for, in milliseconds. Defaults to
0
, which means it will be shown until the window is closed, or the process is terminatedgrid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to
20
img – Numpy array with the background image to use (in BGR colour space), defaults to black
scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)
colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in
visualise()