Filters Description

Librealsense implementation includes post-processing filters to enhance the quality of depth data and reduce noise levels. All the filters are implemented in the library core as independent blocks to be used in the customer code

Decimation filter

Effectively reduces the depth scene complexity.
The filter run on kernel sizes [2x2] to [8x8] pixels. For patches sized 2 and 3 the median depth value is selected. For larger kernels, 4-8 pixels, the mean depth is used due to performance considerations.

The image size is scaled down proportionally in both dimensions to preserve the aspect ratio.
Internally, the filter imposes 4-pixel block alignment for the output frame size width and height. E.g. for input size (1280X720) and scale factor 3 the output size calculation is:

[1280,720]/3 -> [426.6666667, 240] -> [428,240]
*The padded rows/columns are zero-filled.

After the resulted frame is produced, the frame intrinsic parameters are recalculated to compensate for the resolution changes.
The filter also provides some hole filling capability, as the filter uses valid (non-zero) pixels only.

Spatial Edge-Preserving filter

*The implementation is based on paper by Eduardo S. L. Gastal and Manuel M. Oliveira.

Key characteristics:

• 1D edge-preserving spatial filter using high-order domain transform.
• Linear-time compute, not affected by the choice of parameters.

The filter performs a series of 1D horizontal and vertical passes or iterations, to enhance the smoothness of the reconstructed data.

Temporal filter

The temporal filter is intended to improve the depth data persistency by manipulating per-pixel values based on previous frames.
The filter performs a single pass on the data, adjusting the depth values while also updating the tracking history. In cases where the pixel data is missing or invalid the filter uses a user-defined persistency mode to decide whether the missing value should be rectified with stored data.
Note that due to its reliance on historic data the filter may introduce visible blurring/smearing artefacts, and therefore is best-suited for static scenes.

Holes Filling filter

The filter implements several methods to rectify missing data in the resulting image.
The filter obtains the four immediate pixel "neighbors" (up, down ,left, right), and selects one of them according to a user-defined rule.

Design and Implementation

Post-processing modules are encapsulated into self-contained processing blocks, that provide for the following key requirements:

1. Synchronous/Asynchronous invocation
2. Internal frames memory/lifetime management

The filters are capable to receive and process frames from different sources, though in the general case it is not practical for the following reasons:

• Performance overhead as each time a new type/source of the frame is recognized, some filter require re-initialization.
• The Temporal filter effectiveness relies on the frames history being preserved. Switching the frame sources invalidates the preserved history and incapacitates the filter.
  Therefore establishing and maintaining a filter pipe per camera source is strongly recommended.

The filters preserve the original data and always generate a new (filtered) frame to pass on. The re-generation of new frames allows to share frame among different consumers (thread) without the risk of the data being overwritten by another user.

All filters support discreet as well as floating point input data formats.
The floating point inputs are utilized by D400 stereo-based Depth cameras that support Disparity data representation.
The discreet version of the filters is primarily used with SR300 camera, but also
can be applied to D400 devices (though not recommended)

Using Filters in application code

The post-processing blocks are designed and built for concatenation into processing pipes.
There are no software-imposed constrains that mandate the order in which the filters shall be applied.
At the same time the recommended scheme used in librealsense tools and demos is elaborated below:
Depth Frame >> Decimation Filter >> Depth2Disparity Transform** -> Spatial Filter >> Temporal Filter >> Disparity2Depth Transform** >> Hole Filling Filter >> Filtered Depth.

** Applicable for stereo-based depth cameras (D4XX).
Note that even though the filters order in the demos is predefined, each filter is controlled individually and can be toggled on/off at run-time.

Demos and tools that have the post-processing code blocks embedded:

1. RealSense-Viewer
2. Depth Quality Tool
3. Post-Processing Demo

Filter initialization and activation flows:
Using C API calls:

// Establishing a frame_queue object for each processing block that will receive the processed frames
rs2_frame_queue* decimated_queue = rs2_create_frame_queue(1, NULL);
rs2_frame_queue* spatial_queue = rs2_create_frame_queue(1, NULL);
...
// Creating processing blocks/ filters
rs2_processing_block* decimation_filter = rs2_create_decimation_filter_block(NULL);
rs2_processing_block* spatial_filter = rs2_create_spatial_filter_block(NULL);
...

// Direct the output of the filters to a dedicated queue
rs2_start_processing_queue(decimation_filter, decimated_queue, NULL);
rs2_start_processing_queue(spatial_filter, spatial_queue, NULL);
...
// Get depth frame from the device
rs2_frame* depth_frame = ...

// Apply decimation filter
rs2_process_frame(decimation_filter, depth_frame, NULL);
rs2_frame* decimated_frame = rs2_wait_for_frame(decimated_queue, 5000, NULL);
// Inject the decimated frame to spatial filter
rs2_process_frame(spatial_filter, decimated_frame, NULL);
// Get the filtered frame
rs2_frame* spatial_filter_frame = rs2_wait_for_frame(spatial_queue, 5000, NULL);
// Use the filtered data
...

// Control filter options
rs2_set_option((rs2_options*)decimation_filter, RS2_OPTION_FILTER_MAGNITUDE, 3, NULL);
rs2_set_option((rs2_options*)spatial_filter, RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.5f, NULL);

Using C++ API:

// Streaming initialization
    rs2::pipeline pipe;
    ...
    // Declare filters
    rs2::decimation_filter dec_filter;
    rs2::spatial_filter spat_filter;

    // Configure filter parameters
    decimation_filter.set_option(RS2_OPTION_FILTER_MAGNITUDE, 3);
    ...
    spatial_filter.set_option(RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.55f);
    ...


    // Main Loop
    while (true) {
       rs2::frameset data = pipe.wait_for_frames();
       rs2::frame depth_frame = data.get_depth_frame();
       ...
       rs2::frame filtered = depth_frame;
       // Note the concatenation of output/input frame to build up a chain
       filtered = dec_filter.process(filtered);
       filtered = spatial_filter.process(filtered);

    }