`isaac_ros_ess`

Source code on GitHub.

Quickstart

Set Up Development Environment

Set up your development environment by following the instructions in getting started.

Clone isaac_ros_common under ${ISAAC_ROS_WS}/src.

cd ${ISAAC_ROS_WS}/src && \
   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common.git

(Optional) Install dependencies for any sensors you want to use by following the sensor-specific guides.

Warning

We strongly recommend installing all sensor dependencies before starting any quickstarts. Some sensor dependencies require restarting the Isaac ROS Dev container during installation, which will interrupt the quickstart process.

Download Quickstart Assets

Download quickstart data from NGC:

Make sure required libraries are installed.

sudo apt-get install -y curl tar

Then, run these commands to download the asset from NGC.

NGC_ORG="nvidia"
NGC_TEAM="isaac"
NGC_RESOURCE="isaac_ros_assets"
NGC_VERSION="isaac_ros_ess"
NGC_FILENAME="quickstart.tar.gz"

REQ_URL="https://api.ngc.nvidia.com/v2/resources/$NGC_ORG/$NGC_TEAM/$NGC_RESOURCE/versions/$NGC_VERSION/files/$NGC_FILENAME"

mkdir -p ${ISAAC_ROS_WS}/isaac_ros_assets/${NGC_VERSION} && \
    curl -LO --request GET "${REQ_URL}" && \
    tar -xf ${NGC_FILENAME} -C ${ISAAC_ROS_WS}/isaac_ros_assets/${NGC_VERSION} && \
    rm ${NGC_FILENAME}

Build `isaac_ros_ess`

Launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Install the prebuilt Debian package:

sudo apt-get install -y ros-humble-isaac-ros-ess && \
   sudo apt-get install -y ros-humble-isaac-ros-ess-models-install

Download and install the pre-trained ESS model files:

ros2 run isaac_ros_ess_models_install install_ess_models.sh --eula

Clone this repository under ${ISAAC_ROS_WS}/src:

cd ${ISAAC_ROS_WS}/src && \
   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_stereo_depth.git

Launch the Docker container using the run_dev.sh script:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

Use rosdep to install the package’s dependencies:

rosdep install --from-paths ${ISAAC_ROS_WS}/src/isaac_ros_dnn_stereo_depth --ignore-src -y

Download and install the pre-trained ESS model files:

cd ${ISAAC_ROS_WS} && \
   colcon build --packages-up-to isaac_ros_common && \
   source install/setup.bash && \
   ./src/isaac_ros_dnn_stereo_depth/isaac_ros_ess_models_install/asset_scripts/install_ess_models.sh --eula

Build the package from source:

cd ${ISAAC_ROS_WS} && \
   colcon build --packages-up-to isaac_ros_ess

Source the ROS workspace:

Note

Make sure to repeat this step in every terminal created inside the Docker container.

Since this package was built from source, the enclosing workspace must be sourced for ROS to be able to find the package’s contents.
```
source install/setup.bash
```

Note

Limitations on x86_64: ESS plugins only run with GPU with sm80 and above. This limits the GPU devices on x86_64 to devices with compute_80 and above. For CUDA compute capability details, please refer to cuda-gpus.

Run Launch File

Continuing inside the Docker container, install the following dependencies:
```
sudo apt-get install -y ros-humble-isaac-ros-examples
```

Run the following launch file to spin up a demo using quickstart rosbag:

To run ESS at a threshold of 0.0 (fully dense output):

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=ess_disparity \
   engine_file_path:=${ISAAC_ROS_WS:?}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/ess.engine \
   threshold:=0.0

Open a second terminal and attach to the container:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

In the second terminal, play the ESS sample rosbag downloaded in the quickstart assets:

ros2 bag play -l ${ISAAC_ROS_WS}/isaac_ros_assets/isaac_ros_ess/rosbags/ess_rosbag \
   --remap /left/camera_info:=/left/camera_info_rect /right/camera_info:=/right/camera_info_rect

Note

This tutorial requires a compatible RealSense camera from the list of available cameras

Ensure that you have already set up your RealSense camera using the RealSense setup tutorial. If you have not, please set up the sensor and then restart this quickstart from the beginning.

Continuing inside the Docker container, install the following dependencies:

sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-realsense

Complete steps to set up the ESS model as described in the quickstart.

Continuing inside the Docker container, run the following launch file to spin up a demo using RealSense stereo camera:

To run at a threshold of 0.35:

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_stereo_rect,ess_disparity \
   engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/ess.engine \
   threshold:=0.35 realsense_config_file:=$(ros2 pkg prefix isaac_ros_ess --share)/config/realsense.yaml

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=realsense_stereo_rect,ess_disparity \
   engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/light_ess.engine \
   threshold:=0.35 realsense_config_file:=$(ros2 pkg prefix isaac_ros_ess --share)/config/realsense.yaml

Note

This tutorial requires an Argus-compatible stereo camera from the list of available cameras.

If you have an Argus-compatible camera, you can also use the launch file provided in this tutorial to start a fully NITROS-accelerated stereo depth graph.

Ensure that you have already set up your Hawk camera using the Hawk setup tutorial. If you have not, please set up the sensor and then restart this quickstart from the beginning.

Continuing inside the Docker container, install the following dependencies:

sudo apt-get install -y ros-humble-isaac-ros-examples ros-humble-isaac-ros-argus-camera

Complete steps to set up the ESS model as described in the quickstart.

Continuing inside the Docker container, run the following launch file to spin up a demo using Argus stereo camera:

To run at a threshold of 0.35:

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_stereo,rectify_stereo,ess_disparity \
   engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/ess.engine \
   threshold:=0.35

ros2 launch isaac_ros_examples isaac_ros_examples.launch.py launch_fragments:=argus_stereo,rectify_stereo,ess_disparity \
   engine_file_path:=${ISAAC_ROS_WS}/isaac_ros_assets/models/dnn_stereo_disparity/dnn_stereo_disparity_v4.0.0/light_ess.engine \
   threshold:=0.35

Visualize Output

Open a terminal and attach to the container:

cd ${ISAAC_ROS_WS}/src/isaac_ros_common && \
   ./scripts/run_dev.sh

In the terminal, visualize and validate the disparity output using the visualizer script:

ros2 run isaac_ros_ess isaac_ros_ess_visualizer.py
With threshold set to 0.35, the example result is:

With threshold set to 0.0, the example result is:

Try More Examples

To continue your exploration, check out the following suggested examples:

Troubleshooting

Package not found while launching the visualizer script

Symptom

$ ros2 run isaac_ros_ess isaac_ros_ess_visualizer.py
Package 'isaac_ros_ess' not found

Solution

Use the colcon build --packages-up-to isaac_ros_ess command to build isaac_ros_ess; do not use the --symlink-install option. Run source install/setup.bash after the build.

Problem reserving CacheChange in reader

Symptom

When using a ROS bag as input, isaac_ros_ess throws an error if the input topics are published too fast:

[component_container-1] 2022-06-24 09:04:43.584 [RTPS_MSG_IN Error] (ID:281473268431152) Problem reserving CacheChange in reader: 01.0f.cd.10.ab.f2.65.b6.01.00.00.00|0.0.20.4 -> Function processDataMsg

Solution

Make sure that the ROS bag has a reasonable size and publish rate.

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, please check here.

API

Overview

The isaac_ros_ess package offers functionality to generate a stereo disparity map from stereo images using a trained ESS model. Given a pair of stereo input images, the package generates a continuous disparity image for the left input image.

Usage

ros2 launch isaac_ros_ess isaac_ros_ess.launch.py engine_file_path:=<your ESS engine plan absolute path>

ESSDisparityNode

ROS Parameters

ROS Parameter	Type	Default	Description
`engine_file_path`	`string`	N/A - Required	The absolute path to the ESS engine file.
`threshold`	`double`	0.35	Threshold value ranges between 0.0 and 1.0 for filtering disparity with confidence. Pixels with confidence less than threshold will be marked as invalid in the disparity output. Value 0.0 means a fully dense disparity output.
`image_type`	`string`	`"RGB_U8"`.	The input image encoding type. Supports `"RGB_U8"` and `"BGR_U8"`. Note that if this parameter is not specified and there is an upstream Isaac ROS NITROS node, the type will be decided automatically.

ROS Topics Subscribed

ROS Topic	Interface	Description
`left/image_rect`	sensor_msgs/Image	The left image of a stereo pair.
`right/image_rect`	sensor_msgs/Image	The right image of a stereo pair.
`left/camera_info`	sensor_msgs/CameraInfo	The left camera model.
`right/camera_info`	sensor_msgs/CameraInfo	The right camera model.

Note

The images on input topics (left/image_rect and right/image_rect) should be a color image either in rgb8 or bgr8 format.

ROS Topics Published

ROS Topic	Interface	Description
`disparity`	stereo_msgs/DisparityImage	The continuous stereo disparity estimation.

Input Restrictions

The input left and right images must have the same dimension and resolution, and the resolution must be no larger than 1920x1200.
Each input pair (left/image_rect, right/image_rect, left/camera_info and right/camera_info) should have the same timestamp; otherwise, the synchronizing module inside the ESS Disparity Node will drop the input with smaller timestamps.

Output Interpretations

The isaac_ros_ess package outputs a disparity image with dimension same as the ESS model output dimension.

ESS Model

Output Dimension

ess.etlt

960 x 576

light_ess.etlt

480 x 288

The input images are rescaled to the ESS model input dimension before inferencing. There are two outputs from the ESS model with the same dimension: disparity output and confidence output. The disparity is filtered with confidence using a pre-configured threshold. Pixels with confidence less than the threshold is replaced with -1.0 as invalid before the inference result is published. For fully dense disparity output without confidence thresholding, set the threshold to 0.0.
The left and right CameraInfo are used to composite a stereo_msgs/DisparityImage. If you only care about the disparity image, and don’t need the baseline and focal length information, you can pass dummy camera messages.

ESS Model	Output Dimension
`ess.etlt`	960 x 576
`light_ess.etlt`	480 x 288

isaac_ros_ess

Quickstart

Set Up Development Environment

Download Quickstart Assets

Build isaac_ros_ess

Run Launch File

Visualize Output

Try More Examples

Troubleshooting

Package not found while launching the visualizer script

Symptom

Solution

Problem reserving CacheChange in reader

Symptom

Solution

Isaac ROS Troubleshooting

API

Overview

Usage

ESSDisparityNode

ROS Parameters

ROS Topics Subscribed

ROS Topics Published

Input Restrictions

Output Interpretations

`isaac_ros_ess`

Build `isaac_ros_ess`