I've got a robot that needs to see. The end goal is a ROS2-controlled arm picking parts off a conveyor — but parts on a conveyor are the hard version, so I'm starting where every sensible project starts: getting the cameras to reliably show me a picture. Three Intel RealSense D435i depth cameras, on a Linux laptop, feeding a browser. This is the write-up of getting that far, gotchas and all.
A quick note on the kit. These are the D435i — the "i" being the variant with an onboard IMU. (There's also a "D435f" with an IR-pass filter, and the model numbers are close enough that you'll want to check exactly what's in the box. When will manufacturers learn?) Three of them, all on USB3.
Installing the SDK without losing a weekend
I'd braced myself for the usual depth-camera install pain — kernel patches, DKMS modules, building librealsense from source. None of it turned out to be necessary, which I'm still slightly suspicious about.
The trick is that Intel's pyrealsense2 wheel on PyPI (2.58.2) bundles the whole SDK and talks to the camera over a userspace USB backend (RSUSB). It bypasses the kernel entirely, so the fact that this machine is on kernel 6.x — well past what the DKMS packages officially support — simply doesn't matter. No source build, no module, no faff.
Two things to know before you copy this:
- It needs Python 3.12, not 3.13. Wheels exist for both, but ROS2 Jazzy (the 24.04 release) is a 3.12 world, and I want the option of a ROS2 bridge later, so 3.12 it is.
uvwill fetch a standalone 3.12 for you regardless of your system Python. - The wheel does not bundle libusb. This one cost me a confused five minutes.
import pyrealsense2blows up withImportError: libusb-1.0.so.0: cannot open shared object file, which looks alarming but just means you need the system library:
sudo apt install libusb-1.0-0
uv venv --python 3.12 .venv
uv pip install pyrealsense2 numpy opencv-python-headless pyyaml
That's genuinely the whole install. Plus the udev rules from the librealsense repo if you want non-root USB access (you do).
Which camera is which?
The first time I ran an enumeration with one camera plugged in, it found nothing. Cue head-scratching. lsusb showed an "Intel Corp." device, so surely that was it?
It wasn't. A RealSense reports under USB vendor ID 8086 (a nice touch, that). The thing I was looking at was 8087 — Intel's Bluetooth radio. The camera simply wasn't plugged in yet. Once it was, all three appeared as 8086:0b3a.
Which brings up the multi-camera wrinkle: they all share the same product ID. You cannot ask for "the camera" — you have to open each one by serial number. So the toolkit enumerates them and you pick by serial:
244622071799 Intel RealSense D435I fw 5.13.0.55 USB 3.2
244222071624 Intel RealSense D435I fw 5.13.0.55 USB 3.2
244222076366 Intel RealSense D435I fw 5.13.0.55 USB 3.2
A live dashboard, and a 15-gigabit lie
I wanted the feeds headless and viewable from anywhere on the network, so the streaming is plain MJPEG over HTTP from Python's standard library — no web framework. A browser renders multipart/x-mixed-replace natively in an <img> tag, so "live video in a web page" is, pleasingly, just an image that keeps updating. The dashboard lists each camera with a checkbox; tick one and it opens and starts streaming, untick it and it stops. Only the cameras you select consume bandwidth, which matters when three depth cameras are sharing USB controllers.
I added a bandwidth readout against each feed, partly to keep an eye on WiFi usage. And it immediately told me something was very wrong: 15 Gbps from a single 848×480 stream. Over WiFi. Sure.
The meter wasn't broken — it had honestly exposed a bug. My streaming loop was re-sending the latest JPEG in a tight busy-loop rather than waiting for a genuinely new frame, because the frame store returned immediately whenever any frame existed. So it was blasting the same picture thousands of times a second. The fix was to give each frame a sequence number and have the stream wait for the sequence to actually advance. Egress dropped to a believable ~10–30 Mbps and the CPU stopped sweating.
Interestingly, this is exactly the sort of bug that hides forever if you don't measure the thing. I only caught it because I'd bothered to count the bytes.
Depth at 90, colour at 60
A neat detail of the D435 family: the depth sensor will do 90 fps at 848×480, but the RGB camera caps at 60. My first cut ran them locked together at one rate — which throttles depth to 60 for no good reason. So now depth and colour run as independent streams, each at its own framerate, shown as separate feeds with their own settings and bandwidth. Verified on the bench: ~80 fps depth alongside ~60 fps colour, from one camera, simultaneously. (The depth feed eats more bandwidth than colour, oddly — the false-colour depth map is all high-frequency noise and JPEG hates it.)
While I was at it I pulled the actual capabilities off the device rather than trusting the datasheet from memory. A couple of things surprised me:
| Mode | Max FPS | HFOV × VFOV |
|---|---|---|
| Depth 848×480 | 90 | 89° × 58° |
| Depth 1280×720 | 30 | 89° × 58° |
| Depth 848×100 | 300 | 89° × 13° |
| Depth 256×144 | 300 | 22° × 13° |
The two 300 fps modes are geometric opposites, which I hadn't appreciated. The 848×100 mode keeps the full 89° width but slices the vertical down to a 13° ribbon — a wide, thin "trip-wire". The 256×144 mode is the reverse: a narrow 22° window, but a 1:1 centre crop of the full sensor, so it keeps the long focal length. Same framerate, completely different problems they solve. Neither is any use for looking at a whole tray of parts, but worth knowing they exist.
Depth precision is the other axis people forget. The stereo baseline came back as 49.93 mm, and the depth step works out at roughly 7.7 mm per disparity at half a metre on the 1280×720 mode, degrading with the square of distance. So at 2 m you're four times coarser than at 1 m. For anything where the depth number matters, get close.
The IMU has opinions
Since these are the "i" variant, I had a poke at the IMU. The accelerometer does 100/200/400 Hz and the gyro 200/400 Hz, and a quick capture confirmed roughly 9.81 m/s² pointing the right way, so it's alive and honest. Two gotchas though:
- Ask for a rate it doesn't support (I tried accel at 250 Hz) and you get a cryptic
RuntimeError: Couldn't resolve requests. It's just an unsupported mode. get_motion_intrinsics()returns an identity matrix and all-zero variances on this firmware — i.e. no factory calibration is exposed. If you want a trustworthy gravity vector you'll have to calibrate the bias yourself. Good to know before you build anything on top of it.
Usefully, the IMU and depth timestamps share the same clock domain, so fusing them later won't need any clock-wrangling.
Testing when you haven't got the hardware
I won't always have three cameras on the desk, so the test suite is split in two. The detection and plumbing logic never imports the SDK at module load, which means a good chunk of it is unit-testable on any machine with no camera and no libusb — synthetic depth frames in, known hole centres out. Then there's a separate live smoke test that you run with the cameras attached: it enumerates them, grabs frames, spins up the dashboard, ticks a camera, and checks the whole web UI works end to end. Belt and braces.
Where this leaves things
The honest summary: I can reliably enumerate three D435i cameras by serial, stream depth and colour from any of them into a browser at full framerate with live bandwidth figures, and run a first-pass depth-based hole detector. The whole thing installs from a pip wheel and one apt package, and it's now in its own repo.
The detector is the weak link, and I know exactly why. It currently assumes the part is flat and square-on to the camera — fine for a bench demo, useless for the real job, where the cameras will sit at an angle to parts moving on a conveyor and I'll need to fuse the colour image with the point cloud across several views. That's the next big piece, and it needs proper plane-fitting and camera calibration rather than the cheerful approximation I've got now. There's also the small matter of finding out how many of these cameras I can actually run at once before a USB controller cries — I've built the tool to measure it but haven't run the sweep in anger yet.
More on both of those once I've got further. Ta ta for now, and I hope you found this helpful.
Built against pyrealsense2 2.58.2, OpenCV 4.13, Python 3.12 on Ubuntu 24.04. Camera firmware 5.13.0.55. SDK and udev rules from the librealsense project (now maintained at github.com/realsenseai/librealsense).
No comments:
Post a Comment