A transient camera system based on the PMDTechnologies CamBoard nano plus a custom light source. In this document, I will be collecting information to help you build your own capture setup. There is an infinite number of paths to the destination, one being, of course, reading our SIGGRAPH 2013 paper and giving someone with mixed-signal, RF expertise a year or so to build a well-engineered system using the information provided and a sensor datasheet. Our own solution originates from a reasonably good connection to the PMD Technologies company (good enough so they would share datasheets and sell us nonstandard filterless chips; not good enough so they would build custom hardware, firmware, and drivers), along with some experience in digital electronics and endless amounts of tinkering.

Intro

Our work draws inspiration from the "Femto-Photography" direction of work out of the MIT Media Lab (here's an alternative, much nicer project page from their collaborators in Zaragoza). Fascinated by their ultrafast optical impulse responses of real-world scenes, we set out to capture similar data maybe at a reduced resolution, but at a much lower price point and in a more practical way. Our device of choice are time-of-flight imagers that capture a coded version of the desired response. By combining a multi-frequency capture with a numerical reconstruction process (solving a linear inverse problem), we showed that such devices can in fact be used to film light in flight. Our results were presented at SIGGRAPH 2013, both as a Technical Paper and a live demo at Emerging Technologies. At CVPR 2014, we showed how basically the same setup (with just a few collimation lenses added to the laser diodes) can be used to look around corners.

This document aims to provide you with the information needed to recreate our system, or one "like ours". Our system costs less than $1000 in parts to build, that is, if you don't screw up. I'm not formally trained in electronics, but I did start into the project with several years of experience running a little business, plus a supervisor who basically gave me carte blanche for breaking stuff. We must have burned about $6000 until the first fully working setup. The instructions given below are based on my experience after building roughly 12 cameras across 4 hardware revisions (9 of those cameras are actually operational). I've developed a certain confidence in the process - you will see that some of the steps take a bit of a daredevil attitude to perform.

Disclaimer

This is an attempt to document procedures that are to some degree un-documentable. Treat this page as work in constant progress. If in doubt about a certain step or about the big picture, e-mail me (hullin at cs dot uni dash bonn dot de).

High-level requirements and available hardware

We will assume that you are familiar with the basic principles of time-of-flight imaging (CW/periodically modulated, not pulse-based) if you read this document. In order to turn a ToF camera into a transient imager, the following basic requirements have to be met:
  1. Programmable modulation frequency across a wide range: DC to very high frequencies (>100 MHz), and continuously programmable relative phase shift. Or even better: arbitrary modulation patterns.
  2. Access to raw data (four phase images).
  3. Detachable light source with enough power to illuminate your scene.
  4. Nice-to-have: sensor without IR-pass filter, so visible light can be used (reduces SNR in presence of ambient light, though)
Let us have a look at the most widespread ToF systems available, and how they measure up to these requirements:

SystemEvaluation
Microsoft Kinect One /
Kinect for Windows v2
Probably the most powerful hardware platform, but unusable due to severe restrictions.
Modulation: capable of fast modulation and arbitrary patterns.
Firmware, driver and API: no raw data, no choice of frequency
Light source: integrated; probably strong enough (interaction range of several meters) but attached to camera
IR filter: installed
Creative SENZ3D /
SoftKinetic DS325
Modulation: Hardware capable of fast modulation.
Firmware, driver and API: single fixed frequency, but raw data available through SoftKinetic API
Light source: integrated; probably too weak (interaction range <1m)
IR filter: installed
PMD Technologies
CamBoard nano
Hardware: High frequencies supported by sensor but not by light source
Firmware, driver and API: Only single fixed frequency; raw data available (SourceImage)
Light source: too slow (cuts off around 25 MHz); too weak for ranges beyond 1m
IR filter: installed (though they sold us a few rare filterless units)
Mesa SwissRangertbd; probably doesn't modulate fast enough

None of the systems can be used to capture transient images as they come off the shelf; either because the hardware doesn't allow it or because the API is locked down. Solutions range from (1) asking your favorite contact at Microsoft to measure data for you via (2) hacking existing hardware/software to do what you need to (3) developing the whole system from scratch.

Given the current situation regarding off-the-shelf products, solution (3) would be the cleanest if you invest an EE staff for a year or so, full-time. We opted for (2) and implemented it in a modular way. Our camera, based on the PMD CamBoard nano, also contains the signal generator that produces modulation signals for sensor and light source. Our light source is a bank of laser diodes with inputs for RF modulation (BNC/50 Ohm), power and trigger signal.

Recreating our system

Status quo and solution approach

The CamBoard nano is a reference design for the PMD 19k-S3 ToF sensor that runs off the USB port and is reasonably affordable. Unfortunately, it can't do some of the things we need for transient imaging. We need to be able to change the modulation frequency, and we need to set arbitrary relative phases between illumination and reference modulation signal. The nano modulates at a fixed 25MHz (way too slow) and takes phase images in 90° steps (way too coarse). Even if we had access to its FPGA to change the modulation frequency and phase, the LED would still cut off at around 25MHz (above which the modulation contrast goes down).

In short, our solution is

  1. to cut off the nano's own modulation source, and add our own (a DDS circuit based on the Analog Devices AD9958 evaluation board), and
  2. to get rid of the nano's own LED, and add a custom laser source.

Materials needed

You need the following tools:
  • High-quality temperature-controlled soldering iron, fine tip
  • Continuity tester with acoustic signal
  • Handheld electric drill with 1.5mm bit
  • Hot air gun
  • Toaster oven for reflow process (if you populate the circuit boards by yourself)
  • No-clean solder paste
  • No-clean solder wire
  • Cutter knife
  • Quality side cutters
  • Fine-tipped metal tweezers
  • Torx screwdrivers: T7, T8, T9
  • Desoldering utensils and supplies: solder wick, flux, desoldering pump
  • tinned copper wire, approx. 0.25mm (AWG31/30), unclad (only a few cm needed)
Parts:
  • CamBoard nano (sold by PMDTechnologies). we use special development units with no lens and no IR-pass filter, which may no longer be available. The regular version can also be used, but only with IR light (more caution required at high laser powers to prevent eye damage).
  • Printed circuit boards as per layout files provided below. A
  • SMT solder paste stencils for all boards (free when ordering at PCB-Pool)
  • Electronic parts as listed in bill of materials. Everything except iC-HG laser driver and PWB-1-BLB transformers is available at Digi-Key. See iC-Haus website to find a distributor for the iC-HG driver chip, and CoilCraft for the PWB-1-BLB transformers (they ship up to 5 samples for free!).
  • Laser diodes galore. We use the red (650nm) Mitsubishi LPC-826, $6 at RayFoss/China when you take 50 or more. If your CamBoard comes with IR filter, you'll have to get equivalent IR diodes -- e.g., 830nm, 300mW.


Building the electronics

As always, three options:
  1. Have boards made ($400); assemble them yourself. Be warned - this is hard manual labor and requires serious skills (more detailed instructions below).
  2. Have boards made and assembled externally (add $500-$600).
  3. Kindly ask me if my company sells you some pre-assembled boards (be prepared to pay through the nose).

Self-assembling the circuit boards

Order PCBs and solder paste stencil, as in the EAGLE files provided below, from a circuit board manufacturer. I use PCBPool, who are not the cheapest but none the more professional, quite fast, they are prepared for EAGLE files and they offer free steel stencils. You will need 1x cameraboard, 1x laserboard_mother and 4x laserboard_square. Get some more of each to allow for breakage. Specs for cameraboard and laserboard_square are 4 layers, 1.6mm thick, 0.2mm min. drill, 4mil lithography. For laserboard_mother, 2 layers, 1.6mm thick, 0.3mm drill, 6mil lithography. Don't be shocked - a full set of boards may end up costing USD 300-400 - they get drastically cheaper in large volumes. If your board house doesn't accept data in EAGLE format, you WILL need experience in CAM (Gerber) export, since 4-layer designs are rarely used by hobbyists for obvious reasons. Maybe there are tutorials for 4-layer Gerber export from EAGLE somewhere out there, though.

Squeegee solder paste across steel stencil (alignment marks fit nail-shaped wire pins of type Mill-Max 9083-0-00-15-00-00-38-0), and use tweezer to populate SMD parts as indicated in bill of materials. Reflow solder in toaster oven - observe closely and turn off oven when all solder has molten across the board. If you have never done SMD assembly by hand, search the web for tutorials. Some of the parts used by us are fine pitch (0.5mm) -- a steady hand will definitely be required for this.

If you are lucky, your modulation board should now look roughly like above. Give it a thorough round of beeping with a continuity tester, to make sure there are no shorts. As the very last step, get the soldering iron and add the through-hole parts. The male 2x6-pin connectors are attached to the side of the laser boards in a somewhat unorthodox way.

Firmware installation and testing

  1. Test the board for shorts and whatever needs testing before you connect an electronic circuit.
  2. Use ATMEL FLIP to flash firmware through DFU/USB interface. Some button-pressing trickery will be required - see, for instance, this tutorial on the Retrode website.
  3. Connect RF out to oscilloscope (50 Ohm BNC) and check for square wave at 37.5 MHz.

Assembling and adjusting the light source

You need

  1. an amperemeter with accurate and fast response up to 400mA, plus a 2-pin jumper wire to connect it to the laser board.
  2. four jumper bridges.
  3. Small flat screwdriver for potentiometer adjustment.
Now proceed as follows. Current sources need to be adjusted with extreme care. Make sure to follow this protocol and don't reverse the order:
  • Install three laser diodes on a board (check polarity using schematic!), and plug that board into the mainboard. Connect camera to PC and power supply. Verify by oscilloscope that the camera sends modulation signal on RF out (37.5 MHz into 50 Ohm). Connect light source board with BNC and modular cables. Check power LED. Remove modular cable.
  • For each laser diode board {
    
    	Turn all potentiometers fully counterclockwise.
    	Plug laser diode board into slot 1 of mainboard.
    	Bridge 2-pin header on laser diode board via amperemeter.
    	Connect modular cable.
    
    	For each laser diode i {
    
    		Install jumper i.
    		Briefly hit pushbutton TEST TRIG on mainboard.
    		Carefully turn pot iA clockwise, until lasing starts upon pushing TEST TRIG. Check current on ampmeter.
    		Keep increasing current until 50% of laser diode peak current is reached.
    		Remove jumper i.
    
    	} for each laser diode
    
    	With all jumpers removed, keep hitting TEST TRIG and adjusting DC currents via pots 1D, 2D, 3D until each diode is just above lasing threshold.
    
    	Remove modular cable.
    	Remove laser diode board from mainboard.
    	Install all four jumpers. Store laser diode board for later use.
    
    } for each diode board
    
  • In the final step, install all diode boards onto the mainboard. The light source should now be ready for use.

Modifying the CamBoard nano

If the above wasn't tricky enough for you, this is the chapter where you can show what you've got. We start by removing the CamBoard from its aluminum mount using the Torx drivers. Clip off the USB cable using side cutters (before you do, take note of wire colors), and unconnect the LED from the flex board extension. By default, the nano comes with a board lens that is glued on. Carefully soften glue with a heat gun; gently pull off lens. To remove residual glue, repeat heating and scrape off with a wooden toothpick until the sensor is clean on all sides. Yes, this is as little fun as it sounds.


Now take a closer look at the sensor from the "north" side, as seen above. You will find that it is in itself a printed circuit board with the silicon die sitting inside a recessed area and covered by a IR-pass window. (The tiny capacitors probably serve to filter some of the supply voltages.) You can see that most pads of the package are connected to the top through tiny vertical connections, so-called vias. Pin 1, the modulation line, is marked with an arrow.

The drawing above illustrates a cut through sensor package (top) and CamBoard circuit board (bottom) at Pin 1 of the sensor. Red(dish) parts are conductive and carry the modulation signal from the source through the solder pad of pin 1 (3) and a via (2) to the top of the package (1). Points 1-3 are the only ones where the signal can be reached; it is covered by sensor and FPGA elsewhere. Use the cutter knife to carefully scratch off the black varnish layer above point (1), until you see the bare copper. Using the continuity tester, check that point 1 and 3 are electrically connected. Now, with a steady hand, start drilling into the package from the side (2) to interrupt the modulation line. Make sure not to go any deeper than 2mm, and stop immediately once the connection between 1 and 3 is interrupted (no more beeping from the continuity tester).

Mount what is left of the CamBoard on top of the modulation board using board stackers (M2.5 / 5mm stacking height), connecting the USB connector, as well as the GND, HOLD and MOD pins that are marked as such. To make the connection to pin 1 on the sensor, solder a piece of the thin copper wire to the modulation board first, bend as closely as possible to the sensor (avoiding shorts with pins that are located along the way), then hold down with tweezers while attaching with the well-tinned tip of the iron.
Signal  Where to find on CamBoard nano
GNDA. Outer side of large buffer capacitors
HOLD  Sensor Pin 12 (reasonably accessible )
MODSensor Pin 1 (attach to top of package)
(More result / detail pictures to follow)

Files

(firmware and capture tool will be posted once someone starts a serious attempt to build a system)
Schematics and Board Layouts (270KB): [boards.zip]
Bill of Materials (90KB): [bom.xls]
Modulation board firmware (3MB): [firmware.zip]
Capture tool (Visual Studio solution) (15MB): [capturetool.zip]


Citation

If this information turns out useful for your research, I would appreciate if you cite this page:
@misc{Hullin2014,
   Author = {Matthias B. Hullin},
   Year = {2014},
   Note = {http://pulsr.info},
   Title = {Building a Transient Camera}
}

                
(C) Images and text, Matthias Hullin, 2014. No guarantee of completeness, validity, usefulness of information or safety of described actions is implied. Author declines all liability for possible damage resulting from use of information provided. All rights reserved.