Ben Wang’s Blog

Phased Array Microphone

2023-02-26T00:00:00+00:00

A 192-channel phased array microphone, with FPGA data acquisition and beamforming/visualization on the GPU. Phased arrays allow for applications not possible with traditional directional microphones, as the directionality can be changed instantly, after the recording is made, or even be focused at hundreds of thousands of points simultaneously in real time.

All designs are open source:

Block diagram

Glamor shot

Hardware

To create a phased array microphone, a large number of microphones needs to be placed in an arrangement with a wide distribution of spacing. For a linear array, exponential spacing between microphones is found to be optimal for broadband signals. To create a 2d array, these symmetrical linear arrays (“arms”) are be placed radially, which allows the central (“hub”) board to be compact. The total cost for the array is approximately $700.

Arms

The length of each arm is dictated by the limits of PCB manufacturing and assembly. These boards were made at JLCPCB, where the maximum length for manufacturing and assembly of 4 layer PCBs was 570mm.

The microphones chosen were the cheapest digital output MEMS microphone (because there are a lot of them!), which were about $0.5. At this bottom of the barrel price range, there is little differentiation in the performance characteristics between different microphones. Most have decent performance up to 10khz and unspecified matching of phase delay and volume.

These microphones output data using pulse density modulation (PDM), which provides a one bit output at a frequency significantly higher than the audible range (up to 4 MHz), with the high sampling rate compensating for quantization noise. These microphones also support latching the data either on the rising or falling edge of the clock (DDR), which allows two microphones to be multiplexed on a single wire, reducing the amount of connections required.

Each arm contains 8 microphones sharing 4 output lines, as well as an output buffer on the clock input line. This ensures the rise times are reasonable, even with hundreds of microphones sharing the same clock signal.

For some reason (likely the low rigidity of the panel and some suboptimal solder paste stencil patterns combined with the LGA microphone footprints), the yields on the arm PCBs are not very good, with only 50% of them working out of the box. The most common fault was the clock line being shorted to either 3V3 or ground, which unfortunately requires trial and error of removing microphones from the PCB until the short is resolved. Next time some series resistors on the clock line would speed this process up a lot, and improving the panelization and paste stencil would likely improve yields so extensive rework isn’t required.

Even with rework, there are still some microphones which produce bogus data. These are just masked out in the code, as there are enough working ones to make up for it (and it’s too much work to remove a bunch of arms to do more rework…)

Hub

An FPGA is used to collect all the data, due to the large number of low latency IOs available combined with the ability to communicate using high speed interfaces (e.g. Gigabit Ethernet). Specifically, the Colorlight i5 card is used, as it has enough IOs, is cheap and readily available, and has two integrated ethernet PHYs (only one is used for this project). The card is originally designed as an ethernet interface for LED panels, but has been fully reverse engineered. About 100 GPIOs are broken out over the DDR2 connector, which is much easier to fan out than the BGA of the original FPGA.

Other than the FPGA, the hub contains some simple power management circuitry, and connectors for the arm boards as well as an Ethernet connector with integrated magnetics.

Mechanical Design

The arms are attached with M3 screws to the hub using PCB mounted standoffs/nuts , which conveniently can be assembled with SMD processes. The connections from each arm to the hub is made with 8 pin, 2mm pitch connectors.

The original mechanical design consists of slots on the arm PCBs which interlock with circumferential structural PCBs, however the low torsional rigidity of the arms means the whole structure deformed too easily.

The final mechanical design consists of pieces of laser cut 1/4th inch MDF around the outer edge of the array, with each arm attached to the MDF with some zip ties.

As the microphone array is mounted on the wall (which is very susceptible to reflections), a layer of acoustic foam is used to attenuate the reflections to make calibration easier.

Gateware

The main objective for the gateware is to reliably transmit the raw acquired data losslessly to the computer for further processing, while keeping it as simple as possible. Performing decimation and filtering on the FPGA would reduce the data rate, but sending the raw PDM data is achievable with Gigabit Ethernet. This reduces the complexity of the FPGA code and allowing faster iteration. Compiling is much faster than place and route, and it’s much easier to use a debugger in code than in gateware!

There are three major components to the gateware, a module for interfacing with the PDM interfaces, a module for creating fixed size packets from those readings, and a UDP streamer to write the packets to the Ethernet interface.

PDM Interface

The PDM input module is a relatively simple piece of logic, which divides the 50 MHz system clock by a factor of 16 to output a 3.125MHz PDM clock, latches all 96 of the input pins after each clock edge, and then shifts out 32 bits of the data on each clock cycle. Each chunk of 192 bits is has a header added which is a 32 bit incrementing integer.

The PDM interface receives data at a rate of 3.125Mhz * 96 (input pins) * 2 (DDR), which is 600Mbps. With the header, the data rate output from this module is 700Mbps, or approximately 40% utilization of the 32 bit output data path.

Packetizer

The packetizer is essentially a FIFO buffer with a special interface on the input. A standard FIFO marks the output as available whenever there is at least one item in the queue, but this would lead to smaller packets than requested as the ethernet interface operates faster than the PDM output (leading to buffer underruns). Thus, the packetizer waits until there is at least a full packet worth of data in the queue before starting a packet, which ensures constant sized packets.

48 PDM output blocks at 224 bits (192 bits of data with a 32 bit header) are placed into each packet, which totals 1344 bytes of data per packet, plus a 20 byte IPv4 header and an 8 byte UDP header, at a rate of approximately 65k pps.

This leads to a wire rate of 715 Mbps, or about 70% utilization of Gigabit Ethernet.

UDP Streamer

The LiteEth project made this module very easy, as it abstracts out the lower level complexities of UDP and IP encapsulation, ARP tables and the like, and provides a convenient interface for simply hooking up a FIFO to a UDP stream. Occasionally there is some latency, but there is enough slack in the bus and buffer in the packetizer FIFO to absorb any hiccups.

Utilization and Timing

The FPGA on the Colorlight i5 is a LFE5U-25F-6BG381C, which has 25k LUTs. The design is placed and routed with the open source Project Trellis toolchain. By keeping the gateware very simple, the utilization on the device is quite low, and there is lots of room for additional functionality.

Info: Device utilisation:
Info:                 DP16KD:    16/   56    28%
Info:                EHXPLLL:     1/    2    50%
Info:             TRELLIS_FF:  1950/24288     8%
Info:           TRELLIS_COMB:  3701/24288    15%
Info:           TRELLIS_RAMW:    49/ 3036     1%

Info: Max frequency for clock                   '$glbnet$crg_clkout': 73.17 MHz (PASS at 50.00 MHz)
Warning: Max frequency for clock '$glbnet$eth_clocks1_rx$TRELLIS_IO_IN': 124.07 MHz (FAIL at 125.00 MHz)

(Timing violations on eth rx clock is due to false positive from gray counter in liteeth)

Software

CIC Filter

Each microphone produces a 1 bit signal at 3.125Mhz, and needs to be reduced to a more reasonable sample rate and bit depth for further processing. This is done very efficiently with a CIC filter, which only requires a few arithmetic operations to process each sample. For understanding more about CIC filters, this series of blog posts from Tom Verbeure provides an excellent introduction. Following the nice graphs from there, I decided on a 4 stage, 16x decimation CIC filter which reduced the sample rate to a much reasonable 195kHz, at 32 bits.

To ingest the data at 3.125Mhz, the filter must be able to process each set of samples in 320ns. A naive implementation in Rust wasn’t fast enough on a single core, but an implementation with some less abstraction (and a hence some more autovectorization) got there, and is what was used at the end. I also experimented with a version using SIMD intrinsics which was much faster, but ended up running into alignment issues when using it in together with other code.

Even with close to a billion bits per second of data to process, a single CPU core can do quite a few operations on each individual bit!

test cic::bench_cic       ... bench: 574 ns/iter (+/- 79) = 41 MB/s
test cic::bench_fast_cic  ... bench: 181 ns/iter (+/- 24) = 132 MB/s
test cic::bench_simd_cic  ... bench:  36 ns/iter (+/- 0)  = 666 MB/s

Calibration

To perform array calibration, a speaker playing white noise is moved around the room in front of the array. An FFT based cross correlation is performed between all pairs of microphones to compute relative delays.

A cross correlation can be performed by computing the FFT of both signals (cached and computed once for each signal), and then computing the inverse FFT of the complex multiplication of the two. This is quite compute intensive, as there are over 18 thousand pairs! For the window sizes used of 16-64k, the FFTs are memory bound, and thus the IFFT and peak finding is fused to avoid writing the results to memory, which results in a 15x speedup. On a 7950X, this process runs in realtime.

Then the positions of the source at each timestep and the positions of each microphone is optimized using gradient descent (when you know PyTorch, all optimization problems look like gradient descent…). The loss function tries to minimize the difference between the measured correlations and the ideal correlations, while trying to minimize the deviation of the microphone positions from the initial positions as well as the jerk of the source trajectory.

As part of the calibration, the speed of sound is also a parameter which is optimized to obtain the best model of the system, which allows this whole procedure to act as a ridiculously overengineered thermometer.

After a few hundred iterations, it converges to a reasonable solution for both the source positions and the microphone positions, as well as constants such as the speed of sound. Fortunately this problem vectorizes well for GPU, and converges in a few seconds.

The final mean position error is on the order of 1mm, and is able to correct for large scale systematic distortions such as concavity from the lack of structural rigidity. The largest position error between the calibrated positions and the designed positions is on the order of 5mm, which is enough to introduce significant phase errors to high frequency sound if uncorrected, although perhaps not strictly necessary (10khz sound has a wavelength of ~3.4cm).

Beamforming

Beamforming is how the raw microphone inputs are processed to produce directional responses. The simplest method of beamforming is delay-and-sum (DAS), where each signal is delayed according to its distance from the source. This is the type of beamforming implemented for this process, with the beamforming happening in the frequency domain.

In the frequency domain, a delay can be implemented by the complex multiplication of the signal with a linear phase term proportional to the delay required, which also nicely handles delays which are not integer multiples of the sampling period.

Multiple nested subarrays of the original array are used for different frequency ranges. This reduces the processing required for beamforming, as each frequency does not need to be beamformed with all the microphone. This also ensures that the beamforming gains of all the frequencies are matched.

Two different types of beamforming visualizations are implemented, a 3d near field beamformer and a 2d far field beamformer. When the audio source is far away, the wavefront is essentially a flat plane, and how far away the source is does not meaningfully change the signals at the array. On the other hand, if the source is close to the array, the wavefront will have a significant curvature which allows the 3d location of the source to be determined.

The beamformer is implemented as a Triton kernel, a Python DSL which compiles to run on Nvidia GPUs. When beamforming to hundreds of thousands of points, the massive parallelism provided by GPUs allows for results to be produced in real time. Some current limitations with the Triton language around support to indexing with shared memory arrays lead to slightly suboptimal performance, but writing CUDA C++ doesn’t seem very fun…

Near Field 3D Beamforming

Near field 3D beamforming is performed a 5cm voxel grid with size 64x64x64. An update rate of 12hz is achieved on a RTX 4090 with higher update rates limited by overhead of suboptimal CPU-GPU synchronization with the smaller work units. The voxel grid is then visualized using VisPy, a high performance data visualization library which uses OpenGL. Modern games have millions of polygons, so rendering a quarter million semi-transparent voxels at interactive framerates is no issue.

A quick demo of the voxel visualization below, note the reflection off the roof!

Your browser does not support the video tag.

Far Field 2D Beamforming

Far field beamforming works similarly, but can be performed in higher resolution as there is no depth dimension required. A 512x512 pixel grid is used, and the same 12hz update rate achieved. (The far field beamforming uses an approximation of just putting the points far away instead of actually assuming planar wavefront due to laziness…)

A demo of the 2d visualization here, but it’s not very exciting due to the poor acoustic environment of my room around the array, with lots of reflections and multipath.

Your browser does not support the video tag.

Directional Audio

The previous two beamforming implementations compute the energy of sound from each location, but never materializes the beamformed audio in memory. A time domain delay and sum beamformer is implemented to allow for directional audio recording. It takes a 3D coordinate relative from array center and outputs audio samples. An interesting aspect about this beamformer is that it is differentiable with regard to the location from the output. This means the location of the audio sources can be optimized based on some differentiable loss function (like neural network), which might allow for some interesting applications such as using a forced alignment model of a multi-party transcript to determine the physical location of each speaker.

A speaker playing some audio is placed in front of the array, with another speaker placed approximately 45 degrees away at the same distance from array center, playing white noise. The effectiveness of the beamforming can be demonstrated by comparing the raw audio from a single microphone with the output from the beamforming.

Raw audio from a single microphone:

Your browser does not support the video tag.

Beamformed audio:

Your browser does not support the video tag.

Recording

As the data from the microphone array is just UDP packets, it can be recorded with tools like tcpdump, and the packet capture file can be read to reinject the packets back into the listener. All the programs in the previous sections are designed to work at real time, but can also work on recorded data using this process.

The tradeoff with this recording implementation is that the output data rate is quite high (due to faithfully recording everything, even the quantization noise). At 87.5 MBps, a 1-hour recording would be 315 GB! A more optimized implementation would do some compression, and do the recording after the CIC filter at a lower sample rate.

Next Steps

I consider this project essentially complete, and don’t plan to work on it any further for the foreseeable future, but there are still lots of possible cool extensions if you’d like to build one!

Using more advanced beamforming algorithms (DAMAS etc.)
Better GUI to combine all existing functions (e.g. See where sound is coming from, and record audio from there)
Combine differentiable beamforming and neural models (e.g. forced alignment example mentioned above)

PCB Linear Actuator

2022-08-09T00:00:00+00:00

A fully integrated PCB linear actuator, with the motor driver, encoder and motor all on a single PCB. All designs are open source:

A video of it in action:

Your browser does not support the video tag.

Motor Design and Validation

Design parameters of the motor were simulated with FEMM. There are not too many free parameters to optimize, but the goal is to find some off the shelf magnets which can be used, and to estimate the importance of things like air gap on motor performance.

The design ended up having the simplest possible winding layout (3 slots per pole), which maximized the amount of current carrying traces, while also having a high winding factor. 50mm * 10mm * 2mm magnets were used (which are a bit smaller in reality by ~0.5mm in each dimension), as they were easily available on EBay etc.

Flux visualization of magnetics simulation.

Force outputs of magnetic simulation. The force is very linear with respect to current, as there is no iron core to saturate. The simulated force seems reasonably accurate compared to the measured force, with likely the biggest discrepancy coming from the questionable quality of the magnets.

The motor itself is constructed using a 4 layer 2oz PCB (with 2oz inner layers) from JLCPCB. The motor itself uses all 4 layers to effectively get 8 oz of copper. At the end turns, each phase uses a single layer, with the last one used as the star point.

It appears the empirical power dissipation is approximately twice as high as the estimates. Not entirely sure why, but my best guess is a combination of copper being slightly thinner than 75um per layer in reality, as well as the power dissipation of the drivers.

There is space to attach a small heatsink on top of the PCB, and a continuous power dissipation of 25W doesn’t seem to cause any issues.

Motor Assembly

The motor is constructed with two mild steel plates, sandwiching an aluminium spacer. Parts were made at SendCutSend, which was quite fast and reasonably priced. The magnets (not shown in CAD) are simply just placed on the steel plates. No additional attachment required. Assembly is quite tricky due to the strength of the magnet, but can be done with some non-magnetic spacers (I used scrap FR4 PCBs).

The linear rail is mounted to the bottom plate with a 20mm x 20mm aluminium extrusion. The motor PCB is attached to the rail blocks with a 3d printed mounting block.

Motor Driver

I used smart power stage ICs (also known as DrMOS) for the power stage (specifically the FDMF3170). This is a bit of an unconventional use for these chips, but it integrates the high side and low side MOSFETs, both gate drivers, temperature and current sensing and bootstrap circuitry into a single chip which only requires a few passives. This made the power stage layout a breeze, and also allows for operation at very high switching frequencies (up to 1Mhz).

Block diagram of the FDMF3170 power stage. You really get quite a lot for just a few bucks!

The power stage layout, which is now pretty boring as each phase is just a single chip.

However, the top and bottom FETs are not balanced, with the bottom side having much lower RDSon than the top side (These are designed for 12V to ~1V switching converters after all). This means the SVM algorithm needs to be slightly modified, away from the traditional alt-rev SVM (where during the zero period, the 111 and 000 phase vectors are both used alternately to balance heat dissipation between the top and bottom sides, under the assumption that the switches are the same). The modified scheme only uses the 000 phase vector, which minimizes resistance losses.

An additional challenge of driving this type of linear motor is that the inductance of the motor is very low, as there is no iron core. This would necessitate an extremely high switching frequency. I decided not to do this (mostly due to running out of timer resolution), and instead added a small inductor (470nH) in series with the motor leads to increase the inductance. This increases the resistance losses, but it is quite small.

The driver is controlled by a STM32F412 microcontroller, and switches at 200khz with the control loop running at 8khz.

Encoder

The encoder is a reflective IR sensor with 8 channels. 4 channels are used to build a quadrature encoder, with differential signals for the sin and cosine phases. The other 4 channels are for 2 other non-differential quadrature tracks with different frequencies. This theoretically enables for absolute positional encoding (as the phase multiple can be computed using the phase difference between the 3 quadrature signals), but I haven’t implemented this yet.

The signals are a bit messy due to some crosstalk between channels, as well as some switching noise, but achieves approximately 10um peak to peak noise after decoding.

Embedded Software

Firmware is written entirely in Rust, using the RTIC framework. The main task is driven by a 8khz timer interrupt, which triggers an ADC scan and DMA requests to read all encoder and motor driver inputs. The DMA completion interrupt triggers a software task which computes the control loop, which sets the PWM duty cycles. A background task handles the USB communications, which uses any remaining CPU time.

One interesting feature of the firmware is in the communication protocol. Rather than hand coding getter and setter methods for each parameter, I wrote a crate called remote-obj to essentially create getter and setters automatically for nested structs and enums using a proc-macro.

Combined with a way to compactly encode the parameters with synchronized metadata between the host and device, this enables an on-device oscilloscope which can sample variables at up to the control loop frequency (up to 16 values at 8khz). The data is buffered on-device using a neat lockless ring buffer which holds packets contiguously in memory for easy access.

This data is then sent to the host over USB CDC ACM. Even though there is no latency/throughput guarantee when using USB bulk endpoints, I’ve found it to be quite fast and low latency, reliably achieving up to 8 mbps out of the theoretical 12 mbps of the USB full speed PHY built in to the STM32, while also having <10ms of latency.

Host Software

Your browser does not support the video tag.

The host software is also written in Rust, using the egui framework. Most of the complexity is involved with handling the USB communication in a low latency way. In particular, dedicated threads drive reading from and writing to the USB endpoints.

To create a low latency USB stream, the easiest way is to continuously send read requests in a loop, and have the USB device terminate the transfer when the device runs out of buffer (using a non-full packet). Importantly to have high throughput is to immediately send another read request when another completes, which keeps the bus busy. The results are communicated over channels to other threads.

The software uses runtime reflection facilities provided by remote-obj to view and set parameters (and to know what parameters are available to view and set).

Performance

100mm moves take under 200ms. (each horizontal tick is 100ms)

From the velocity plot above, it takes approximately 60ms to achieve 2100mm/s, for an acceleration of around 35m/s^2, and it takes approximately 40ms to decelerate back to 100mm/s, for deceleration of around 50m/s^2.

Better performance can likely be achieved with higher currents, but I’m limited by the quite low OCP of my power supply (which is a macbook charger…)

Previous Versions and Future

It took a few hardware revisions to get to this point, and I’ll probably do one more before I call this project “done”

Get in touch if you would like a few assembled boards from the next revision! (contact info in the footer)

Custom Axial Flux Motor

2020-07-10T00:00:00+00:00

This personal project started in 2018 as a way to experiment with electromagnetics simulation and prototyping. A set of relatively arbitrary constraints and goals (~200g, ~60mm diameter, maximise low speed torque) were chosen as this project was primarily an exploratory project with no immediate application.

With these constraints, magnetics simulations were run with a python script, which using gradient descent and simulated annealing, optimized the topology and parameters of the motor with respect to the torque, efficiency, weight and estimated manufacturing cost.

When the optimized design was produced, the models were drawn up in CAD, and these drawings were then sent to be manufactured in China, including milled carbon fiber, custom neodymium magnets, and electric discharge machined magnetic steel.

The assembled motor was then tested, and performed within 20% of the simulations, which validated the design and prototyping process for custom motors. Furthermore, the motor constant obtained was approximately 2x higher than motors of comparable weight and diameter, with higher continuous torque and efficiency.

LFTag

2020-07-10T00:00:00+00:00

LFTag is a new visual fiducial standard, similar to AprilTag, ArUco and ARTag, designed to maximize the use of Low spacial Frequencies (hence the name), and significantly advances the state of the art in dictionary size-detection range tradeoffs for visual fiducial markers. It was a independing research project completed in the summer of 2020, and will be presented at IEEE CVIV 2020.

A preprint is available here, and the conference presentation slide deck is available here.

The gist of the algorithm consists of topological connected component analysis to find potential markers in the image, with data encoded within the marker using the relative positioning of marker elements.

MOC900X

2020-07-10T00:00:00+00:00

MOC900X is a series of high power density motor drivers which I am designing for the University of Pennsylvania’s Formula SAE Electric team, Penn Electric Racing. This project was started in late 2019, and is currently ongoing.

The goal of this project is to design a high voltage (~600V), high power (~60kW) motor controller for the 2021 season, with strong priorities placed on weight and efficiency. Due to these goals, high performance GaN transistors were selected and evaluated on a custom insulated metal substrate half bridge test board controlled by an FPGA:

The half bridge design has been validated both electrically and thermally, and performs at over 99% efficiency at 500V, 100A. The current design iteration puts three half bridges together, and is on target to achieve extremely high whole system specific power, on the order of 170kW/kg including cooling, enclosure and connectors

RemoteBoot

2020-07-10T00:00:00+00:00

RemoteBoot was a Kickstarter project which I ran in 2016, when I ran into the frustration of having to leave my computer on all day to be able to remotely access it, having no way of powering it on remotely. Thus, I came up with the idea of RemoteBoot, a module fitting into any desktop computer that allows for remote turn on and monitoring over WiFi.

This device consists of a WiFi module (ESP8266), transistors for turning the computer on and off, as well as optoisolators to enable LED monitoring. The firmware was written in C++, with a Javascript web UI, along with a PHP based cloud management interface.

This Kickstarter project was extremely successful, raising AU$17,589 from 431 backers, with 1000 units manufactured in Shenzhen and successfully delivered to all backers. This project was also covered by some engineering youtubers, such as Dave Jones from the EEVBlog.

The over 800 devices in use today save significant amounts of energy worldwide, which to date have saved over 600MWh of energy, and reduced CO2 emissions by over 400 tons.

Perf+

2020-07-10T00:00:00+00:00

Perf+ was a series of crowdfunding projects which I ran in 2015 and 2017, which improved the way electronics prototyping was traditionally done.

By combining a traditional perfboard (FR4 board with 0.1” grid of plated through holes) with sets of wires built into the board which are disconnected by default to the holes, but could be joined with a solder blob. There were horizontal wires on one side of the board, and vertical wires on the other, allowing any two points on the board to be connected easily without using any jumper wires.

The above is a simple routing example, the yellow dots represent the points to connect and the red is solder on each side to the points they are connecting.

In 2015, I created a Kickstarter project for this concept, which successfully raised AU$15,461 from 614 backers. Over 8,000 boards were manufactured in Shenzhen, and all rewards were fulfilled before schedule.

In 2016, with customer feedback from the first revision, I revised the design to improve the user experience of the boards, and created a project with CrowdSupply, another crowdfunding platform. On CrowdSupply, I raised $11,898 from 271 backers. This time, over 10,000 boards were manufactured, and all rewards were fulfilled on schedule.

These boards were reviewed by many online publications, such as Hackaday and EEVBlog, and are now available on Amazon.

Welcome

2020-07-10T00:00:00+00:00

My name is Ben Wang and I am an undergraduate student studying electrical engineering and management at the University of Pennsylvania.

I have a wide range of interests, including power electronics, embedded devices, FPGAs, blockchains, computer vision, networking and AI (Phew that was a mouthfull). I hope to share my journey exploring these topics here.

If you like the work shown here and are looking for someone in these fields, I would love to hear from you!