Networking the Sequencer Board

Overview

The pulse sequencer board communicates with computers over a standard 100 Mbps Ethernet interface using the Pulse Transfer Protocol (PTP).  The original firmware determines the IP address from the device ID (set by DIP switches 2-5) when DIP switch 6 is flipped down.  This allows one to easily determine the IP address of a sequencer board, but is problematic if another system on the network is using the assigned IP address.  When DIP switch 6 is flipped up, the sequencer board will find an unused address, avoiding this problem, but now one has to send a broadcast packet from the computer to find out what the address of the board is.

The usefulness of this feature is highly debatable, since it addresses problems for which there are already effective workarounds.  Most users have a router for each sequencer board, meaning that if the board chooses its own IP address, it’s not likely to choose an address that’s already in use.  Implementing these features will hopefully make the firmware more robust, however.  This operation is thus more of a “keep Jeff busy” task until the physics community gives me something more urgent to do.

API Change to Support Multiple Devices

This change applies to the Python software in the “pulse-sequencer” SVN repository.

In the past, the networking API for the test configuration (/pulse-sequencer/software/python/test-config.py) did not support addressing multiple devices.  Device discovery was already implemented in the discover_devices() method defined in /pulse-sequencer/software/python/sequencer/ptp/__init__.py .

I made a change to this API to support multiple devices.  Now, when you call setup(), it calls discover_devices() to determine which device IDs correspond to which IP addresses.  You can call select_sequencer(device_ID) to “select” that device, meaning all subsequent commands will be sent to that device.  To choose a different device, simply call select_sequencer() with the device ID of the device you want to use next.

This change has (probably) not been committed yet, because discover_devices() is a time-consuming process.  It’s less than five seconds, but runs every time setup() is called.  I’m not sure if physicists’ scripts are calling setup() only once per experiment (as they should), or if they’re calling setup() thousands of times — which could incur a huge delay.  I’m planning to introduce a workaround where setup() caches the discover_devices() data, and simply re-uses that data whenever it’s called again.  If it detects the data is invalid, it will call discover_devices() to set things right.

Dynex Router Support

The sequencer board is designed to operate on a network.  In order to do so, it needs to obtain an Internet Protocol (IP) address, which identifies it on the network.  To obtain an IP address, it uses the Dynamic Host Configuration Protocol (DHCP) to communicate with a DHCP server.  The server offers an IP address that isn’t currently being used by something else, and the board accepts and stores it.  This process consists of sending four messages:

1. DHCPDISCOVER – the sequencer board sends a broadcast message asking for an IP address.
2. DHCPOFFER – the DHCP servers on the network offer IP addresses to the sequencer board.
3. DHCPREQUEST – the sequencer board contacts a specific DHCP server to confirm its offer.
4. DHCPACK – the server ACKnowledges that the sequencer board has accepted the address.

The old sequencer board firmware only worked with Linksys routers.  When we tried to use a Dynex router, the sequencer board could not obtain a DHCP lease.  The problem was a malformed DHCPREQUEST message: the board assigned itself the offered IP address BEFORE sending the DHCPACK message.  The Dynex router didn’t like this, so it kept sending DHCPNACK  messages back — to indicate failure.

The fix was remarkably simple: I added DHCP Option 53 to the DHCPREQUEST message, to indicate the IP address the board chose.  I also wrote a hack to force the DHCPREQUEST messages to come from 0.0.0.0 instead of the board’s chosen IP address.  It worked almost immediately, and I completed this task within a single lab period — a grand surprise given that hardware labs in class often take several lab periods.

DHCP Lease Renewal

When a network device like the sequencer board obtains an IP address from a DHCP server, it doesn’t get to keep the address forever.  The DHCP server sends a lease time, in seconds, indicating how long the sequencer board gets to have its IP address.  At the end of this lease time, the DHCP server may give the address to another network device.  If this happens, two devices will have the same IP address, which is never supposed to happen!

To resolve this problem, I’m working on implementing lease renewal.  When the lease time is almost over, the board will need to ask the DHCP server to extend the lease time, preventing the server from giving the board’s IP address to another system.  This system involves adding some logic to the DHCP module, and adding a counter to track the time.

Logic elements are the small pieces on the FPGA that make up the circuits in the firmware.  The Altera QuartusII compiler configures these logic elements to make the FPGA behave in the way we specify in VHDL code in the sequencer-vhdl repository.  We’re using an Altera Cyclone FPGA, which has a measly ~12,000 logic elements.  I say “measly” because we’re at 87% utilization — we have around 1500 logic elements remaining.  Since we would like to add new features to the chip, we need to make the most of our remaining logic elements.

The lease counter needs to hold a maximum lease of about 1 day.  If the server sends a longer lease, it gets truncated.  1 day = 86400 seconds, which requires 17 counting bits.  Since our board clock operates at 100 MHz, we need to count 100 million clock pulses for each second, so our counter will require 43 bits.  Even with optimizations, that’s still a big counter — it required about 120 logic elements when I coded it in Verilog.

To make the counter smaller, I implemented a Linear Feedback Shift Register (LFSR).  Like a counter, an LFSR changes its combination of bits on each clock cycle.  If you have 2^n memory bits, a LFSR will cycle through (2^n – 1) different combinations of bits, using only n registers and one additional logic element.  However, there’s no straightforward way to determine how many clock ticks have occurred by examining the LFSR’s current state, unless you use more logic elements than a regular counter would require.  If you know an LFSR’s start state, you can calculate its state after X clock ticks while you’re designing the hardware, and use the counter to tell you when X clock ticks have passed.  This means I could use an LFSR to tell me when a second has passed, and use a regular counter to count the seconds remaining in the lease time.  This implementation required ~96 logic elements when I coded it in Verilog.

Couldn’t I track the time in minutes or hours instead of seconds?  That could eliminate some bits, but remember: the lease time is transmitted in seconds, so I would need circuitry to convert it.  Also, why bother with such a complicated design when a mere 20 logic elements are at stake?  I thought it would be an interesting exercise to see if this optimization would really make a difference.

Future Projects

Connecting and Disconnecting from a Network

Currently, the sequencer board doesn’t appear to detect when it has been connected and disconnected from a network.  This detection is important because the network module needs to “reset” itself when the network goes down.  Otherwise, it might try to renew its IP address on a network it’s not connected to, or assume that it still has its old IP address if the router it’s connected to gets rebooted.

The obvious workaround is to power-cycle the sequencer board (turn it off, then on) after plugging in a network cable or rebooting a router.  However, this could be tedious if you have lots of sequencer boards.  Given that most people don’t change their networking configuration very often, though, this might not be a significant issue.

Remote Access

Ah, the dream of every physicist: running experiments while sitting on the beach!  Remote access to sequencer boards would allow physicists to operate them without being in lab, which could reduce transportation costs and allow more people to share a trapped ion quantum computer setup to run experiments.

The complexity of this setup depends on the connection between your computer and your sequencer board.  If you are connected via a shared public network, for example, you need security.  You can purchase a Linksys VPN router here for just $99 ($50-ish used) that lets you connect to that router from anywhere (using a username and password).  You’ll need to make sure it works with your organization’s Internet setup (some organizations have firewalls that don’t allow you to connect from outside their network).  You could also try to download new firmware onto your existing router, but that runs the risk of “bricking”, or ruining, your router.

We could try to make the pulse transfer protocol routable, but we would still have to encrypt it somehow, and the encryption hardware would (probably) not fit on the FPGA.  If you’re interested in remote access, let us know!

Advertisement