Networking the Sequencer Board

Overview

The pulse sequencer board communicates with computers over a standard 100 Mbps Ethernet interface using the Pulse Transfer Protocol (PTP).  The original firmware determines the IP address from the device ID (set by DIP switches 2-5) when DIP switch 6 is flipped down.  This allows one to easily determine the IP address of a sequencer board, but is problematic if another system on the network is using the assigned IP address.  When DIP switch 6 is flipped up, the sequencer board will find an unused address, avoiding this problem, but now one has to send a broadcast packet from the computer to find out what the address of the board is.

The usefulness of this feature is highly debatable, since it addresses problems for which there are already effective workarounds.  Most users have a router for each sequencer board, meaning that if the board chooses its own IP address, it’s not likely to choose an address that’s already in use.  Implementing these features will hopefully make the firmware more robust, however.  This operation is thus more of a “keep Jeff busy” task until the physics community gives me something more urgent to do.

API Change to Support Multiple Devices

This change applies to the Python software in the “pulse-sequencer” SVN repository.

In the past, the networking API for the test configuration (/pulse-sequencer/software/python/test-config.py) did not support addressing multiple devices.  Device discovery was already implemented in the discover_devices() method defined in /pulse-sequencer/software/python/sequencer/ptp/__init__.py .

I made a change to this API to support multiple devices.  Now, when you call setup(), it calls discover_devices() to determine which device IDs correspond to which IP addresses.  You can call select_sequencer(device_ID) to “select” that device, meaning all subsequent commands will be sent to that device.  To choose a different device, simply call select_sequencer() with the device ID of the device you want to use next.

This change has (probably) not been committed yet, because discover_devices() is a time-consuming process.  It’s less than five seconds, but runs every time setup() is called.  I’m not sure if physicists’ scripts are calling setup() only once per experiment (as they should), or if they’re calling setup() thousands of times — which could incur a huge delay.  I’m planning to introduce a workaround where setup() caches the discover_devices() data, and simply re-uses that data whenever it’s called again.  If it detects the data is invalid, it will call discover_devices() to set things right.

Dynex Router Support

The sequencer board is designed to operate on a network.  In order to do so, it needs to obtain an Internet Protocol (IP) address, which identifies it on the network.  To obtain an IP address, it uses the Dynamic Host Configuration Protocol (DHCP) to communicate with a DHCP server.  The server offers an IP address that isn’t currently being used by something else, and the board accepts and stores it.  This process consists of sending four messages:

1. DHCPDISCOVER – the sequencer board sends a broadcast message asking for an IP address.
2. DHCPOFFER – the DHCP servers on the network offer IP addresses to the sequencer board.
3. DHCPREQUEST – the sequencer board contacts a specific DHCP server to confirm its offer.
4. DHCPACK – the server ACKnowledges that the sequencer board has accepted the address.

The old sequencer board firmware only worked with Linksys routers.  When we tried to use a Dynex router, the sequencer board could not obtain a DHCP lease.  The problem was a malformed DHCPREQUEST message: the board assigned itself the offered IP address BEFORE sending the DHCPACK message.  The Dynex router didn’t like this, so it kept sending DHCPNACK  messages back — to indicate failure.

The fix was remarkably simple: I added DHCP Option 53 to the DHCPREQUEST message, to indicate the IP address the board chose.  I also wrote a hack to force the DHCPREQUEST messages to come from 0.0.0.0 instead of the board’s chosen IP address.  It worked almost immediately, and I completed this task within a single lab period — a grand surprise given that hardware labs in class often take several lab periods.

DHCP Lease Renewal

When a network device like the sequencer board obtains an IP address from a DHCP server, it doesn’t get to keep the address forever.  The DHCP server sends a lease time, in seconds, indicating how long the sequencer board gets to have its IP address.  At the end of this lease time, the DHCP server may give the address to another network device.  If this happens, two devices will have the same IP address, which is never supposed to happen!

To resolve this problem, I’m working on implementing lease renewal.  When the lease time is almost over, the board will need to ask the DHCP server to extend the lease time, preventing the server from giving the board’s IP address to another system.  This system involves adding some logic to the DHCP module, and adding a counter to track the time.

Logic elements are the small pieces on the FPGA that make up the circuits in the firmware.  The Altera QuartusII compiler configures these logic elements to make the FPGA behave in the way we specify in VHDL code in the sequencer-vhdl repository.  We’re using an Altera Cyclone FPGA, which has a measly ~12,000 logic elements.  I say “measly” because we’re at 87% utilization — we have around 1500 logic elements remaining.  Since we would like to add new features to the chip, we need to make the most of our remaining logic elements.

The lease counter needs to hold a maximum lease of about 1 day.  If the server sends a longer lease, it gets truncated.  1 day = 86400 seconds, which requires 17 counting bits.  Since our board clock operates at 100 MHz, we need to count 100 million clock pulses for each second, so our counter will require 43 bits.  Even with optimizations, that’s still a big counter — it required about 120 logic elements when I coded it in Verilog.

To make the counter smaller, I implemented a Linear Feedback Shift Register (LFSR).  Like a counter, an LFSR changes its combination of bits on each clock cycle.  If you have 2^n memory bits, a LFSR will cycle through (2^n – 1) different combinations of bits, using only n registers and one additional logic element.  However, there’s no straightforward way to determine how many clock ticks have occurred by examining the LFSR’s current state, unless you use more logic elements than a regular counter would require.  If you know an LFSR’s start state, you can calculate its state after X clock ticks while you’re designing the hardware, and use the counter to tell you when X clock ticks have passed.  This means I could use an LFSR to tell me when a second has passed, and use a regular counter to count the seconds remaining in the lease time.  This implementation required ~96 logic elements when I coded it in Verilog.

Couldn’t I track the time in minutes or hours instead of seconds?  That could eliminate some bits, but remember: the lease time is transmitted in seconds, so I would need circuitry to convert it.  Also, why bother with such a complicated design when a mere 20 logic elements are at stake?  I thought it would be an interesting exercise to see if this optimization would really make a difference.

Future Projects

Connecting and Disconnecting from a Network

Currently, the sequencer board doesn’t appear to detect when it has been connected and disconnected from a network.  This detection is important because the network module needs to “reset” itself when the network goes down.  Otherwise, it might try to renew its IP address on a network it’s not connected to, or assume that it still has its old IP address if the router it’s connected to gets rebooted.

The obvious workaround is to power-cycle the sequencer board (turn it off, then on) after plugging in a network cable or rebooting a router.  However, this could be tedious if you have lots of sequencer boards.  Given that most people don’t change their networking configuration very often, though, this might not be a significant issue.

Remote Access

Ah, the dream of every physicist: running experiments while sitting on the beach!  Remote access to sequencer boards would allow physicists to operate them without being in lab, which could reduce transportation costs and allow more people to share a trapped ion quantum computer setup to run experiments.

The complexity of this setup depends on the connection between your computer and your sequencer board.  If you are connected via a shared public network, for example, you need security.  You can purchase a Linksys VPN router here for just $99 ($50-ish used) that lets you connect to that router from anywhere (using a username and password).  You’ll need to make sure it works with your organization’s Internet setup (some organizations have firewalls that don’t allow you to connect from outside their network).  You could also try to download new firmware onto your existing router, but that runs the risk of “bricking”, or ruining, your router.

We could try to make the pulse transfer protocol routable, but we would still have to encrypt it somehow, and the encryption hardware would (probably) not fit on the FPGA.  If you’re interested in remote access, let us know!

leading up to squint

Posted by Paul Pham

In a few days I’ll be leaving for SQuInt, just a man, a van, a plan, and Panama. Don’t be surprised if there is a renegade poster or private demonstration of a yellow box, or if people are waving around FPGA sequencer boards. For the past few weeks, Rob, David, and I have been hard at work getting the input counting feature to work in time for its big unveiling in Santa Fe. I am fully confident that we’ll finish testing it before we leave.

Software support is already added in the v1 Python compiler, and the latest version in Git works for four out of five subopcodes (counter reset, counter latch, counter compare to a threshold, counter branch-on-compare). The last subopcode, counter write-to-memory, is tricky because it involves stopping the PCP fetching instructions from memory, writing to a new address (via an address accumulator that increments by one each time), and then resuming the PCP instruction pipeline at the same address as before. As you would expect, there are issues with delay slots (just insert more nops!), converting addresses between 32-bit (use by PCP) and 8-bit (used by PTP), and clocking issues (synchronizing between triggers and the system clock).

The other members of our team have also been busy in the meantime.

Jeff is finishing up new support for network device discovery. No more hard-coded IP addresses or dedicated routers! With his new changes, the sequencer can get a dynamic DHCP address from any gateway / router, count down the lease time, and then release and renew its IP address. This, combined with re-enabling the ICMP (ping) module, will make the sequencer a first-class network citizen again. He has also added support to the Python v1 compiler for discovering all sequencer devices on the network dynamically, and selecting between them via the software API when loading pulse programs. I won’t reveal any more, but hopefully while I’m gone he’ll write up his own blog post.

David and John have started work on the clock board, which should make the existing system a little more compact and less like a cable jungle. This is their very first circuit design, using one of my favorite open source projects, gEDA/PCB, complete with 70s X11 widgets and arcane keyboard shortcuts. Some day, when you are a crotchety old man, you can also make your students use weird toolchains because it is cheaper and builds character.

Peter worked on designing a proposal for pcp5 to support parameter scanning, which is sleeping for now while physicists decide if they really need it. But arithmetic and register support are pretty useful, and I am frustrated that ginormous hard-coded sequences are necessary for parameter scans. Eventually, my engineer’s sensibilities will boil over and force the issue. For now, Peter is working on modifying the firmware build system to generate multiple binary targets. This is the first step to creating alternate configurations and even unit tests for firmware.

Finally, our own Aaron Avril will be presenting a SQuInt poster on his yellow box. Here are some candid photos in lab, courtesy of Jeff Booth. See if you can guess what all the parts are.

Wait-Trigger & Branch Trigger Level Detection

Posted by John Williams

This post covers the addition of two requested features to the Python v1 Compiler and the underlying pcp32 firmware.

• Branch-on-trigger Level Detection
  • Python API Call: branch(label, triggers, level = 1)
  • PCP32 Instruction: btr [triggers] [level] [address]

• Branch Wait
  • Python API Call: branch_wait(label, triggers, level = 1)
    

Branch-on-trigger Level Detection

One feature that has been requested is the ability to select the level to trigger when using a branch event.

When this flag is set to zero, the trigger mask will select bits of the trigger inputs that must all be low in order for a trigger to occur. If all of the bits of the trigger mask are zero, the default behavior is to never trigger regardless of whether high or low level detection is used.

Branch Wait

The branch-on-trigger (btr) instruction was previously used mainly in loops to spin the processor while waiting for some trigger input.

It was discovered that this functionality could be duplicated by taking advantage of a property of the halt instruction that causes the last instruction in the pipeline to be executed every cycle.

The branch_wait event will be compiled down to the following sequence:

halt, nop, nop, nop, nop, nop, btr

filling all 6 branch delay slots and achieving the desired behavior.

Note: These features are not included in the recent 0.32 release but are in the repositories and will be included in all subsequent releases; additionally, the new level argument is optional for each and will default to the previously assumed value of  ’1′

Input Counting Instruction added to the Python Compiler v1

Posted by David Nufer

Overview

The Input Counting instruction (icnt) has now been added into the Python Compiler v1 and is available on the SVN.  The icnt instruction has 5 subopcodes that define different functions:

• Reset — resets one of the input counters.
• Latch — latches the current value of one of the input counters to its associated register.
• Write — writes the value of one of the input counter registers to the memory address in the address accumulator, and increments the address accumulator.
• Compare — compares an input counter register to its associated compare register and sets the compare flag it the comparison is true.
• Branch — branches if the compare flag is set.

New API Calls

There are five new API calls for the icnt instruction, one for each function:

• reset_input_counter(input_channel) — input_channel :  input trigger
• latch_input_counter(input_channel) — input_channel :  input trigger
• write_input_counter(input_channel) — input_channel :  input trigger
• compare_input_counter(input_channel) — input_channel :  input trigger
• branch_input(label) — label : the instruction to jump to

There are 8 input triggers are defined in /software/python/sequencer/constants.py, one for each input channel.

Test Script

There is a small test script called test_input_counter.py that uses all the new API calls.  At first glance, it appears that all the new API calls compile properly.  The test script is also being used to test that the firmware implementation of the icnt instruction is working properly.