Design for Raspberry Pi carrier boards
This ticket collects requirements and notes for the design of the Raspberry PI CM4 carrier boards for CloudLab.
In contrast to other carrier boards, this will be designed for density, dual onboard networks, and manageability. The goal is to get as many CM4s per board as is practical given physical constraints and availability of things like Ethernet ports on the on-board chips. Our current attempts at design have tried aiming for N 5 or 8 experiment nodes per board.
In addition to the experiment nodes, there will be one onboard control node that will handle booting the experiment nodes, power control over them, configuring the onboard switches, etc.
The goal is for each board to have the minimum number of external connectors (power, Ethernet, USB, etc.) while meeting the requirements below.
These boards are designed to be used with CM4 with onboard eMMC (size TBD, but likely 16GB), no WiFi. RAM on CM4 TBD, but possibly 4GB) eg. part number CM4104016
.
Requirements for each experiment node (CM4 socket - N of them):
-
Power control (via access to the GLOBAL_EN
pin, connected to aGPIO
on the control node - note: this is at 5V, so we likely want something between it and theGPIO
to prevent leakage current) -
Control over boot from onboard eMMC or rpiboot (USB) (via the nRPI_BOOT
pin, connected to aGPIO
on the control node) -
Control over EEPROM
write enable/disable (via theEEPROM_nWP
pin, connected to aGPIO
on the control node) -
Serial console (through built-in UART
pins (GPIO14/TXD0
andGPIO15/RXD0
) connected to aUART
on the control node`) -
USB client device for rpiboot/flashing (eg USB_N
/USB_P
pins) connected to a USB hub/port on the control node -
Experiment network: builtin Ethernet connected to an onboard switch -
Control network: Ethernet via PCIe Ethernet controller (eg. RTL811H
) connected to an onboard switch -
LEDs for debugging (exact set not yet determined, these are some possibilities) -
Power via CM4_3.3V
, which is off whenGLOBAL_EN
is pulled low -
Status of nRPI_BOOT
and/orEEPROM_nWP
pins -
Pi_nLED_Activity
-
Requirements for the control node:
-
Sufficient GPIO
s for functions listed above (may require anI2C
GPIO
expander depending on N) -
Sufficient USB 2.0 ports to act as a host for all N experiment nodes via an onboard hub -
Sufficient UART
s (via USB orI2C
UART adapter) for all N experiment nodes -
Its own RJ45 connector on the board connected to its builtin Ethernet for out of band control -
SPI
orI2C
connections to onboard switches to configure them -
Connection to the onboard control switch (unless this would cost us a node, in which case, we could handle by giving it its own jack on the board) -
Power control (pins to control from another machine, plus an onboard reset button) -
Serial console (via either USB or directly exposed RS232) -
Some way to reflash, though this can be rare (eg. by swapping an SD card) -
Sufficient storage to cache disk images locally -
Same debugging LEDs as experiment nodes
Requirements for each switch (2 of them, one experiment and one control)
-
At least N+1 gigabit Ethernet ports (one for each node, plus one to connect off-board - needs N+2 if we want an onboard connection to the control node) -
SPI
orI2C
control from control node, ability to manipulate VLANs -
Reset or power control via GPIO
pin on the control node -
RJ45 connector for off-board connectivity
Requirements for the board in general:
-
Ideally a single power connector, either barrel or ATX - will need 5V for the CM4s and 3.3V for various other devices
Questions remaining:
-
Can we connect GLOBAL_EN
,nRPI_BOOT
,EEPROM_nWP
directly toGPIO
s on the controller, or do we need a transistor or voltage divider for better safety and/or reliability? -
See questions regarding network and management interfaces for control node
Current design at Gepetto/Upverter
I don't see a way to share designs without the other person having an account, so make an account at https://geppetto.gumstix.com/ and let me (@ricci) let me know what email address you signed up with.
The current design hosts _N_=5 nodes plus one control node. It's at the maximum board size for the tool: 22.8 cm x 15.2 cm. Everything is mounted flat on the top of the board, so it should fit easily in 1 U (1.5 inches) with plenty of room for heatsinks.
Things that are not ideal:
-
Power distribution: as designed, it is powered at 36V and needs two barrel jacks to reach the rated capacity of about 88W. It requires a 5V/5A regulator for each CM4, plus a bunch of 3V/1.5A regulators for other components (which have to be fed from the 5V regulators because they don't take 36V input). A much simpler (and cheaper) design should be possible: power the whole thing at 5V if we can handle the current, use a smaller number of larger regulators, etc. Possibly see if we can use an ATX power supply since it provides both 5V and 3.3V
Big things missing:
-
Geppetto does not have a PCI Ethernet chip. I've put USB ones in as placeholders, but we'd need to get them to add one. I've tested the RTL811H
and it seems to work well, though has the downside that the driver is out of tree. -
I don't see a way to put a single transistor or 74xx06
between theGPIO
and theGLOBAL_EN
line as noted elsewhere in this ticket. -
I haven't added 'out of band' flashing, power cycling, etc for the control node, see question elsewhere in the ticket.
Small things missing:
-
Geppetto does not currently provide access to the GLOBAL_EN
,nRPI_BOOT
andEEPROM_nWP
pins on the CM4 connector. I'm guessing this is easy for them to add -
I have not put status LEDs or mounting holes on yet (including mounting holes for the CM4s)
Factors limiting number of CM4s:
-
Board space: board space is currently not completely full, and it should be possible to squeeze a few more on there physically, especially if the power distribution messiness can get fixed, the regulators currently take up quite a bit of space. There are also options to mount the CM4 on an SODIMM adapter board (like the CM3), but this would cost more and run into potential problems fitting into 1U. I think their adapter boards also don't expose all of the lines we need. -
Switch ports: the board currently uses switches that have 5 ports, plus 2 that are to be used for uplinks. So, to add more nodes, we need to either: - Get Geppetto to add a larger version of the same switch with more ports (which will involve engineering costs)
- Add 2 more switches, and connect them together on the board in a way that will introduce bottlenecks and complexity
- Add 2 more switches, and give each its own uplink port. This doubles the number of Ethernet cables and switch ports we need
-
USB hubs: the USB hub I'm currently using only has 7 device/client ports so we'd need to daisy chain a few together to get enough ports -
GPIO
pins: with 3 pins needed on the control node per experiment node, we would likely use up all available pins and would need to add more through anI2C
GPIO expander or similar. -
Power and heat density: I have not even conclusively verified that we are OK at this density, and of course more density is even harder