Network Hardware – Supply Chain Security – The Big Hack analyzed

Network Hardware – Supply Chain Security – The Big Hack analyzed – iRis Networks

Recently there have been several articles challenging Chinese hardware security. The articles describe servers manufactured in China for a US company. The initial claims indicate a “hacker” added silicon (alien) devices to the PCB (Printed Circuit Board) assembly allowing modification of the server’s BIOS and server’s eventual “hacking”. There are several technical issues with the article, challenging the article’s validity, and major parties have denied the charges. True or not, the topic does highlight a concern; should your company rely on China manufactured hardware in your company’s servers and network? Links to supporting articles are at bottom of blog.

Article Summary
The Bloomberg article claims SuperMirco Computer Inc servers, subcontract manufactured in China, had alien devices added during the manufacturing process for the sole purpose of allowing the servers to be hacked by outside sources. On reading, the article there is a little confusion over exactly how the hacking occurs. However, it appears the alien hardware modifies the Baseboard Management Controller’s (BMC) code in route to the controller, changing the baseboard LAN configuration and maybe modifying some of the actual BMC code. Once modified an outside person can access the server and make changes to BIOS and firmware, or remotely control server with the KVM (keyboard video mouse).
This leaves more questions than answers; the Bloomberg writer did a good job given the audience and complexity of issue. However, it is hard to validate claims without knowing more. For a more detailed analysis and questions, please see section, “Technical questions with the article / charges”

China supply chain impact on iRis and iRis’s customers
• iRis’s network is 100% free of Chinese assembled hardware.
• Ciena is iRis’s primary network vendor for backhaul and MetroE. Ciena assembles hardware primarily in NAFTA countries. iRis does not use any Ciena hardware assembled in China.
• Juniper is iRis’s primary network vendor for Routers. iRis does not use any Juniper hardware assembled in China.

How does iRis manage its vendors and supply chain?
When choosing a vender, iRis goes through an extensive RFI/RFP (Request for information / Request for Pricing) process. This process achieves many goals; understanding the vendor’s manufacturing process and supply chain is a primary goal. Some of the important areas related to the topic of secure supply chain are:
• Country of Origin: Where is the device assembled?
• Where designed: Where are the designed located?
• Where is product support based: Once a device is in production the original designers are no longer involved in product support, change control, or hardware improvements. Therefore, where support is located is just as important as original design location.
• Second Sourcing: All devices on an assembled board should have a second source where possible. Second sourcing allows production to continue even if single supplier or factory is lost.
• Four Corners Testing: Four corners testing involves testing the device at min/max temperate and min/max voltage ranges. This simple functional test uncovers signal integrity issues and increases the product’s reliability. If the manufacturer adds alien devices or traces, this test would produce failures requiring additional testing.

Technical questions with the article / charges
I have not been involved with server hardware design in over thirteen years (wow time flies). While the theory does not change, I am sure I have missed many manufacturing and engineering advances. I hope that some of my esteemed colleagues, still in the server design business, can provide feedback to my blog. If nothing else, I hope this section promotes discussion on possible hardware hacks.
This section has thoughts and questions on adding an alien device to a motherboard as described in the article. While questioning the technical explanation of the hack, please keep in mind the goal is not to discredit the possibility of a hardware hack. A compromised supply chain is a huge security risk, not taken lightly.

Controlling a data bus:
There are several issues with trying to control a data bus and injecting data. Here are some of the unresolved issues of implementing hack as described by the article.

Serial vs Parallel bus:

At one time, the BMC used both a slow parallel data bus and an extremely slow I2C serial bus to collect data. Many of the buses on motherboards have migrated to differential high-speed serial buses. In the server described, I do not know the BMC’s bus architecture. However, here are issues seen in either case:

Most of the high-speed buses on a server are now differential high-speed serial buses. These buses use two signal traces for each signal. To increase bandwidth designers add additional serial buses. Intercepting and changing signals can be challenging for two main reasons. First the bus is fine-tuned for signal integrity; you cannot add a stub or any load to the bus without drastically affecting the ability of the bus to send data reliably. Second, the serial bus is encoded prevent too many zeros or ones in a row and for other signal integrity concerns.

If the BMC uses a parallel bus, there would need to be at least eight data signal traces. This would be a challenge for the small footprint of the alien device used by the hacker in the attack described. How do you get all those signal traces to a single device that is so small?

I doubt that a new server uses the I2C bus for BMC memory storage but if it did, this slow speed, shared, serial bus would be the easiest to attack. The I2C bus architecture allows for multi masters. The bus has pull up resistors and the master device pulls the signal low to transition the data. Therefore, this is the most technically feasible bus to attack, but I doubt the server described by the article uses I2C bus for BMC memory access.

How do you over drive a bus without damaging devices?
In any data bus other than I2C bus, there is a single device sending the signal, it is not a shared bus. On a non-shared bus, how does the alien device drive the bus signal high, if the source device is pulling the bus low? Pulling the data signal high while the source device is pulling the bus low cannot be done without adding additional devices (to the best of my knowledge).

Power to alien devices
The alien device must have contacts/pads/pins for the signals it is controlling and power and ground. The device pictured does not seem large enough for this quantity of pins. It is also likely that a via would need to be added to bring power and ground to the alien device.

Feasibility of relaying out PCB (motherboard) to connect the alien device:
The article mentions a foreign entity, acting as SuperMicro, changing the board files resulting in new motherboards (PCBs). There is nothing preventing this but there are a few hurdles. A few of the hurdles:

• The PCBs manufacture occurs at a different facility and company. Therefore, hacker will need to compromise at least one additional supply chain entity, two if the PCB is second sourced.
• Modifying the CAD files used by the PCB manufacturer without the original design files. This can definitely be done but extreme care would need to be taken when adding via’s. Vias are the copper links connecting traces on different motherboard layers.
• Release control – The design owner, SuperMicro, should have release control mechanisms in place even with 3rd party manufacturing. There are checks and balances that should prevent a manufacture from going rouge. Going Rouge includes swapping out devices to increase or decrease cost. In this case SuperMicro completely lost control of the manufacturing and release control mechanisms. When releasing a new board layout many changes are made to the manufacturing process, from in circuit test file, pick and place files, and image scans. If the manufacturer takes complete control of generating these files, making changes without SuperMicro’s knowledge is possible.
• What if, SuperMicro modifies the PCB to correct a problem? A SuperMicro design change would completely foul up the works for the hacker. With enough time and money it is possible the hacker can overcome this hurdle.
• It is worth noting for simple trace additions, the PCB could have additional traces printed on post PCB manufacture. This process is used to “blue wire” boards, implementing quick fixes while design team relays out the board. “Blue wire” fixes only work for slow speed buses.

Connecting the BMC LAN Ethernet port to the outside world?
The BMC LAN Ethernet port is not the same as the NIC (Network Interface Controller) ports used to connect the server to the network. The attack described by the article assumes the end user connects the BMC LAN Ethernet port to a local network and that that local network has access to the outside world (Internet). While connecting this port to a LAN is common, giving BMC LAN access to the Internet would be a huge security failure.

Why bother with a hardware based attack at all?
This is a good question. This is a lot of effort given how easy it is to change code on the server’s FPGAs, FLASH, or ROM. The one benefit of a hardware attack is that change is permanent. A hardware-based attack will overwrite any future firmware and BIOS upgrades.

What makes China different, why the concern?
While working on MBA, emphasizing international business, I did a paper on China in the early 2000’s; There were three major points emphasized: lack of intellectual property protections, major companies owned by the military (PLA – Peoples Liberation Army), and requirements for “Chinese partner or joint partner” companies. All of these areas make manufacturing in China less secure than in other countries. I have not had time for an in-depth re-analysis; some preliminary searches indicates many of these same concerns still exist.
• Intellectual property protections – It appears China has made great leaps in protecting its and other countries intellectual property. At one time these protections were nonexistent.
• Major companies owned by the Chinese Military – A 2015 Fortune articleindicates that this has not changed a great deal. While the military no longer runs major companies, the government owns most companies.
• Joint Partners – One of the ways China has improved its domestic companies is by requiring foreign companies to form partnerships with a Chinese company to gain market access. In accepting a joint partner, the foreign company gives up many of its trade secrets. This Money article indicates nothing has not changed since the early 2000’s.

Other possibilities – What if hacker’s intent is just to brick the server?
The intent of the reported hack was to gain access to the servers and collect data. A much easier attack could involve bricking the server entirely. Without delving into why someone would want to brick many servers, here is how to implement this type of attack assuming compromised manufacturing supply chain.
There are certain common signals on the motherboard that can shut the entire motherboard down, keep all the logic from working. One of these signals is the power_good signal. Under normal operation, this signal goes high when all power supplies on the motherboard are operating in their correct range. To brick the server a hardware hack would only need to hold this signal low after meeting a given event condition, like when reaching a certain date. Hack requires a very small device, similar the device pictured in the article, and only needs to connect to a single power_good trace, power, and ground. While this would be a onetime event and not allow for any data collection, it would certainly shut down the server or network device forever.
The point is, once compromised your supply chain can permit limitless hacking possibilities. Therefore, know your supply chain and keep an eye on it.

Summary:
Be aware of your network and server supply chain and rest assured if you use iRis for your private or public network needs, iRis does not used systems manufactured in China.

Initial article by Bloomberg and follow-up:
https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

Other similar articles:
https://www.thenational.ae/business/technology/fresh-evidence-of-china-hacking-hardware-found-in-us-telecom-1.779054

Denied by Apple and Amazon:
https://www.cnet.com/news/apple-tells-congress-its-was-never-hacked-by-chinese-spy-chips/