Saturday, December 15, 2018

Bloomberg Business' Hit Piece on Supermicro Mother Boards

Last October, Bloomberg Businessweek ran an article alleging that Chinese contractors to American computer companies were implanting parts on the computer boards that spied on the users.  I didn't run with the story because it honestly sounded improbable to me.  The components were shown to be about the size of chip component but with at least one extra terminal.  Shown here.

It's not the size that I thought was suspicious - that's a pretty common size these days - it was the idea that they could put a chip on there, with one extra pin (assume the other two are power and ground - those are pretty necessary - and yeah, there are some tricks to use pins for more than one thing) and use that to send data somewhere.  Exactly how does this chip get put on the board if the printed wiring board wasn't designed for that part?   There are no solder pads in the design, so the board has to be modified.  Nobody would notice that?  Exactly how does "China" get the data?  Radio?  With something that small and no obvious heat sinking, it's going to be low power, which means it's going to have to be transmitting to someplace close.  Again, how?  What frequency?  What data rates?  If signals are going around on the motherboard at "a few" gigahertz, you'd better be sending that rate if you're trying to capture what the users are doing.  If it's not radio, the questions get even more uncomfortable.  Serial link?  Pretty much has to be in the same building.  Use the computer itself to send the data?  And nobody: no software, no traffic analysis, nothing notices?  

It strained credulity, but the rumor-fest article quoted several people and gave them credibility.  Trade magazine Electronic Design even proposed how it could be done, by attacking the baseboard management controller (BMC) which has complete control of the motherboard.  One of the companies that Bloomberg singled out by name was San Jose-based Supermicro.  From Bloomberg
To help with due diligence, AWS [Amazon Web Services], which was overseeing the prospective acquisition, hired a third-party company to scrutinize Elemental’s security, according to one person familiar with the process. The first pass uncovered troubling issues, prompting AWS to take a closer look at Elemental’s main product: the expensive servers that customers installed in their networks to handle the video compression. These servers were assembled for Elemental by Super Micro Computer Inc., a San Jose-based company (commonly known as Supermicro) that’s also one of the world’s biggest suppliers of server motherboards, the fiberglass-mounted clusters of chips and capacitors that act as the neurons of data centers large and small. In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says.

Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community. Elemental’s servers could be found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers.
In that Bloomberg article, Supermicro, Amazon and others denied that any of this was true.  This week, news was released that Supermicro had hired an independent company to audit their motherboards from China and that contractor had found no such things.
The company was unable to find any evidence supporting the Bloomberg report. The implants, according to Bloomberg, were camouflaged as tiny passive components and were added to the boards during manufacturing. The report said that the malicious chip was designed to give Chinese intelligence agents a secret door into the networks of almost 30 American companies, including government contractors.

Charles Liang, Supermicro’s founder and chief executive officer, said Tuesday: “no government agency has ever informed us that it has found malicious hardware on our products; no customer has ever informed us that it found malicious hardware on our products; and we have never seen any evidence of malicious hardware.” He added: “Today’s announcement should lay to rest the unwarranted accusations.”
This is a tough question of "who do you believe?"  The only people who are likely to know the truth are probably in Three Letter Agencies somewhere, forbidden to tell the story.  Amazon and Apple denied the Bloomberg allegations about finding this malicious hardware.  The thing is, it's kind of in the best interests of Amazon, Apple and Supermicro to cover up anything found.  If customers don't have confidence in their products, sales collapse.  On the other hand, it's kind of the in the best interest of Bloomberg Businessweek to spread stories like this.  Increase sales of their publication.  Perhaps affect the stock price of the companies they've named so owners or subscribers could get a good deal?  It seems everyone has an incentive to tell less than "the truth, the whole truth, and nothing but the truth".

Bloomberg's diagram showing how the attack worked.


  1. First fact to remember....there are NO "Free" personally owned Chinese companies. ALL of them are beholden to and in bed with the Chinese communist government. It is simply not possible for a company of any size or importance to exist in China without it being controlled by, supervised by or involved with in some way the government. Next fact. China has gotten where it is today almost exclusively via espionage....of all types, industrial, military and political.
    It is a high order probability that ANY electronic devices coming from China that have even a remote chance of being used for military, financial or similar purposes have been compromised in some way to facilitate spying. Just because one "specialist" can't figure out how a part is compromised does not mean it isn't. Continuing to buy and use ANY parts of ANY importance made OUTSIDE the USA, especially from China or Russia is the pinnacle of stupidity. The Chinese did NOT achieve near parity with the US on their own...and spying via embedded electronic devices is one way they could EASILY obtain data needed to improve their military and economic machine.

    1. This comment has been removed by the author.

    2. But please also note that the "three letter" agencies in the US have their own capability to act in a similar manner. And that every US tech company understands what FedGov can do to them if they are unhappy. Any bets on just how much is going on with US-made hardware? And software as well? Or haven't you heard about "backdoors" in same, and the FBI request for ALL software sold in the US to be so equipped?

  2. It was not a chip sending data that I have thought about. It's a chip receiving a signal, say from a satellite, that cause it to shut down. A kind of kill switch for whatever was being run by the board the chip was installed on.

    Everything from computers to cars to washer/dryers have boards manufactured by the Chinese. If a flyover signal would shut down all the boards, what's the impact? What's the recovery look like? Would we recover?

    1. See, that seems even less likely. We’re not talking about a 3 foot diameter satellite dish, we’re talking a chip about .040 long by .020 wide. Not just an indoor antenna, an indoor antenna in a metal box. To get the antenna performance those outdoor TV dishes get they have to be precisely dimensioned, built and pointed - with nothing blocking the dish. A part that small inside a box isn’t going to receive a signal from space unless the satellite is transmitting far more power than they usually can. Maybe hundreds of thousands of watts. It’s not physically impossible but nobody does that.

  3. I can actually see how that could be done. I've done plenty of high-density PCB design, and if you had a chip that was disguised to look like a passive that was supposed to be there, it would be pretty easy to add a couple laser microvias in pads under the part to make connection to a peripheral bus tied to the board management controller (as proposed by EDN). The communications problem is resolved by the board management controller, as it's set up for remote administration in a server farm. If you implanted the chip and know its address, it wouldn't be that hard to communicate with it. Now, as for what it could do, one it could obviously snoop on the local bus, but since the companies building the boards are in on the hardware mod, they know how the board management controller is configured, and probably load the firmware on board including the BMC as part of system test and config before shipping. I could see having a chip that would swap firmware on the BMC to a compromised version that would allow remote access after some period of uptime, and swap back on command, thus covering its tracks. There would be other ways to utilize it too, so I don't actually find this that far-fetched. I think you can rule out any form of wireless communication, but piggybacking on the built-in remote admin? That doesn't seem tough to do. I don't think slipping it past the customer would be hard either. They will present a certificate of conformance, electrical test and functional test results for the board, and the customer will probably just do a first article inspection and firmware audit. For that matter, deliver the first few hundred boards with the unmodified design and add the changed design later. That would be easy. Dropping one 0402 decoupling cap and replacing it with a chip that looks like a cap but has 2-3 micro BGA pads under it certainly isn't pushing PCB technology, and I'd be shocked if the loss of one cap would materially affect the power network. Also remember, this is a commercial product, not DoD, so it's not going to get the level of incoming inspection most military or avionics parts would.

    If someone gave me the money, I don't think this would be hard to do.

    1. That's actually the most believable thing I've read. I wish Electronic Design had gone down that route.

      Whenever I write something like this, I'm painfully aware of becoming the next guy telling the Wright brothers, "it'll never fly". OTOH, snooping on everything going on in every computer in a large company, government agency, or country makes drinking from the firehose sound like sipping a cocktail glass.

      The only thing I question is (still) what are they getting over that link? PC buses are fast compared to serial data links. It's not that I don't think serial data links "will never fly", but they're not trivial, and it seems like long distance high speed serial links would stick out. There's a lot of specialized infrastructure getting the internet around. It would have to mimic that or take it over.

      Shutting down the computer would be infinitely easier than sending every fetch going on in the computer over a serial link.

    2. If the point of the chip was to just swap to a compromised version of the board management controller firmware that would allow remote admin access, the only thing of note going over the serial bus would be the new firmware load. That also minimizes the complexity of the attack chip - it's just a controller for whatever peripheral bus they use with enough flash for the compromised and legit versions of firmware so it can swap back and forth. The point is to add a backdoor to the remote admin capability on the board, at which point you can do anything you would normally do for remote admin, which probably includes pulling images from the system memory. I'm not particularly knowledgeable on the system admin side, so I'm not sure what's available there, but it seems like a useful attack vector. I'm still in your camp of having no idea whether this really happened, but my last employer was a major DoD contractor and we had the Chinese in our system for months before our IT guys found out about it. My current employer is under constant spear-phishing and other attacks trying to compromise our systems, and I have no doubt a large portion of those are Chinese. They work hard at getting into interesting networks, whether DoD or commercial. This definitely seems like something they would and could do.

  4. I agree with the later ideas here - I doubt there would be a point in adding anything RF to a board like this; I DO assume that firmware changes, or extra sources not visible to normal checks would allow, in some way, for internet communication or takeover of the computer. I think first of a parallel to the old 'Boot on LAN' chips that could be added to many network cards - a non-BIOS device making commands to the computer. While I'm not a electrical engineer or programmer, I know that there are many ways to get access to information on a computer or a network once you have a compromised device inside it. And of course, the easiest way to get a compromised device inside is to sell it to the network owner!