Fishing Around in the Sea of Hardware

Have you ever thought about how Amazon and the likes manage to take care of all their servers? And no, buying admins Segways to drive around the data center is not the right answer (albeit a cool one). Interested?

Prehistoric creatures of the deep

If you ever talked with a system administrator for more than a few minutes, you almost certainly heard about the IPMI . This specification evolved in the silicon-rich primordial soup a few million years ago.

Joking aside, the specification was brought to life in 1998 by Intel and is supported by more than 200 computer system vendors. And what is the key ingredient to its success? “Out-of-band management capabilities” is probably the answer to that question.

As your mother would recommend, if the computer is misbehaving, the surest way of fixing it is to force it off by holding the power button down for as long as it takes, and then turning it back on. And out-of-band management means exactly this: being able to interact with the system even when the operating system is not cooperating.

IPMI specification and implementations have quite a few security issues (partly due to its age) that can be mostly resolved by not allowing bad people to access the IPMI management port. But still, can we do better?

Various computer system vendors tried to create something better than IPMI. Examples of such systems are Lenovo’s XClarity and Dell’s iDRAC , but unfortunately, those solutions are vendor-specific, which means that when we are implementing some functionality on top of these systems, we must either restrict ourselves to the common functionality that is present in all systems or code around the differences in the systems.

Since none of the solutions suggested is particularly developer-friendly, DMTF decided to create a new, secure and extensible standard called Redfish .

What fishy thing are you talking about?

As already stated before, the Redfish standard is a modern replacement for the IPMI that was first released in August 2015 with the main goal of delivering simple and secure management for converged, hybrid IT and the Software Defined Data Center (SDDC) in both human readable and machine capable form.

What?!? Let us put it this way: Redfish defines a RESTful api that we can use to view the resources available to us and make them do stuff. For example, if we send GET request to /redfish/v1 on Redfish service, we will get back something similar to this (example is taken from the Lenovo SR650 server output):

{
  "@odata.context": "/redfish/v1/$metadata#ServiceRoot.ServiceRoot",
  "@odata.etag": "W/\"bb7f4494b922dde991a940cc8251e8fc\"",
  "@odata.id": "/redfish/v1",
  "@odata.type": "#ServiceRoot.v1_2_0.ServiceRoot",
  "AccountService": {
    "@odata.id": "/redfish/v1/AccountService/"
  },
  "Chassis": {
    "@odata.id": "/redfish/v1/Chassis/"
  },
  "Description": "Service root for a Redfish implementation.",
  "EventService": {
    "@odata.id": "/redfish/v1/EventService/"
  },
  "Id": "RootService",
  "JsonSchemas": {
    "@odata.id": "/redfish/v1/JsonSchemas/"
  },
  "Links": {
    "Sessions": {
      "@odata.id": "/redfish/v1/SessionService/Sessions/"
    }
  },
  "Managers": {
    "@odata.id": "/redfish/v1/Managers/"
  },
  "Name": "Root Service",
  "Oem": {
    "Lenovo": {
      "FirmwareServices": {
        "@odata.id": "/redfish/v1/Oem/Lenovo/FirmwareServices/"
      }
    }
  },
  "RedfishVersion": "1.0.2",
  "Registries": {
    "@odata.id": "/redfish/v1/Registries/"
  },
  "SessionService": {
    "@odata.id": "/redfish/v1/SessionService/"
  },
  "Systems": {
    "@odata.id": "/redfish/v1/Systems/"
  },
  "Tasks": {
    "@odata.id": "/redfish/v1/Tasks/"
  },
  "UUID": "REMOVED_FROM_MOCK",
  "UpdateService": {
    "@odata.id": "/redfish/v1/UpdateService/"
  }
}

This is the only resource path that we actually need to know in advance, because all other resources can be reached by following the links in the values, associated with @odata.id keys. And yes, Redfish is one of the few RESTful APIs that respect HATEOAS constraints.

Swim fishy, swim

To get an idea how things work, we will reboot one of the systems using Redfish. First, we need to locate the system that we would like to reboot. We do this by sending a GET requests to /redfish/v1/Systems/ (this is the path to the Systems collection, listed in the previous JSON dump of the root Redfish resource). And this is what we get back:

{
  "@odata.context": "/redfish/v1/$metadata#ComputerSystemCollection.ComputerSystemCollection",
  "@odata.etag": "W/\"e48557da1bf040a5d45d1e5aa726bf3a\"",
  "@odata.id": "/redfish/v1/Systems/",
  "@odata.type": "#ComputerSystemCollection.ComputerSystemCollection",
  "Description": "A Collection of ComputerSystem resource instances.",
  "Members": [
    {
      "@odata.id": "/redfish/v1/Systems/1/"
    }
  ],
  "[email protected]": 1,
  "[email protected]": "/redfish/v1/Systems/Members",
  "Name": "ComputerSystemCollection"
}

So, we only have one system that we can reach at /redfish/v1/Systems/1/. I guess we will reboot this one then;) But before we can do that, we need to get the details of this system. Somewhere in the response of the GET /redfish/v1/Systems/1/ call we can find this thing that we are after:

{
   ...
  "Actions": {
    "#ComputerSystem.Reset": {
      "[email protected]": [
        "On",
        "Nmi",
        "GracefulShutdown",
        "GracefulRestart",
        "ForceOn",
        "ForceOff",
        "ForceRestart"
      ],
      "target": "/redfish/v1/Systems/1/Actions/ComputerSystem.Reset",
      "title": "Reset"
    },
    ...
  },
  ...
}

Because we have no time for politeness, we will restart the server by force (push the power button down). And we do this by sending POST request to the /redfish/v1/Systems/1/Actions/ComputerSystem.Reset endpoint with the following payload:

{
  "ResetType": "ForceRestart"
}

Time to congratulate ourselves for the job well done and for possibly ruining someone’s day by forcibly rebooting the server in the middle important git server update .

But where do those red fishes live?

If you are lucky enough to have access to a fairly new server from any major vendor, chances are that there is at least partial support for Redfish API present. And the simplest way of checking this is to plug the cable into management port and navigate to server.addres/redfish/v1. If the JSON comes back, you are good.

On the other hand, if ruining other people’s days is not your favourite activity, you may be interested in playing with the readonly service recording that we made available on GitHub .

Off we go

And this is the end, since we ran out of fish-related puns. Join us next time when we will try to fry some fishes… (Get it? Fry the fishes? Bah, whatever …). And if you cannot wait for the next installment of this, you can talk to use on twitter or Reddit in the meantime.

Edit (2018-07-27): Next installment is finally here .