Apollo Comms Part 14: Sending Commands to the Spacecraft
This episode has been made possible thanks to the sponsorships of PCBWay, ElectroRent and Keysight Technologies. Greeting earthlings. If you follow the channel, you know that we love all things Apollo, and that during our last visit to Steve Jurvetson’s amazing space collection, we were given the opportunity to take two holy boxes of Apollo electronics to our lab. These are the boxes that brought you voice, data and live TV from the moon, and should be early masterpieces of microwave electronics, the blackest of black arts in analog electronics. Our spacecraft transponder is now fully up and running, and we have turned our attention to the ground part of the link. In episode 12, we succeeded in recreating the ground transmission infrastructure with modern Keysight equipment, gracioulsy loaned to us by ElectroRent.
And a bit earlier, in episode 9, we had worked on powering up a completely undocumented piece of Apollo ground test equipment: the Up-data Link Confidence Test Set. We returned it to partial life, thanks to Ken Shirriff’s incredible reverse engineering. This test box should enable us to send commands to the spacecraft from the ground.
But although we eventually succeeded in making it read in a tape after a lot of effort, we still don’t understand it completely. And worse, it is missing some important parts. In this episode, we’ll attempt to complete the bring up of the box, and see if we can temporarily replace the missing parts with more HP equipment. We hope we can then tie it to our Keysight ground installation. And since this all happens at audio frequencies, we might even be able to hear what these commands sound like. Hey Mr. Ken, can you explain what you're doing? [Ken] So my latest project is, I'm trying to
figure out how this box is wired up. It has 25 boards that plug in, and then a wire-wrapped to back plane that's just a tangle of wiring. So it's hard to figure out what is connected to what. [Marc] All the cables are the same color. And they all leak into each other. So we tried with a wire tone tester, and that didn't do anything. [Ken] And we tried the current tracer as well, that didn't work. [Marc] Yeah. [Ken] The multimeter, I could get a lot of the wiring,
but it's very, very slow. And then down here, you can't even access the pins, there's so much wiring. [Marc] Yeah, and the pins are into the connectors, it's really hard to get to them. [Ken] So I made an automated system to help me figure it out. What I do is,
I have two boards here. I plug them into two slots, and then I run through all the pins, activating each pin in sequence, and then reading to see what it's connected to. And so, that gives me the wiring between these two slots. And so, then I put the two boards into every pair of slots. So that's 300 pairs altogether. And then it will give me the entire wiring for the the box.
[Marc] And we wanted to give our traditional shout-out for our sponsor PCBWay. They make all of our boards, from the rather simple one here, to the very complicated, and they have been our sponsor since the channel was minuscule, so we thank them very much. [Ken] These boards are very simple. There's three 16-bit general purpose I/O chips to drive the pins here, either reading or writing. And it's controlled over I2C, a simple serial protocol. It's not really Ethernet. I'm just using Ethernet cables because they're convenient, I had them around. And it's driven by the Teensy here,
an Arduino-like microcontroller, that just sends serial signals to the two boards to activate each pin and read the results. And then the results are sent to my laptop. And I record it in a data file, where I can then apply graph algorithms to figure out what the nets are, what is connected to what. So I'll plug the pair of boards into slots 17 and 18 here. The boards pop in. And I tell the program that I'm using slot 17 and 18. And there, it scanned all the approximately 5000 potential wire pairs between those two boards in under a second.
So now I've moved it over to slot 19. I run the test on slot 19. [Marc] Already done, eh? [Ken] Yeah, it's under a second, it's very fast! I put it in slot 20. So, inconveniently, half the slots are upside down, so I have to keep flipping the board over. [Marc] Really? [Ken] In the actual modules they have two boards sandwiched together. [Marc] Oh, I see! [Ken] So they're
inverted relative to each other. [Marc] So it's pairs of connectors. [Ken] Yeah. And so I just keep going through all 300 pairs of slots. [Marc] You have made a reverse engineering machine! [Much later...]
So we have everybody: we have Ken, Mike and Eric. And we are all admiring our up data... No. Yes. Uplink? Updata box? What is it? [Ken] Updata test box. [Marc] Updata test box, which, Ken, you have sort of almost finished your reverse engineering of it? [Ken] So I've spent way too much time reverse engineering this. And basically, what it does is, it reads a message off paper tape, loads it into the digits here. And then it sends the message, encoded using phase shift keying to
the real updata box. Which sends messages then to other parts of the spacecraft such as the Apollo Guidance Computer. This box then reads reads back in the message from the external boxes, and verifies that it came through the whole system correctly. [Marc] Last time, we just had this box barely working, and then you made a contraption to read the bus, so you could get all the connections on the bus? [Ken] Yeah, so I made some boards that would probe the backplane, which is a massive point-to-point wiring. So I could figure out what boards, how they're connected, and try to piece together what they're doing. [Marc] And you actually did! [Ken] So I made this diagram. So each box represents one of the
boards here. So, the big picture is, message comes in from the paper tape reader, gets loaded into the 13 digits here. These digits are set up as a shift register, so it can then rotate through all the digits, send the digits to other parts of the system. And they get sent out externally. And then, the inputs come in, and the inputs get compared to what the box thinks they should be.
And then the lights will light up with with red or green, whatever worked or not. [Marc] Let's load one tape on the indicators. I have one ready. 'Program Start', that should work. [Eric] Is there another code for the end of the tape, or is it just that bit to start? [Marc] No, there's just that bit for the start I could find. There's another control bit that makes you ignore the stuff and that's probably so you can...
Oh, hop, hop, hop! 'Single'! All right, there you go. So, it was reading commands and sending them. But we loaded zero, one, two, three, four, five, six, seven - no eight - zero, one, two three, four, because you run out of numbers. So the only other control bit is to ignore the input. It will just skip right over it. [Ken] If you mistype something, you can just
cancel that that and keep going without having to redo your whole tape. [Ken] That makes sense, if they were punching these tapes by hand, you know, it would have been a pretty error prone process. [Marc] Yeah. [Ken] Marc, was a pretty error-prone? [Marc] Yes! But we're doing pretty good here. We have no tape errors, so we got that to read properly. [Ken] So we read 13 digits off the tape. The first three are a message number which is not sent. It's only used for knowing what test you're doing. The next digit is the vehicle address. So, you can send a
message to the command module, the lunar module, and make sure that the right spacecraft gets the message. The next digit is what system on the spacecraft you want to control. And then the next is the message which depends on the system you're sending it to. The message can turn a relay on or off, it can send a command to the Apollo Guidance Computer, it can update the clock on the spacecraft. So, these last digits are the ones where it actually does something interesting. [Marc] Most of our knowledge on how these commands work come from this drawing from the CSM 114 Command Module Systems Handbook, which happens to be the command module for Apollo 17. It’s a quite complicated drawing, so let’s spend some time on it with some elevator music. Let’s start at the right side of the drawing, which shows the commands structure.
There are four of commands: CTE commands, for the Central Timing Equipment CMC commands, for the Apollo Guidance Computer Test messages, for checking bits and RTC commands, for Real Time Control The CTE is mostly a clock. There is a control line for reset, and one line each for the seconds, minutes, hours and days. It’s very simple. Each time you put a pulse on the line, you increment the corresponding digit.
Let’s take a look at at CTE command bits in detail. The first 3 bits are called VA, for the Vehicle Address. For the command module, it is always hardwired to 4. The second digit, also 3 bits,
is the System Address, of which only 4 are used. For CTE, it’s hard wired to 4. So a CTE command starts by 44. The next values are in bunches of 6 bits each. They are the seconds, minutes, hours and days. Basically, after receiving this command, the hardware will reset the CTE to zero, and pulse each set line to make them advance. So, that’s how you reset the clock.
Let’s look at the CMC messages, which is the name of the AGC in the command module. The AGC is connected to the message system via only two lines, one for sending ones, and one for sending zeroes. But note that there are two switches in the way. They both say block or accept. One is on the main command panel.
And the other one is in the lower bay, next to the star alignment equipment. These are marked block and accept, and unless the astronauts put both switches on accept, access to the computer from the ground is not allowed. So, you can see, they were really worried about possible interference, in particular from Russian interference. So, before each access to the computer, the ground would have to call the astronauts and ask them to flip the switches, and you can actually hear that on the loop. Let’s see how the computer commands are formed. A computer command always begins with 43.
Then, 5 command bits are transmitted 3 times: in normal form, in bit inverted form, and in normal form again. This is obviously for error checking. The 5 bits in question represent a single DSKY key press code. So basically, the ground is simply remotely pressing on the keys of the DSKY, one at a time. Although very rudimentary, this is deceptively powerful. The ground can do everything the astronauts can do with the AGC, which is pretty much flying the whole spacecraft. You can reorient it, fire the engine, load burn data, check your trajectory.
The ground can also read the computer results via the much faster telemetry downlink. For example, they could do a full memory core dump if they wanted to. Let’s now look at the test message system, which allows to test each bit sent to the system. The test results are once again sent via the telemetry downlink. One test message sends zero bits, and the other sends one bits. You can then rotate them and check them one by one. The system gives you the results via the telemetry, and there are only a few messages: - standby, if nothing is decoded - validity, if any error-free message is received - Results of the check of either the ones and zeroes and off if system is off.
Last but not least are the Real Time Control messages, which flip a bunch of relays. Here are the coils of the relays controlled by this message. An RTC command begins with 46. The rest of the bits indicates which relay coil is to be energized. There are also reset messages that flip a whole bunch of relays at a time, called salvo reset. They start with 45. They are used to put the spacecraft back into a default configuration.
The RTC messages are actually spelled out over here. We can see for example that 4600 will turn relay 00 off, which will turn the Abort Light A off. 4601 will turn the abort light on. And here are the salvo resets. Salvo message 4530 followed by
4532 will flip this whole bunch of relays. Now let’s see how these relay contacts are implemented. Here are the contacts connected to coils 42, 43, 46, and 47. You can see
that they control our S-Band amplifier: Off, Low Power, High Power, or Reset. But wait, didn’t the astronauts also have a switch that did the same thing? And yes, they did! Here is the astronaut switch that does the same thing! So, who wins? The ground or the astronauts? Well, if you follow the wiring, it’s mostly the ground. You can see that the astronaut switch only does what it says on it, if the RTC relays are in the Reset position. Any other RTC command on these relays, and the ground wins,
it does not matter which position the astronauts have set their own switch. I have not looked at all the wiring, some get quite complicated, like this one, which is the SBAND AUX control but it appears that ground wins most of the time. So you can see how the up-data commands were done. You get all you need with relatively unsophisticated electronics and a few simple messages. The simple things are done directly via relays,
and the complicated ones are done via remote pressing the keys on the AGC. This whole Apollo thing is really about cleverly managing complexity. And, so, we actually got enough of it to work, that we can do the rest with some crutches. So, we got the bits coming out, the bits and the sub-bits. Actually, we have to explain this. Oh, that's tough. Do you want to explain the bits and the sub bits? [Ken] I'd be delighted to! [laughter] So, the message, of course, is sent in binary. So each octal digit turns into three bits.
To make this reliable, they use something they call sub-bits, where each bit is expanded into five bits, which are the sub-bits. So a zero turns into one sequence of five bits, a one turns into another sequence of five bits. And if any of those bits are corrupted, you can detect an error. So it makes it much less error-prone, because you want to have reliability for the spacecraft. They also use a different sub-bit encoding for the vehicle address, so they can tell if it's the start of a message, versus if they pick up in the middle of a message.
And so, that is another way they get reliability. [Marc] For the for the framing, actually. They need a framing mark. So they need a code that only appears at the beginning of the frame, that's different from everything else. [Ken] And that's done by using the different sub-bit encoding. [Marc] That's one of the methods to do it. [Ken] And then for testing, they have the sub-bit spoil switch, which will spoil a few of the sub-bits. So you can test if the error checking is working correctly. [Marc] Just to annoy.
But we found that the box was generating the sub-bits, I'm hooked up to them. Do you want to hit one of the - hold on a second - hit one of the tapes? [Ken] Okay. [Marc] Okay, so it just sent that. [Ken] So, in yellow, you can see the the bit pattern it's sending. And then in green is the sub-bits. So each yellow bit has been expanded into five green sub-bits. So for instance, here you can see the pattern for a one sub-bits, and then the zero turns into a different pattern of sub-bits, the complement. [Marc] You want to
load another tape that's just ones or zeroes, so we can see the beginning of the sub-bits? There you go. We have the twos. Okay, so here we have a bunch of twos, for example. And I was going to move at the beginning of the sequence, and it's different. This whole beginning of the sequence here is encoded completely differently. [Ken] So you can see that at the beginning, the zero turns into a one-one-zero-one-one sub-bit sequence, which is entirely different from the zero sequence you get here. So that they can tell where the beginning of the of the message is. [Marc] Okay. [Ken] And if we zoom in, we can see the PSK signal. [Marc] Oh, no! Now we have our encoded messages, then our bits divided into sub-bits, but that’s still not what we need to transmit. What is transmitted is actually PSK, or
Phase Shift Keyed signals. I explained it in video 12, when we made our modulation infrastructure. So let’s restart the elevator music for a short review, because this is not simple. So we have our message bits. We break them down into sub-bits. We then take a 2 kHz carrier,
and use PSK to modulate the sub-bits onto it. This gives the weird waveform where the phase abruptly shifts by 180 degrees when there is a bit transition. And to ease the decoding, we add a phase synchronous reference signal at 1 kHz. Which gives that contorted modulation signal. Our Updata box is supposed to do all that, except it does not, because we are missing part of these circuits. So I found a roundabout way to recreate it with HP / Keysight equipment. And I was quite surprised and relieved when I got the exact same contorted signal on the scope.
So let me explain for a second how I do that. So, we are missing parts, and so we don't have the PSK generating stuff working quite yet. So I pick up the green bit which is the sub-bit, and that goes over to my Keysight equipment. And that one has a two kilohertz, on which we do the PSK, and a one kilohertz, which is just a reference tone. And there is one more crutch to that box, it's back there. We use some friendly 1960s equipment, because this box is missing is its 4 kilohertz clock. [Ken] Here you can
see the 4 kilohertz. [Marc] And it's not any 4 kilohertz, it's 4 kilohertz that's phase-locked to the 2 kilohertz of the Keysight. Because you have to phase-shift it at exactly the right time. So those two boxes have to be synchronized. And that's the box I just repaired, and I used the PLL capability of that box. [Eric] Did you do any other phase adjustments between that reference 4 kHz and the 2 kHz? [Marc] Yes, you need to. Actually, you can tell that I'm not perfectly in phase here - I'm pretty close actually! But yes, there is a button that allows you to choose exactly. Not only lock on the right harmonic, but then you choose
which phase you have to be in. [Eric] I see, okay. So normally when this is generating the signal, that 4 kilohertz clock comes in, and that's used to generate the 1 kHz and the 2 kHz, everything else you need. But in this case, you can't really do that. So you've got an external instrument that's generating your 1 kilohertz and your 2 kilohertz. And then to get the 4 kilohertz, you run that into your little 1960s PLL. [Marc] I need the help of some Nixie tube stuff! [Ken] So there's supposed to be a 4 kilohertz oscillator on this board, but somebody took it. [Marc] Right. [Ken] Do you want to put on the sound so people can hear what the message sounds like? [Marc] Yes, let's... So we are actually transmitting! So then, since we had the
Keysight equipment, and it's already transmitting an uplink - right now it's transmitting just idle. So you can see it here. And that sounds like a tone. And then, if you send something. [Ken] Okay! [fax-machine like sounds] [Marc] It just went 'meep'! Okay, do it again. [more fax-machine like sounds] Oh that's the three, okay. [higher pitched tone] [Marc] Beep! [Ken] That sounds a little different for the different messages. [Marc] Right.
[Ken] We need to put a song on here somehow. [Marc] Let me try to send a sequence. [bursts of fax-machine like sounds] So I've made a very special tape of Apollo commands. Let's see if we can... Ah, it's changing as we speak! So there is really one module not connected properly, is my guess. So we have to re-seat all the modules. 'Clear'. Okay, put it on 'Start'.
Zero. 4601: it's a nasty message. It's Abort A Light. So I'm sending Abort A. Abort B light. Crew Alert. And then finally, I want to cool things off. And I send Reset Message 30,
Reset Message 32. And those will reset the ship propulsion and all the alarms to zero. So now, after we've woken up the astronauts, we say: oh, no, never mind! That will reset telecoms: that's a Reset Message 70, Reset Message 72.
That will do relay 22 and relay 27 - see it changed right there - relay 27. And that's to turn ranging on. Test Message A, Test Message B. So, if I redo this one, and if the box doesn't die on us. I'm just going to do the whole - okay, we're at zero. I'm just going to do the whole sequence.
Then we're going to listen to it. Here we go. [mix of tones and fax-like bursts] Yeah, it's hard to hear any difference, they all sound the same. Okay anyhow, that was our first transmission of commands. Now we have to decode them. We actually did get a little bit further, transmitting and recovering sound and data at the same time, showing that there is no crosstalk between the two.
As usual, we’ll update you in the next episode. See you then!
2022-03-26 12:45
