Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I am very eager to study best practices when it comes to space hardening. For instance, I've read (though I can't find the article any longer) that some core parts of the Mars rovers did not use dynamic memory allocation, in fact it was forbidden. I've also read that old fashioned core memory may be preferable in space.

I was looking at some of the projects associated with the Google Lunar Challenge and wondering what it would feel like to get code on the moon, or even just into space. I know that space hardened boards offer some sanity in such a harsh environment, however I'm wondering (as a C programmer) how I would need to adjust my thinking and code if I was writing something that would run in space?

I think the next few years might show more growth in private space companies, I'd really like to at least be somewhat knowledgeable regarding best practices.

What happens to a program if radiation, cold or heat bombards a board that sustained damage to its insulation? I think the goal is keeping humans inside of a space craft (as far as fixing or swapping stuff) and avoiding missions to fix things.

Furthermore, if the board maintains some critical system, early warnings seem paramount.

How does one gain experience in this through testing and trial & error (barring the launch of your own personal satellite?)

share|improve this question

migrated from stackoverflow.com Jun 9 '11 at 2:30

This question came from our site for professional and enthusiast programmers.

3  
I've sent e-mails to space-x and others asking them to join SO and help answer this. If anyone knows anyone at NASA, now is the time to e-mail them. Likewise, maybe you know a retired egineer? I'm not going to close this any time soon. –  Tim Post Feb 25 '09 at 10:32
6  
Worth noting that the "forbidden dynamic memory allocation" isn't unique to space probes, but in fact fairly common for any tightly constrained embedded hardware (even handheld video games). –  Crashworks Feb 28 '09 at 5:13
    
en.wikipedia.org/wiki/NOP_slide –  rwong Nov 5 '10 at 13:28
    
@Mark, is humor now enough to get answers deleted? –  user1249 Jun 9 '11 at 10:07
2  
It can't be that hard, it isn't rocket science. Oh wait... –  Mark Ransom Jun 9 '11 at 21:41

11 Answers 11

up vote 45 down vote accepted

Space software is not arcane magic. You are still using 0's and 1's, not 1's and 3's. So there’s probably no wow factor involved in describing what goes into developing software.

Some slight differences that come to mind at the moment are:

  • Extremely process oriented.
  • Space software will always have both software and hardware watchdog timers.
  • Every space system I’ve worked on was a hard real-time system.
  • You simulate (to great accuracy) every external actor to the system. This usually involves building (sometimes really expensive) custom hardware that is used solely for testing.
  • You spend enormous effort and expense doing formal testing.
  • The customer (usually JPL) is extremely involved in the test process.
  • You generally are using old and known compilers and development environments, rather than the new ones.
  • Code reviews, code reviews and code reviews.
  • You better be very comfortable switching between the hardware and software worlds. You don’t have to know how to design the hardware but you have to know how it works.
  • Extensive use of test equipment, like oscilloscopes, logic analyzers, synthesizers and spectrum analyzers.
  • At least 3 locations for storing the application program. The default is burned in ROM. This will never change. The other 2 are for the current version and the next/last version.
  • Failure analysis (MTBF) is really important.
  • Redundant systems and failover plans for the critical components.
share|improve this answer
    
Up to now, but wait until the memristor was coming! –  lsalamon Apr 30 '09 at 14:14
    
You say code reviews three times like they are a negative. –  Kortuk Mar 4 '11 at 11:14
3  
@Kortuk: That was to emphasize that you will be doing code reviews WAY more often than most other types of projects since the consequence of a failure is only the loss of a several hundred million dollar satellite. And personally, I believe they are definately a negative but necessary evil. I hate holding the reviews and I hate performing the reviews but they have their value as they find problems as no other method can. –  Dunk Mar 17 '11 at 21:19
    
100% agreed. Necessary evil is an acceptable description. –  Kortuk Mar 17 '11 at 23:56
7  
"Space software is not arcane magic" however, if it is sufficiently advanced space software it would be indistinguishable from arcane magic. –  Robert Jun 9 '11 at 3:10

I just stumbled into your interesting question.

I was at Instrumentation Lab during Apollo, and again later when it was called Draper Lab during the "cold war".

For the Apollo guidance computer, core was used for RAM, and a special braided core was used for ROM. The machine itself was made entirely out of NOR gates and was clocked quite slow, for reliability.

I didn't work directly on Minuteman missiles, but I was aware of some of the issues. If a nuclear warhead goes off in the vicinity of some electronics, it basically shorts it out. The guidance computer had a radiation sensor that would instantly shut off Vc so nothing would burn out. Then the computer would restart, having had its registers erased.

To handle this, the computer would periodically snapshot its registers into core, and when it restarted it would start up from that checkpoint. To make this work, the software (all in ASM) had to be analyzed to see that it could take any number of such hits, at any frequency, without getting wrong answers. That was called being "restart protected". Very interesting problem, given that (thank goodness) it never had to be used.

share|improve this answer

To get tough environment reliability specifically in C, here are some really concrete things that I've seen done.

MISRA-C : The Automotive C subset. A bit like Ravenscar ADA/Java.

watchdogs : ensure the program doesn't lock up

ecc memory (sometimes)

checksums: looking for flipping bits. I've seen all three of these in one system:

1) checksum the program continuously (it was in EPROM but still got flipped bits).

2) checksum certain data structures periodically.

3) CPU sanity checks periodically.

4) check IO registers have what they are supposed to have in them.

4b) read back outputs onto independent inputs and verify.

share|improve this answer
    
And, have all the failure responses thoroughly planned, on the conviction that they will be needed. –  Mike Dunlavey Mar 25 '09 at 22:18
    
Failure responses are best put in the code. The error occurs at a time of it's choosing. Needs to report faults, epecially when recovered from. The machine has to cope for itself, up until to the point where the "computer fail" annunciator goes off. –  Tim Williscroft Mar 25 '09 at 23:54

Far more important than the programming language are the requirements on the underlying system (OS and Hardware). Basically, you need to ensure (and prove) deterministic and predictable behavior of the overall system. Much related research has been done in the real-time community. I strongly recommend reading two books if you really want to study this subject: Real-Time Systems by Jane Liu and a book with the same name by Hermann Kopetz. The former covers scheduling in a very theoretical fashion while the latter gets your feet back on the ground and pretty much covers all related aspects of (real-time) system design, e.g. fault tolerance.

Furthermore, the following two incidents nicely illustrate the quality of problems software engineers have to face when sending something into space:

share|improve this answer
    
Mars Polar Lander. ( inadequate testing) –  Tim Williscroft Jun 9 '11 at 4:44
1  
Mars climate orbiter: units confusion. Just use SI and be done with it. –  Tim Williscroft Jun 9 '11 at 4:46

Space-proof computing systems are all about dependability. A deep introduction to the field can be found in Fundamental concepts of dependability by Algirdas Avižienis, Jean-Claude Laprie & Brian Randell.

Real-time is also a key concept for space computing. As Pankrat, I would recommend Real-Time Systems by Hermann Kopetz.

For giving a pragmatic sense of the challenges of space computing, think of:

  • extreme conditions in space: very hot when oriented to the sun, very cold the other side, many cosmic rays which may invert bits in memory, huge accelerations and vibrations when being lauched, ... Hardware for space must be far more robust than hardware for military.

  • When a failure occurs, except in the International Space Station or for Hubble Space Telescope, nobody comes and replace failed system. Everything must be fixed from ground through maximal observability and commandability and through spare systems to switch to. This is easy for Earth satellites. This is more difficult with space probes for which communication delays may be one hour long. Indeed, everything must be as reliable as possible in the first place.

share|improve this answer

I found this document (circa 2009) for the JPL Institutional Coding Standard for the C Programming Language on the Laboratory for Reliable Software (LaRS) at the Jet Propulsion Laboratory site.

Here is a summary of the rules documented:

  1. Language Compliance

    • Do not stray outside the language definition.
    • Compile with all warnings enabled; use static source code analyzers.
  2. Predictable Execution

    • Use verifiable loop bounds for all loops meant to be terminating.
    • Do not use direct or indirect recursion.
    • Do not use dynamic memory allocation after task initialization.
    • * Use IPC messages for task communication.
    • Do not use task delays for task synchronization.
    • * Explicitly transfer write-permission (ownership) for shared data objects.
    • Place restrictions on the use of semaphores and locks.
    • Use memory protection, safety margins, barrier patterns.
    • Do not use goto, setjmp or longjmp.
    • Do not use selective value assignments to elements of an enum list.
  3. Defensive Coding

    • Declare data objects at smallest possible level of scope.
    • Check the return value of non-void functions, or explicitly cast to (void).
    • Check the validity of values passed to functions.
    • Use static and dynamic assertions as sanity checks.
    • * Use U32, I16, etc instead of predefined C data types such as int, short, etc.
    • Make the order of evaluation in compound expressions explicit.
    • Do not use expressions with side effects.
  4. Code Clarity

    • Make only very limited use of the C pre-processor.
    • Do not define macros within a function or a block.
    • Do not undefine or redefine macros.
    • Place #else, #elif, and #endif in the same file as the matching #if or #ifdef.
    • * Place no more than one statement or declaration per line of text.
    • * Use short functions with a limited number of parameters.
    • * Use no more than two levels of indirection per declaration.
    • * Use no more than two levels of dereferencing per object reference.
    • * Do not hide dereference operations inside macros or typedefs.
    • * Do not use non-constant function pointers.
    • Do not cast function pointers into other types.
    • Do not place code or declarations before an #include directive.

*) All rules are shall rules, except those marked with an asterisk.

share|improve this answer

What I learned from the one project I was involved in as an intern:

Your hardware specs will change, usually for the worse!

For example, the space hardened CPU that was being used in the design was promised, promised, mind you, that it would run at 20 MHz.

The final result ran at 12 MHz. The senior programmer on the project spent a lot of time redesigning algorithms in order to meet the hard real time requirements of the control systems and much of the telemetry software ended up offloaded to a second system instead of running on the primary CPU.

So, try to leave some extra resources available in the original design and try not to use all the available CPU and memory.

share|improve this answer

For a software perspective, write a privileged task that occasionally, randomly, flips bits in your code, and see how it deals with that. That's your simulator.

Hardware-wise, the parts will be old, because it takes a long time to get something to be space-rated. Also, new parts are continually shrinking in size, and the smaller a feature is (think memory cell on an I.C.) the more susceptible it is to corruption from a radiation event.

share|improve this answer

I worked on a safety critical device and we had to go through some similar hoops.

We had safety critical variables. There was a copy of the inverse of the variable. After each loop, the variable was checked against its inverse.

We had a walking ones and zeros test of ALL registers. That included the Program Counter!

We had a test of all the opcodes of the micro instruction set. We had to be sure that if you added 2 registers, the registers were added.

Some of this is probably not related to programs in space, but it gives you a sense of the magnitude of checking that is possible.

share|improve this answer

I believe the worse an environment is the more Error Correcting Codes is used, and there is ECC memories that can be used to some extent.

If one can estimate the level of errors one can construct a Error correcting code that can handle the introduced errors.

share|improve this answer
  • Yes, core memory is on the research boards.
  • Dynamic memory is not good for embedded systems. Reliability issues.

I would guess that software ECC of data and using information theory and a custom loder to spread the data around the system to manage the memory failures would be a start. But, I don't study rad-hard software so I'm not familiar with it, that's just a guess.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.