2009-10-30

Opportunity Sol 273 (Spirit Sol 294)

I'm not on shift thisol. But when I'm on shift, I can't get any other work done -- not even reading my email, usually -- and now I've got at least two weeks' worth of stuff demanding my immediate attention. So I come in today just to catch up.

Frankly, I don't really want to know how the drive went. If yestersol (272) didn't work, there's no reason to think thisol has a chance. Maybe we'll get lucky and at least make enough distance on the last leg of the drive to be sure we're safe to deploy the IDD, so they can do the magnet observations they had to cancel last week. Still, I don't want to know. I avoid the sequencing rooms altogether.

But I can't escape Jeff's email:


I see we're back on the paving stones - should be smooth
roving from here. Nice driving!


I stare at this for nearly a minute, not really believing it. Then I jump out of my seat and literally run upstairs to the fifth floor. Cooper and Ashitey are wearing the fancy 3-D goggles, looking at the front HAZCAM images. The front HAZCAM shows rocky stuff, not sand.

"We made it," Cooper says. "Great driving!" ("It was a beautiful drive," echoes Art, "and I hate you for it." Art wants out of the crater, and this success means we'll be staying a while. The traverse to Burns Cliff is back on.)

"Did you see that NAVCAM?" Ashitey asks.

One of the NAVCAM images shows where we started yestersol. It's worse than we thought -- at least three of the wheels were dug in, not just the left front wheel. In 3-D, it's a dramatic picture. It's hard to believe we got out.

It's even harder to believe we got all the way uphill. Much as I want to, I can't really take credit for the success. The imagery confirms our suspicions -- there was just a thick band of impassable soft sand, and we spent a couple of weeks getting repeatedly caught up in it. (Where the sand is deep enough to form an opaque layer -- common in the crater -- you can't tell how deep it is until you drive across it. Maybe the sandy layer is a mere 1cm deep, so the rover won't slip at all. Yestersol we got caught in stuff that was more than 25cm deep. It's like crossing a swamp, where at any minute you might plunge in right up to your neck.)

Sol 270's experimental drive worked, right up to the point where we got caught up in the stuff again, which explains why we made no progress on the following sol. Yestersol we happened to get across a shallower layer of sand and onto friendlier terrain -- apparently early on, too, so that the rest of the drive worked and we're way uphill now.

So what changed was maybe 10% a matter of skill (if that) and 90% a matter of what happened to be under the wheels. I can't take credit, but I can sure be happy about the outcome, and you'd better believe I do that for all I'm worth. It takes sustained conscious effort to wipe the dumb grin off my face.

It's still a long way to Burns Cliff. The promised (threatened?) traversability meeting -- which grows every time we hold it, incidentally -- works out a plan for driving there. Optimistically, it might take another two weeks; more likely, longer. There's still a "Bermuda Triangle" of sand along the way, which we're not sure we can pass. The slopes look worrisome.

But at least we're back in the running.

[Next post: sol 297 (Opportunity sol 276), November 2 (at nearly midnight).]





Courtesy NASA/JPL-Caltech. Holy smoke! It worked! ... Uh, I mean, I never had any doubts.


Courtesy NASA/JPL-Caltech. Well, that explains a lot. Two weeks of struggling, encapsulated in a single NAVCAM image -- with Wopmay lurking in the background, and the crater floor beyond.

2009-10-29

Opportunity Sol 272 (Spirit Sol 293)

Because of all the time we've been devoting to driving, several of the instruments have gotten no new data in the last several sols. Flash has been full for a related reason -- in order to gain as much ground as possible, we haven't been turning to optimize downlink, so most of our downlink passes have been unusually poor.

Not that there's much for the instrument teams to do anyway. We've been in essentially the same spot for a couple of weeks now; they've taken pictures of everything there is to take pictures of and already IDDd the hell out of Wopmay, which was the only interesting thing in reach.

Two weeks -- that's how long we've been struggling to get away from Wopmay and on our way to Burns Cliff. Happily, yestersol's successful experiment made us hopeful that we're on our way at last.

We won't know for a while, though; we're in that phase where the downlink is coming in later and later, so our sequencing starts later and later. As we've been doing, we plan the sol in the blind, assuming a successful drive. We're going to do the same thing thisol whether the drive worked or not: we're going to spend a long time driving (or trying to), and fill up the rest of the time with bits and pieces of random science.

For the past few sols, I've been hearing more and more about the possibility of a return to working weekends, at least on Opportunity. Jim Erickson, at least, is getting worried about Opportunity's apparent lack of progress. He seems to be concerned that when they went to Congress and asked for an extended-extended mission, they had this long list of things Opportunity might do, and so far they've barely done any of it. This is likely why Jim's anxious to flee the crater and get on with other things. (Squyres's rejoinder to this is that the list was explicitly a list of things we might do, and not having checked off many items isn't necessarily a problem when we go back for more funding early next year. I don't think Jim's convinced.) Working one weekend day each week would allow for the possibility of an extra drive sol each week.

Both Andy and Jim sound me out about the possibility of working weekends. I'm cool with it -- actually, I even kind of miss it. ("You're all too eager to work weekends," Andy tells me later, his reproachfulness only semi-joking. Apparently, he even gets lots of volunteers to work Thanksgiving. It's fantastic to work with a team that has such high morale.) Jim's blown away when Mark Maimone and I both tell him we miss Mars time.

"I've never slept so much as when we were on Mars time," Mark shrugs as I nod agreement. "And I don't have kids, so ...."

"Don't you have a wife?" Jim asks Mark.

"A girlfriend."

Jim pauses. "And you want to keep her?"

Not long after, we get the downlink, and -- well -- it's not all that we'd hoped for. Not only didn't we make any progress, we actually lost a little bit of ground, ending up maybe 30cm downslope of where we started, and only about 70cm cross-slope. Looks like we don't have a reliable strategy for making uphill progress after all. Or, as Art puts it: "So our euphoria from the previous day's drive has evaporated."

Looks like I was wrong about whether we should continue trying to drive. And that's the general feeling (I mean, not that I was wrong, but that we've about reached the point where we should give up). Since we've already made a plan for tomorrow that includes a drive, we decide we'll give it one last chance. But we're going to meet again tomorrow, after the drive results are in, and decide what to do. And it's hard to see how it's going to go any other way than abandon the cliff and head back out of the crater.

Cooper has to go to the doctor, so we sketch out the first version of a drive together and he heads out. The rover is dug in very deep, worse than I've ever seen, with the left front wheel buried in a hole (of its own making) deeper than it is tall. We can't turn clockwise to escape the hole, since there's a rock on top of the pile of dirt the rover kicked up on that side -- likely, this will be too much for us to get over. We can't turn counterclockwise because we can't see off to that side, though of course it stands to reason that there's a big pile of dirt there as well. And we can't get out by going uphill, because that's how we dug ourselves in.

So the only way out is straight ahead -- downhill. The only sure way to get completely clear of the hole is to drive the rover about 2m forward, losing more altitude than we gained on the experimental drive. This is all very, very discouraging.

After that, we revert to the plan that appeared to work well for us on the experimental drive: a switchback, so that if the rover gets stuck, at least she'll change direction and perhaps make some progress. Three switchback legs of about 4m each, and thanks to the 2m downhill bump at the start, about the best we can hope for is that we'll break even -- that we'll be more or less back where we started before this latest drive. No closer to Burns Cliff -- and, considering what another failure will mean, in a sense we'll be much farther away.

At least we've got one more chance. Just to make it dramatic, Cooper's appointment means I get to do it all by myself. So, great: we're going to fail to get to Burns Cliff, and I'll get to feel like it's all my personal fault.

Well, what can I do to improve the odds? When I look at the drive Cooper and I sketched out, I spot a problem: switchback driving tends to carry you, on average, straight uphill. But the hole we've just dug is straight uphill from where we start the switchback. We might avoid it, but with all the slip we've been encountering, there's no certainty. So I change the drive: instead of three legs of 4m each, I use one leg of 8m and another of 4m. If this gains us enough uphill distance to get us onto the rockier terrain (the so-called "paving stones") we can see there, we won't care that we're a few meters farther from the cliff. (Just a few meters is all it would take, damn it! You could walk it in seconds.) And if it doesn't, then we'll be that much closer to the exit.

Assuming we don't just dig an even deeper hole, that is.

I name the drive "Hail Mary."

As it happens, I don't have to plan it completely solo. As I'm nearing completion, Spirit finishes its planning for the sol, so Jeff Biesiadecki comes upstairs to see how I'm doing, and Art presses him to backstop me on the sequence. Then John Wright joins him, and Art ropes him in as well.

So we decide to thoroughly review the thing among the three of us. This will delay uplink even longer, but considering the science we're trying to preserve, who cares?

Though it's a reasonably short sequence, reviewing the drive takes maybe 45 minutes or an hour. To paraphrase an old joke, if you put three rover drivers in a room, you get four opinions. Jeff and John do excellent work, poking at pretty much every decision I made, and coming up with some points I hadn't thought of. In the end, little changes -- the drive stays mostly intact, except that we convert the switchback turn from a turn-in-place into a K-turn. (The rover backs uphill through the long leg, then arcs halfway into the turn, reverses course, and arcs ahead to swing into the shorter second leg, driving forward.) I don't expect this to make a lot of difference, but it might save us some slippage, and it definitely looks cooler.

The only question is, will it work?





Courtesy NASA/JPL-Caltech. Well, crap. Looks like it was smart to keep that whole "NEENER NEENER NEENER" thing to myself.

2009-10-28

Opportunity Sol 271 (Spirit Sol 292)

I arrive a little early so that I'll be there when the downlink starts to flow. I'm a little anxious about the drive, yet confident.

While we're waiting around, Eddie Tunstel asks me a question. "I was looking over your uplink report from yesterday and noticed something about the starting position. It doesn't seem to match what we have in the telemetry."

Suddenly I don't feel so well. I look back at my notes and realize that I must have used the wrong INCONS (initial conditions) file for planning this drive. The file I used matches the one before the most recent drive, so it starts the simulated rover about 3m away from Opportunity's true position. Which could be very, very bad.

Luckily, the more I look into it, the more it looks as if we dodged a bullet. What we needed in order for this drive sequence to execute successfully was (a) that we start out at a 25-degree heading, and (b) that there be lots of room behind us. It so happens that we had a 25-degree heading in the INCONS file I loaded, just about exactly Opportunity's actual heading (or else we'd have been more likely to catch the error). And we checked imagery to establish that there was free space behind us, and that's not affected by the choice of INCONS file.

The only thing that might screw us up is that we chose the rover's starting position to establish a 10m-radius "safety zone" for the drive -- if we left that zone, the rover would stop driving. But since our true starting location was only about 3m away from the one we were using in planning the drive, we're still likely to stay within the zone for the whole drive.

Or so I hope.

Still, I get more and more nervous as more and more time goes by with no data. The SOWG meeting starts and we still have nothing from the spacecraft -- half an hour late, without so much as a byte on the ground. I can't help it. I start thinking of ways that using the wrong INCONS might have killed the rover.

Of course, it turns out Opportunity is just fine. Our problem's not with Opportunity, it's with Spirit. Both rovers are sending thisol's downlink via Odyssey, and Spirit got in ahead of us, with a whopping 148 Mbits -- all of which has to come down before we get anything from Opportunity. And there's hardly anything we can do before we get that data.

So we take a couple of hours off ....

When the data does arrive, it brings good news. It's about time we had some good news! I'd almost forgotten what it looks like. In this case, it looks like a front (downhill-facing) HAZCAM image showing switchback tracks downhill of us. We made some progress uphill!

Happily, Justin Maki made sure every image file that comes from the spacecraft includes metadata saying, among other things, where the image was taken, so we know just about exactly where the rover is. Turns out we made about 4.25m of total distance from our starting point (our actual starting point, not the wrong one I was using). About 3.5m of that is uphill motion.

It may not sound like much, but I'm practically ecstatic. The 3.5m itself is good news, but even better, it means we've got a technique to make more uphill progress whenever we want to.

About then the data stops coming. There's hope of more later, but it won't arrive for a while, and most people are still out to lunch and stuff, so I tell them to call me if anything interesting happens, and I go off and have a long talk with John Wright about what we need to do to adapt RSVP for Phoenix.

When that's over, I head back upstairs and -- oops. I guess everybody wasn't out to lunch and stuff. Frank has the next sol's drive pretty much ready to go, and Saina's pushed the process up to the activity plan approval meeting already and is ready for the walkthrough.

Well, that's embarrassing, but it's not enough to put a damper on my mood. My mood is further bolstered by the traversability meeting that follows, which has pretty much the same participants as a couple of sols ago, but now they're not so sure we can't make it to Burns Cliff.

All I have to say about that is: NEENER NEENER NEENER! I TOLD YOU SO!

But I say that only in my head.





Courtesy NASA/JPL-Caltech. Progress! At last!

2009-10-27

Opportunity Sol 270 (Spirit Sol 291)

The test drive is a go for today. Steve Squyres likes our planned drive. He calls it a "mobility potpourri," then "a well-thought-out hodgepodge of mobility commands."

Since I did all the work yesterday, you'd think today would be easy. But I know better, which is why I kicked out a version of the drive yesterday: so we'd have time to do it over today. Because there's always something ....

For one thing, as soon as I look at the drive, I realize we chose one of the switchback angles wrong. One of the pairs of switchback legs has the rover going at 60 degrees to the slip vector, which in the face of the level of slippage we're seeing is almost guaranteed to fail to make any uphill progress. Instead, we make that switchback be 30 degrees from the slip vector, which ends up changing other things -- at that point, the angles to the slip vector are such that we should do all the driving in the same direction (backward), rather than switching between backward and forward driving as a strategy to minimize turns.

Then Squyres raises another problem at the APAM. We're starting the drive against a rock and with the wheels in a self-dug hole, and the wriggles at the start of the drive might not suffice to get us past the rock, which would basically make that part of the test useless. Rather than wriggles followed by switchbacks, he suggests switchbacks followed by wriggles. Frank and I talk it over and agree with Steve, but because it's not as simple as just reordering chunks of commands, we end up having to rewrite almost the entire sequence from scratch.

Well, that's why I made sure there'd be time to do it over. I'm learning.

2009-10-26

Opportunity Sol 269 (Spirit Sol 290)

At first blush, the data look encouraging. We don't have any drive errors, which is definitely good news. But when we get a little more data, we can tell that the drive didn't complete -- Opportunity's nowhere near the predicted final location. In fact, she's only about three or four meters from where she started.

Plugging the reported position into RSVP makes one thing clear. Yestersol's drive consisted of a short downhill leg, a cross-slope leg, then an uphill leg. Opportunity appears to be around the start of the third leg, as if she'd gone down and across, but not very far back up.

That strangled groan you hear is coming from me.

We get a front HAZCAM -- since we're driving backward uphill, this shows the downhill view -- which shows the front wheels dug into the sand. The current-draw plot doesn't show anything suspicious -- but that tells us something. If one of the wheels were hung up on a rock, we'd expect its current draw to be much higher than the others; since we're not seeing that, we can tentatively conclude the rover's just spinning its wheels in dirt.

This is getting monotonous. But Jeng comes up with another approach we can try: switchbacks.

That's how Opportunity got out of Eagle Crater, the one they landed in. Trying to power straight up the crater wall didn't work, so they used a switchback technique to climb out more gradually, and this worked great. Not only might this help us climb the wall, it would also tend to help keep us from getting hung up on rocks: if we were unable to progress past a particular rock, we wouldn't just keep trying to push past it. By the nature of the switchback, at some point we'd turn and head the other way.

This idea has real merit. I pitch it to Frank, who agrees.

But we might not get a chance to try it. "I'm not sure we can do this," Art says. "If we have to go 25 or 30 meters cross-slope in 100% slip territory, we might just not be able to make it." I think this is premature, but I have to admit things haven't been going so well lately. (Not that it isn't par for the course in the Land of Opportunity, but still.)

Frank and I talk to the Mobility/IDD team about how to proceed. The general consensus is, basically, to take a sol off. Because of ongoing ground processing problems, we don't have the rear HAZCAM, which means we don't have an uphill view, and we don't want to proceed in that direction without one. The same rationale that kept us from deploying the IDD on Friday applies today, so we can't do that either. Instead, we'll take the time to prepare a test drive for the next sol, probably a switchback maneuver of some kind.

The next step is to have a long and painful meeting about this. The meeting draws in Daniel Limonadi, Jim Erickson (it's always good news when the project manager gets involved), Jake Matijevic, Jeng Yen, and Rich Petras, along with Frank and myself. Steve Squyres teleconferences in. The subject is, broadly, what the hell do we do now -- not just tomorrow, but beyond that: do we want to keep trying to get around Wopmay to Burns Cliff or give up and go for something else? Jim Erickson, for one, doesn't seem to think we can make it to Wopmay at all along the proposed route. He seems to have begun to adopt Art's view that we should quit trying now.

My take is that the nervous Nellies are giving up too easily. We haven't tried every weapon in our armory yet, so it's too early to give up on a high-value science target. (To be fair, one of the proposed approaches is not to give up, but to exit the crater, go around, and re-enter directly above Burns Cliff. But this would take a long time, several weeks at least, and it's still too early, IMHO, to decide to do all that when we have more things we can try cheaply in situ.)

My opinion is bolstered, at least in my own view, by the rear HAZCAM, which popped out of the ground system pipeline at last. The image shows the right rear wheel on a small rock. But Opportunity doesn't appear to have been struggling very hard with this rock. It's entirely possible we were already nearly at the slip limit and the rock was just enough to push us up to 100% slippage. Here's why I see this as good news: the vehicle is doing what we want, except when it runs into things we can't see. The switchback approach helps us avoid things we can't see, or rather minimize their effect, so it's very much worth giving it a try.

I try to make this point, but, like the vehicle, I rapidly cease to make headway. I give up in frustration, and it occurs to me that it's a mistake to push for continuing on the current path anyway. To put it another way, it's a mistake to get invested in any particular plan. This is the science team's vehicle, I just drive it. Maybe I've lost sight of that.

Instead, I just sit there with my mouth closed and wait for the naysayers to talk themselves out and leave. Then Frank and I work out the drive.

The first thing we need to do, we realize, is spend a sol evaluating different approaches. The point of this drive will not be to get us uphill (though if that happens, so much the better). Rather, we'll spend the time trying different strategies, figure out which one worked best, and then use the best strategy to get us uphill so we can get around Wopmay and on our way.

We decide to start out with "wiggles," small turns-in-place followed by short uphill arcs. We'll turn just to one side of the slip vector, take a short uphill step, turn just to the other side of the slip vector and take another step, and so on. The theory behind the wiggles is that our uphill drives are working except when we get stuck on something, and the small turns could help us drive around any little obstacles. We also try a couple of curvy wiggles -- building the heading change into the uphill step itself.

After that, we'll switch to a switchback test, experimenting with 2m legs at different angles to the slip vector.

I draw this out on a whiteboard as we build it, and the result reminds me of the shape of the rat-tail-shaped carmen figuratum in Alice in Wonderland.

Okay, I guess you have to be me to get the connection.

Frank was up earlier than I was, so he goes home and I work out a prototype of the drive.





Courtesy NASA/JPL-Caltech. The downhill view, showing the wheels -- particularly the right front wheel -- dug in fairly badly.

2009-10-23

Opportunity Sol 266 (Spirit Sol 287)

Seriously, what does it take to get out of here? The downlink brings unwelcome news: we appear to be stuck on another submerged rock. ("The Presumptive Son of Bane," Andy dubs it.) The good news is that the drive was doing what it was supposed to do, right up until we got stuck. The bad news is, that happened less than halfway through the drive. After turning, we made maybe 4m uphill before we got hung up.

Worse, we have a motion error, which is the flight software's way of telling us something went physically wrong. This turns out to result from a stall in the left front wheel, which occurred because the rover dug itself a nice deep hole under that wheel while trying to push uphill. At some point, while trying to perform a turn to reinforce its heading, the sides of the hole kept the wheel from steering to the needed position, and the rover declared a fault and gave up.

To make things more complex, we don't have imagery showing the rock. We don't even know we're on a rock, though the motor current data suggests that's what's happening. If we are on a rock, we don't know where it is with respect to the vehicle, so we're not sure how we can get away from it.

Looking at Mark Maimone's drive plot clears up part of the mystery, at least. By looking at the rover's heading, we can see that as it tried to drive straight backward, it would yaw clockwise. This suggests that whatever's blocking us is behind one of the left-side wheels. That means we should rotate clockwise to get away from it.

The drive Brian and I work out is, in its way, a thing of beauty. We first turn in place to get the left front wheel out of the hole. (To do this, we have to temporarily disable that wheel's steering actuator so it won't just repeat the stall from yestersol, but luckily it's already mostly in the turn-in-place position, so that won't matter much.) Then Brian comes up with a series of twisty little arcs that, with the help of slip, will wiggle the vehicle around the hole it dug, so the other wheels won't get stuck there. Then we turn to face uphill and somewhat away from Wopmay. (As we slip through this turn, RSVP shows, we'll end up with Bane dead ahead of us -- perfect for a HAZCAM! But we let the opportunity go.) After that, we drive uphill at an angle -- if slip were wind, we'd be tacking across the slope -- before turning once more and driving directly uphill.

It's a more complex drive than I'd like to shoot for in this area in a single sol, but I can't wait to see it happen.

Along with the drive, we have IDD sequences to do. It's a Friday, meaning it's a three-sol planning day. So they plan to spend two sols using the APXS and MI on the filter and capture magnets, and do the drive on the third sol.

But Andy thinks of something at the walkthrough. "If we know the terrain under the rover has changed, and we don't have a picture of it," he asks, "how do we know it's safe to deploy the IDD?"

Damn, I should have thought of that.

Which is not to say I think there's a problem. Andy suggests that we might have dislodged a rock in such a way that the rock is in the space the IDD moves through while deploying, making the rock invisible to the HAZCAMs but a threat to the arm. But in order for this to have happened, all of the following would have to be true:



  • The left rear wheel, which is drawing the highest current -- suggesting it's the wheel directly against the dislodged rock -- would have to be drawing more current for some other reason. The rock (Presumed Son of Bane, or PSOB) would have to be under the left front wheel instead, which is not showing a particularly high current draw. (That would mean it's moving a rather large mass around without giving us any evidence.)


  • PSOB would have to be half a meter or so long, in order to extend from the left front wheel into the unseen part of the IDD work volume. This would make it about twice as large as Bane, maybe bigger.


  • Rather than being pushed straight back in Opportunity's direction of travel, PSOB would have to have rotated more or less 90 degrees, and find a convenient fulcrum underneath it that would cause it to extend diagonally upward toward the IDD at an angle of about 60 degrees.


  • This would have to have happened in a way that left no traces in what we can see -- nothing in the numbers, nothing around the left front wheel itself. No shadow, no cracking, no evidence of disturbed soil, nothing.




So I don't believe this for a nanosecond. But I should have thought of it anyway, if only to dismiss it.

Unfortunately, I can't seem to convince anyone else I'm right. As soon as Andy raises the idea, Steve gets nervous, and that makes Jim Erickson (who's been hanging around a lot ever since we nearly collided with Wopmay) nervous. We have a big debate about it, with me the only defender of the proposition that it's safe to deploy. ("I'd bet my job on it," I declare at one point. "And you know how much I love my job.") Even Brian joins them. In the end, I'm heavily outvoted.

OK. Well, what the hell. We can do this science later -- wherever we go, we take the magnets with us. Anyway, having only the drive to worry about simplifies my job. And who am I to force science down the throat of the project's principal investigator? If he doesn't want to do it, we won't do it. I graciously concede defeat, and the IDD sequences are dropped from the plan.

When it's all over, Saina announces: "New rule!" She mock-glares at Andy. "No more thinking during the walkthrough!"

[Next post: sol 290 (Opportunity sol 269), October 26.]

2009-10-22

Opportunity Sol 265 (Spirit Sol 286)

Did we make it? That's what I'd like to know. When I come in, the downlink pass is over, all the data is on the ground, and we can't see any of it. By one of those rare but weird coincidences, there are independent problems with the network, SAP, and MIPL's downlink processing, all showing up at more or less the same time. The upshot is that we don't have the data in a useful form and therefore don't know anything.

The SOWG meeting starts anyway, on the assumption that everything's OK. This official optimism is pleasant but, given recent events, it hardly seems well-founded. But when the data starts coming in, Art's able to announce: "This is very, very, very preliminary, but it looks like the drive was successful."

And whaddaya know, it was. Sure, it was simple, but it did just what we needed, backing away from Wopmay carefully and safely, and ending up pretty much exactly where we'd wanted it.

Now we have breathing room. Our next step is to try to get uphill, so we can get past Wopmay on our way to Burns's Cliff. As usual, the drive is simple, but deciding what it should do is complex. We decide the best course of action will be to drive straight uphill, which means at a heading of 25 degrees. (To be more precise, the rover is facing 25 degrees, but we'll be driving uphill backward, so our direction of motion is 205 degrees.) We settle on this heading after analyzing slip data from yestersol, which is a bit spread out but lies in the range 16 to 34 degrees.

We want to do the longest drive we can in the time available, and as I'm thinking about how to do this I have an insight. Visual odometry limits how far you can go between position updates, so we usually keep our commanded distance to 60cm or so. But since we know we'll be slipping a lot, we can command larger steps. After all, what matters is not how far you told the rover to move, but how far it actually moves, between taking visodom pictures. To leave a little margin for error, we command 80cm steps -- this should come out to 40cm or so after slip, easily small enough for visodom to work.

With that approach helping to extend our limit, we command an uphill drive of about 20m, which should come out to about 10m after slip is accounted for. That should leave us plenty of room to get past Wopmay, and from there on we're clear.

I have one other quick-witted moment thisol. I've still got laryngitis left over from the flu, so when Art overhears me talking to Mark Maimone, he says, "You sound terrible."

"Yeah, but you look terrible," I reply, "and I'll get better."

If you're going to steal, steal from the best.

2009-10-21

Opportunity Sol 264 (Spirit Sol 285)

Somehow I don't have a good feeling about yestersol's drive. This is borne out when I walk into the SOWG meeting and Cooper says, "Did you hear about what happened?"

Sure-fire bad news opener.

"The Brain Tractor Beam kicked in," he says. This is Frank's concept: the Dino Brain -- Wopmay -- has magic powers and can remotely screw up the rover's drives. "We hit a submerged rock and got 100% slip. Now we're back near the Brain somehow, with a solar panel near the rock."

Oh, shit. This is very bad.

We don't know enough about what happened yet to decide how we're going to try to fix it, so we can't tell the SOWG meeting what we'll need. Since we're going to be their highest priority, this means they can basically do nothing, so they decide to give us some time and reconvene in an hour or so.

So Brian and I go talk to Mark Maimone, who's Mobility/IDD today. He's got a graph up, showing the rover struggling to climb uphill and breaking free at last. He's also got an image showing Wopmay off our port quarter. It's about 1.6m from the center of the WEB, which would make it about 30cm -- roughly a foot -- from the edge of the solar panels, in the worst case.

This is not a lot of room to maneuver, especially because any motion on our part -- even steering the wheels -- will tend to cause us to slip further toward Wopmay. In actuality, we have a little more clearance than that. The farthest tip of the solar panels is downhill from the tallest part of the rock, giving us a little extra room. But it's still not a lot.

Clearly, we need to get away, and fast. Mark's suggestion is to go downhill, to use gravity instead of fighting it. This frustrates me, because I'd been thinking exactly the same thing the past couple of sols: gain cross-slope distance from Wopmay by going downhill, then come back uphill later. But I hadn't said anything, so shame on me. Anyway, Mark's suggestion is the right one, and we adopt it as our tentative plan.

Mark has another image, a very surprising one. It's the rear HAZCAM image, and it shows a rock about a meter behind the rover. That's not so unusual. The unusual part is that the rock wasn't there before.

The rock's in the spot where we got stuck, too, which is telling. And there's a big hole behind it. It looks like what happened was this: as we were driving uphill, the rover somehow pushed a rock up out of the ground, then got itself stuck on the rock.

Think of a pond in winter. It's got a thin crust of ice over the top, and as you walk around on it, the ice starts to crack. Something like that is happening underneath Opportunity. This is not precisely news -- for one thing, a few sols ago, a rear HAZCAM showed a curious kind of ridge along the wheel track, caused by the rover crunching through a lightly cemented top layer of the soil. We've seen similar effects on Spirit. But up to now, all that had been revealed underneath the cemented layer was soil, so nobody really cared. Yestersol's drive was the first time we managed to kick up a rock.

Even stranger, one of the scientists, Rob Sullivan, looks back at the previous sols' images of the spot where the rock -- Andy christens it "Bane" -- was dislodged. Even in hindsight, there's no sign of the thing. Which has a disturbing implication: we can't tell where these submerged rocks might be, so our drives might stall on any of them at any time.

Later, I'd come to realize that kicking up hidden rocks, probably performing the first extraterrestrial excavation in history, was seriously cool. But at that moment, my reaction was: we're sitting right next to a rock whose size makes it a threat to the vehicle, any motion we make will tend to slip us closer to the threat, and now you're telling me we're in a mine field.

For the future, Rob has some practical advice. His guess is that unearthing (unmarsing?) these rocks is more likely when we're driving over terrain we've already driven over a lot. Where the ice is already cracked, we're more likely to fall through, so to speak. That doesn't help us thisol, maybe for the next couple of sols, since we have no choice but to drive away through terrain we've already spent a lot of time in. But at least it gives us hope that this won't be a problem forever. (We also get some welcome feedback from Steve Squyres: "Don't feel any pressure from the science side to get us to the other side of Wopmay fast. Our number one goal is for you to keep the rover safe. I've grown attached to these rovers.")

But for now, we need to worry about the short term. We decide not to try anything fancy. We're going to back away, mostly downhill, in a short shallow arc that simultaneously avoids both Wopmay and Bane, then tighten the arc to take us more directly across the slope. If all goes well, we'll end up a few meters from Wopmay, plenty of room to do an uphill drive on the next sol.

Turns out it's hard to type with your fingers crossed.





Courtesy NASA/JPL-Caltech. I cannot describe the sick horror I felt when I first saw this image: Wopmay, just off Opportunity's port quarter. This is not how I wanted my first Opportunity drive to go. Even now, it dredges up shame and fear and embarrassment.


Courtesy NASA/JPL-Caltech. The rock, "Bane," we unearthed -- or, uh, I guess, unmarsed -- completely screwing up our drive.

2009-10-20

Opportunity Sol 263 (Spirit Sol 284)

We didn't make it.

Yestersol's drive was supposed to back away from Wopmay, turning uphill as it went. Then if it got far enough, it was to turn to face uphill. Then if it was still reasonably close to our goal for it (it could slip downhill during the turn), it would drive on.

The way it worked out, the backup went reasonably well, not gaining as much uphill distance as we'd hoped but still OK. So it went on to do the turn, and the turn took it out of the goal region, so the second leg of the drive didn't execute.

This was a good thing, in the sense that the conditionals we put in protected the rover exactly as they were supposed to -- if the rover had executed the drive's second leg, it would likely have tracked across the slope and whacked into Wopmay, which, since Wopmay is big enough to damage the solar panels, would be bad.

"Is this what driving in this area has been like?" I ask Frank.

"Yeah," he says resignedly, "pretty much."

So we're trying again. Frank suggests I drive today, and I don't need to be asked twice. There are a few scary moments at first when I realize I haven't planned a drive for months -- been RP-2 too long. But then it all comes back to me, and I come up with a structure for the drive.

I decide to do this one a little differently. Since our real goal is to get away from Wopmay, let's try telling the rover that. I define a point on Wopmay (calling it "WopmayEdge") and structure the drive in terms of getting away from that point. We command the rover to drive six meters uphill, using visodom to see how far it actually gets. Then it'll check to see if it's reasonably far away from WopmayEdge, and if not, drive a little farther. Then, if it's far enough from WopmayEdge, it'll start driving cross-slope.

All of this would be easier if we just went downhill. But then we'd have another problem; we'd have to get even farther uphill later. So we might as well pay the price now ....

So I hack out this drive, and Frank and I go over it and make some changes he suggests, and that's about it. Somehow this seems wrong.

"I don't know how to put this," I say, "but what went wrong with the decision-making process that put me in charge of a four-hundred-million-dollar piece of equipment on another planet? Isn't there supposed to be a government body or something that decides what you can do with these vehicles?"

"There is," Frank points out. "We're it."

"True, but what I mean is ... well, isn't somebody else supposed to do this?"

"You mean, adults?" Frank says.

"Exactly!"

"I get that feeling all the time, dude."

The rest of the sol mostly goes smoothly. In the end, the drive sequence ends up taking longer to execute (or so our model predicts) than we'd originally projected, but since the drive is a high priority, Cindy obligingly makes time for us. There's just one problem, and it shows up very late. It has to do with our use of visual odometry.

Visodom is an application of parallax. That is, it works by comparing two pictures (really, two stereo pairs of images) taken a short distance apart. It computes how far common features have moved between the two images, then figures out how the rover must have actually moved (vs. how far it was commanded to move, an important distinction in such slippery terrain) in order to cause that apparent change. Hence, in order for visodom to work, there has to be a certain amount of overlap between successive images. This places limits on the kind of moves we can make between visodom steps, and given the camera pointing we're using on Opportunity, one of those limits is that we need to keep turns under about 15 degrees. If we want to turn farther than that, we have to break up the turn into multiple smaller turns, with visodom updates after each part.

Walking through the sequence at the CAM, I notice that I inadvertently exceeded this limit during the sequence. One of the turns I thought was 15 degrees is in fact 20 degrees, which isn't much over the line, but ....

The turn is right at the end of the sequence, meaning it won't affect the drive itself, but it might affect our ability to image Wopmay after the drive. If visodom fails to update, the rover isn't where it thinks it is, so it won't aim the cameras where it should, so we won't get the pictures we need in order to decide whether to explore the other side of Wopmay before driving farther away.

Since that makes this a science concern, Squyres, teleconferenced in, pipes up. "Is this going to make us miss our post-drive PANCAM?" he asks.

I think about it. We've already had to ask for more time for the drive, and I don't want to add more time to do another visodom update. Besides, we're running late, as usual, and I know people just want to leave. And it'll probably be OK .... "Nah," I answer at last, "I think the visodom update will work."

"OK," he says. He doesn't sound completely convinced, but what can he do but trust the rover driver?

That might have ended the matter, but it doesn't. The more I think about my answer, the less I like it. It's a matter of pride to make these things work, and besides, people are trusting me. The visodom update probably will work, but what's the point of taking the risk? Sure, we save three minutes by not doing it, but if it causes us to blow the PANCAM image, we could end up having to spend another sol here just to get the imaging. That's a heck of a downside. And it'll be all my fault, because I was too lazy to do the right thing.

The CAM is coming to an end. I make my choice.

"Hey, I hate to be a pain about this, but I want to change my answer. I really think we should break up that turn and add a visodom update."

Squyres sounds relieved: "I think we'll all sleep better." And, as happens every time, the uplink team springs into action. I make my changes and deliver, and Rich rebundles, and Cindy and Rich and Andy and I double-check the changed sequence. Fifteen minutes after I spoke up, we're done, and that's all it took to do the right thing.

Maybe this spacecraft is in the hands of adults, after all.

2009-10-19

Opportunity Sol 262 (Spirit Sol 283)

So we nailed the drive on Spirit. It cost us a lot of sweat Friday, but the post-drive front HAZCAM made it all worthwhile: there's Uchben, dead in front of us, pretty as a -- well, as a picture.

Too bad I don't care, because today's my first real shift on Opportunity! If I'd realized this before late last night, I'd have read more about what our sister mission has been up to. As it is, well, I'll just have to figure it out as I go along.

Working on Opportunity reminds me of nothing so much as visiting Australia. One of the things I was most struck by when I went to the wonderful land of Oz was that it was a lot like America ... but different. At times you were obviously in a foreign land, but at times it was almost like you were home. And a lot of the differences would be weird little things, things that wouldn't quite register consciously, like details of how people dressed. Exactly how they feel when they come here, no doubt.

One of the differences between the two missions is the habits of the RPs. On Spirit, the RP-1 comes in early and leaves early (usually leaving just after the walkthrough); the RP-2 comes in before the APAM and leaves at the end of the day, after the CAM. And there's a moderately formal handover in between. But on Opportunity, the RP-1 and RP-2 come in at about the same time and often leave at about the same time, and they kind of work on everything together, so there's very little distinction between the two roles. Luckily, I'd heard about this already, so I get my ass up early for the SOWG meeting.

And see, this is what I mean about it being almost, but not quite, like home. They're on Mars, with the same kind of vehicle, but they have a lot more solar energy available, and the rocks are all different and have weird names, and the terrain is different (crater vs. hills). Not that I mind, it's just ... weird. (And that reminds me of that one scene in Pulp Fiction ("It's the little differences"), but that's a little too much of a tangent for now.)

Fortunately, I'm here just in time to help explore one of the most interesting rocks we've seen on either mission. This is a huge rock officially dubbed "Wopmay," but unofficially everyone calls it "the dino brain." Because that's what it looks like: an enormous, crenellated brain. (Yes, I know real dinosaur brains were tiny. You look at the damn rock and tell me what comes to your mind.) They spotted it weeks ago and have slowly made their way over to it, reaching it a few sols back, after some tricky driving. Thisol will be our last sol here, and I get to help with what might be our last IDD sequence on it before we drive away for good. (We're going to drive around to the other side after the IDD sequence completes, and maybe come back and poke at it a little more if the other side looks interesting.)

Only the IDD sequence has a problem. ("Figures," Frank grumbles. "The days with tricky drives are the days when the IDD sequence gets tough.") The problem is that the target they've chosen -- Hercules -- is pretty much at the upper edge of reachability. When we try to MI spots above the target (that is, higher up on the rock face), we get errors from the simulation. Once we realize that's the problem, the fix is easy enough: I just shift the mosaic down the face.

Hey, it's my first sol upstairs, and I'm already helping! Woo-hoo!

Frank and I commiserate about the science team's habit of choosing targets on the edge of reachability. I think the problem is caused by the way the reachability information is presented to them -- using SAP, they get an image with a translucent green overlay showing approximately what area the IDD can reach. My theory is that the interesting part of a big splotch of color is the edge -- that's where your attention goes, so that's where the scientists pick targets.

"Maybe for the next mission we could define some notion of degrees of reachability," I muse. "Then we'd have multiple colors, splotches within splotches --"

"-- like a bullseye!" Frank says, picking up on the idea.

"Yeah, and that would draw their attention in, toward easier-to-reach spots, rather than out, toward harder-to-reach spots," I finish.

Note to self: pitch this idea to Jeff Norris.

So the day's other sequence is a drive. We back 2.5m away from the dino brain, turn, and head about 7m to get to a spot on the far side of the big rock, using visodom all the way. Wopmay's a big rock, big enough to clip the solar panels, so we need to be sure to avoid it. Frank shows me a trick they've been using to simulate slip, inserting extra commands to make the simulated rover periodically drive a short distance downhill. He fiddles with this until the simulated rover is driving more or less in a straight line to where he wants to go, then takes out the extra commands. This is so much better than how we've been dealing with slip on Spirit ("Um ... and then we should end up about ... I dunno ... there, ya think?") that I decide immediately to adopt the trick. Until the next version of RSVP comes out, which will model slip elegantly, but that's another matter.

Frank's sick, so he leaves a little early, but fortunately he calls in with a potential problem he thought of on his way out. We introduced a bug at the last minute: because of the conditional commands we chose to use, the rover could back off, decide not to do the turn, and then drive the second leg -- likely whacking it pretty much straight into Wopmay.

Naturally, I quickly fix this. After all, I don't want to break Opportunity on my first day.

I should at least wait until next week.




Footnotes:


Courtesy NASA/JPL-Caltech. Uchben, positioned right where Spirit can start poking at it.


Courtesy NASA/JPL-Caltech. Opportunity makes shadow puppets using the rock dubbed "Wopmay" (a.k.a. "the dino brain") as a backdrop.

2009-10-16

Spirit Sol 280

It's Friday, which means it's another three-sol planning day. I bring in Krispy Kreme for everybody -- on both floors -- making me the man of the hour.

Again, the weekend is not terribly complex for the RPs. Sunday, we do a little IDD work, mainly a short MI series intended to establish whether we can speed up the MI imaging sequences. Since the start of the mission, we've had to wait 15 seconds before taking an MI image, since that's how long the IDD is permitted to take to dampen its vibration after moving. This doesn't sound like much, but it adds up on sols with multiple MI series. Opportunity has experimented with waits as short as five seconds, with good results, so we'll try that ourselves on Sunday. After that we stow and go, attempting to retry the drive to Uchben that was interrupted by the latest appearance of the dynamic brake anomaly.

Speaking of which ... I manage to make it to part of the daily meeting on the anomaly investigation. Things are looking better and worse. Some problems that would definitely be unfixable from the ground, such as loose wires and failing solder joints, have been ruled out as the cause. It's good to know those aren't the problem, but we still don't know what the problem is. The leading candidates that come out at the meeting:



  • The relay contact is sticking when closed.

  • The relay contact is "dirty open" (that is, mostly open but with enough physical contact to appear electrically closed).

  • The relay is jammed. (It's not clear to me how this differs from the preceding cases -- I think they mean physically jammed, which would differ from merely slow or dirty.)

  • The electronics that sense the relay's state are faulty in some way.



Some of these could be worked around from the ground, for instance by telling the flight software to give the relay a little more time to open. Some can't. Guess what I'm hoping for.

Leo's been heading the anomaly investigation, and I think he's getting tired of it. He wants to decide by this time next week whether to just blow the fuse and live without the dynamic brakes for the rest of the mission.

I hope it doesn't come to that, as it'll likely be annoying to try to deal with wobbly, unreliable steering on top of the other challenges. But if it comes to that, we'll deal with it.

At one point during the day I overhear Andy discussing the changing character of the mission. "The first 90 sols were a sprint," he says. "Now it's turning into a marathon. And you can't do those two kinds of missions the same way."

As usual, it looks to me like he's right. I think there are a lot of people on the team who could use a break. Leo's one of them. Emily Eelkema is another. The TUL position is one of the most stressful, and thanks to a TUL shortage, she's been the TUL every day for the last couple of weeks. Plus, she's on call this weekend and is working part of next week.

I don't know how they'd have made it through the day if I hadn't brought donuts.

[Next post: sol 283 (Opportunity sol 262), October 19.]

2009-10-15

Spirit Sol 279

As predicted, we're not driving thisol. The scientists chose a couple of soil targets for us to IDD, and we put together a simple IDD sequence to handle it.

Word comes that we'll drive on the weekend, which I actually do find surprising. The plan is to straighten all the wheels first -- if the steering problem is with the electronics (not the relay), we'd much rather fail with the wheels straight than with them turned. Then we'll do a tank-turn, driving the wheels forward on one side and backward on the other side, possibly with the wheels unaffected by the dynamic brake problem turned to help. And then we'll finish the drive to Uchben.

Remember when we were routinely covering a hundred meters per sol?

2009-10-14

Spirit Sol 278

I'm a little worried about yestersol's drive. It was in two legs, meant to position us with a rock named "Uchben" in the IDD work volume. But I think we went about it the wrong way. When we're doing drives in multiple parts, we really need to ensure that the first leg of the drive gets where it's supposed to go before starting the second leg of the drive. And we didn't do that. We left the first leg of the drive with maybe 50cm uncertainty, enough to mess up our placement at the end of the second leg. And it wouldn't have been that hard to fix -- heck, I've fixed such drives before, by adding fine-positioning commands at the end of the first leg. I resolve not to repeat the mistake.

As it turns out, it's moot. "The [dynamic brake] relay freaked out again," Chris says. "We drove only two meters or so. We didn't even complete the first leg."

So that's bad. Especially because it suggests that Spirit really does have a hardware failure of some kind, not the one-time software failure we were all hoping for. This is something we're going to have to diagnose and, likely, deal with for the rest of the mission.

We're planning a drive anyway, to complete the one that was interrupted thisol. Since the failure -- whatever causes it -- shows up at the start of a drive command, Chris is working on a way to do it with as few commands as possible. Which means leaving out fine-positioning commands at the end of the first leg, which means I'm going to repeat the mistake I just vowed not to repeat. But at least it's for a good reason.

Or maybe I'm not going to have to repeat that mistake. There's a meeting going on to decide what we're going to do about the anomaly, and it turns out they've decided we're not going to drive thisol after all, to reduce the risk of unintentionally damaging the vehicle. So the uplink team switches to plan B -- an all-remote-sensing sol, requiring no RP work -- and I shrug quietly and go hang out in the anomaly meeting.

The upshot of which is, we'll likely be sitting still for the next few days. Jeff Biesiadecki is working on a diagnostic program to run on the rover, but it won't be ready until Friday, a couple of days away. "I don't want to run a new test on the flight vehicle for the first time on a Friday," Erickson says, which means it won't run until Monday. Meanwhile, we might IDD, but no driving.

I look through Rick's meeting notes, which are projected on the big screen, seeking something hopeful. One of them says something about one of the fuses -- they seem to have a known failure mode that could be causing the brakes to respond more slowly. If that's all it is, it seems to me, a simple patch to Jeff's mobility flight software code could fix the problem -- just increase the amount of time it waits for the brakes to respond.

The rest of the failure scenarios I have time to read look grim. I'm rooting for the fuse.

2009-10-13

Spirit Sol 277

I was out sick yesterday, so Ashitey asks me, "Are you better?"

"So much better."

"Good," interjects Craig Leff, "because we didn't like you the old way."

Ouch.

The day itself seems to take my revenge on Craig, though. He's got a relatively simple job to do, mostly delivering a couple of small NAVCAM panoramas. But they're supposed to be doing this in a new way: instead of building the sequences by hand, they're picking them from a huge library of thousands of sequences recently created by Justin Maki. This is supposed to simplify the process for the more lightly staffed extended-extended mission. Only Craig seems not to know how to go about figuring out which two of the thousands of available sequences he needs, and throughout the day there's an increasingly voluble and creative stream of profanity issuing from his vicinity.

I have my own tool problems to deal with. We're gearing up for another delivery of RSVP, and the various developers have been delivering different bits and pieces into their home directories for testing. Running RSVP now is like ordering at a Chinese restaurant -- combine pieces from here with pieces from over there, and hope the meal stays down. Something in the combination of pieces I've been using suddenly (perhaps because of a recent behind-the-scenes update to one of them) exposes a bug: HyperDrive hangs whenever you try to simulate the sequence.

Ashitey tells me Chris saw this same problem yesterday and, frustrated, resorted to using his (Ashitey's) favorite combination of pieces instead. I decide to learn from Chris's example and order Ashitey's combination plate, which works fine.

It's enough to make a person believe in configuration management, I find myself thinking. I must not be entirely well yet.

2009-10-06

Spirit Sol 270

Thisol's another simple one, continuing our IDD work on Temples. After a couple of MI series, we stow the IDD so we can perform a test to help out the dynamic brake anomaly investigation. The steering test consists of steering all wheels straight, then turning the right front and left rear wheels -- the ones affected by the anomaly -- from zero to five degrees and back, in one-degree increments.

So not the hardest sequence we've ever had to build. But gratuitous complexity still crops up, at least for others: power modeling, for instance, continues to be a problem.

"After 270 sols, I'm still frustrated by power modeling," Rick sighs.

Arvidson replies: "Rick, I think you'll still be frustrated at sol 540, too."

Sadly, he's probably right. Some things have gotten about as good as they're going to get.

[Next post: sol 277, October 13. Yes, more dead time thanks to the dynamic-brake anomaly investigation. Hmph.]

2009-10-02

Spirit Sol 266

I'm just sitting down to catch up on the latest pictures when I hear Henry Stone stride by in the hallway outside, speaking tensely into his cell phone. I only hear a few words, but it's enough to make me worried.

Screw the pictures. I go to the SMSA and sit down with Rich Petras and Jeng Yen, our Mob/IDD folks.

"How's Spirit?" I ask.

"It's broken," Rich says. "We did the half-meter and 45cm arcs but failed at the turn-in-place." He says it's something to do with the dynamic brakes, which I've never heard of before. Whatever the dynamic brakes are, they were off when they were supposed to be on (or maybe the reverse). The flight software detected that the brakes were in the wrong state and correctly refused to drive on.

"So it's a hardware problem?" I ask. "Not a sequencing problem?"

"Yeah, it's a hardware problem," he confirms. This makes me feel better and worse. Better, because it's not my fault. Worse, because I'd rather have screwed up than have this be a hardware problem. Much rather. When you're two hundred million miles away from the hardware, you don't like hardware problems.

Dynamic brakes turn out to be this. They're brakes (duh) on the wheels that are intended to prevent unintended steering or rolling. So there are brakes for the steering actuators (arranged in pairs) and drive actuators (likewise, I think). The brakes that failed are steering brakes, in particular the pair that controls the right front and left rear steering actuators. The idea is that the brakes should be released when the steering actuators are being run, and then reapplied while you're driving, so that the wheels won't wobble (inadvertent steering) while you're moving. Roughly, then, the way the rover drives is: release the steering brakes, steer the wheels to the desired orientation, apply the steering brakes and release the drive brakes, drive, release the drive brakes. And somewhere in there, the brakes were apparently in the wrong state.

When I showed up, Rich and Jeng were digging around in the flight software, trying to better understand the error message. What they're hoping for is that it's a flight software problem that just looks like a hardware problem. It might be, for instance, that the software just happened to check the state of the dynamic brakes at the wrong time, one of those one-in-a-thousand bugs that you'll nevertheless run into if you do it enough.

Even if it is a hardware problem, it might be something temporary. For instance, the relay that controls the dynamic brakes might have been slow to respond due to the cold temperatures. It might work perfectly the next time we try it, and might never fail again.

And even if the this set of dynamic brakes is permanently stuck, we can still drive. We can deliberately blow the fuse from Earth, permanently releasing the brakes. Then we tell the flight software to ignore the state of those brakes, and we drive normally. Well, not normally, unfortunately -- we'll have to change the way we drive to account for the fact that the wheels will wobble during the drive, possibly taking the rover off course. But at least we'll be able to do it.

Naturally, this changes our plans. The scientists are going ahead on the assumption that we can safely use the IDD, though this is not guaranteed -- some failure modes could involve electronic problems that using the IDD would exacerbate. But this is considered unlikely, and anyway the science team has nothing else to do.

With the engineering teams, it's a different story, as you might expect. The mechanical team has already started working on the problem, and a meeting is scheduled for 13:30 to discuss the results. Rick Welch schedules an 11:00 meeting to work out what they're all going to say at the 13:30 meeting -- which, since the two meetings have different audiences, is less useless than it might sound. The 11:00 meeting is the actual working meeting; the 13:30, since it's the formally declared meeting, will attract more high-level types who want to know what's going on.

I go to the 11:00.

In addition to the various graphs and charts people are already coming up with, the participants work out a list of possible causes. There might have been some failure in the command's attempt to send a signal to the relay, the relay itself might have failed (or might be in the early stages of degrading -- someone's assigned to find out the part's design lifetime and known failure modes), the relay might have worked but incorrectly communicated its status. There might also be a bug in the flight software, including a race condition, but the responsible developer -- Jeff Biesiadecki -- calls this unlikely. The relay is supposed to respond within 40msec under all circumstances, and his code gives it 65. Still, he volunteers to look into it.

With the mobility subsystem off-limits for now, we're left with IDD sequences only. The scientists choose to spend most of the weekend calibrating the APXS. We're going to take several APXS readings of a rock that happened to end up in front of us when the rover stopped driving -- first with the APXS directly on the rock, then at 5mm, 10mm, and 15mm away. Last of all, we change from the APXS to the MB.

I could do that in my sleep. If I could sleep.

[Next post: sol 270, October 6. Due in part to a mobility stand-down to allow time to investigate the dynamic brake issue, we didn't resume driving for a few sols.]

2009-10-01

Spirit Sol 265

Daniel Limonadi's trying to scare up a scientist to give the science lecture Friday, but nobody's around; they're all at their home institutions. ("You should see the end-of-sol science discussion upstairs right now," one of the few remaining scientists commiserates. "It's the saddest thing I ever saw.") So it falls to Matt Golombek -- thisol's SOWG chair -- to be a good sport and volunteer.

"What do I have to do, exactly?" Matt asks.

"Anything," says Daniel.

Matt thinks about it. "Okay."

Thisol's drive is short, about 1.5m, but tricky. We want to get a particular rock (Coba) in the IDD workspace, but the rock's kind of narrow -- at least along the axis from which we're approaching it. We also want to be turned to a heading of 90 degrees for comm, and we want to ensure that we avoid a particular direction (160 degrees) that's extremely bad for comm, in case the drive faults out in the middle.

Luckily for us -- if you want to think of it that way -- it looks like we're going to get some extra time to work on it. This is because of a problem in the Planner, the software that helps automate the construction of a valid activity plan. The problem turns out to be due to a hard-coded limit in the Planner. Essentially, the developers chose a hard-coded time -- 150,000,000 seconds since the "epoch," 2000 Jan 01 -- to represent "infinity," the time beyond which nobody would ever conceivably want to plan. The software won't let you make plans that extend past the point where the clock runs out.

Guess when the clock runs out.

If you guessed "during today's plan," give yourself five points. You are correct.

It transpires, during the resulting conference call, that the Planner's developers knew about the limitation but hadn't bothered to figure out exactly what it converted to in terms of the real-world calendar. ("Sometime in October" was apparently as far as they went.) They also hadn't bothered to tell the project.

The good news is that they know what the fix is, and will start working on fixing and delivering it now. (Start working on it now? It makes me want to scream. They knew about this problem and couldn't alert the project, fix it, and redeliver the month before it failed? I understand that the rovers have lasted longer than anyone expected them to at the start of the project, but the Planner's developers should have handled this situation better. End soapbox.) And they have a workaround: you can lie to the Planner about what sol it's planning for, although this makes coordination with other uplink tools difficult, slowing the process.

The other good news is that, as Ashitey points out, we rover planners are, for once, not the ones holding up the uplink process. We even have time to sample the free root beer floats given away by the company that designed JPL's Rose Bowl float this year. It's a nice reminder of the long-gone days of free ice cream.

Ashitey's planned a very careful drive that avoids nearby rocks that might cause us to slip, and that also avoids driving over Coba. As I'm hacking on his sequence, though, I realize that there's a much simpler, more direct, faster, and safer drive -- but it'll cause us to drive one of the wheels over Coba, which the scientists often don't want us to do. In fact, if the drive goes exactly as planned, Spirit would drive her left front wheel onto Coba, turn the wheel, and then drive off as she turned in place, leaving Coba directly in front of her. This would be likely to scuff the rock, at least a bit.

I think about it and decide to ask Matt if we can drive over Coba. He asks the rest of the team and comes back with, "Sure, scuff it up all you want. Fresh surface would be great. Hell, if you can break it, that'd be even better. Stomp on it."

And to think I almost didn't ask. Naturally, we change the drive.

As I'm working out the details, Daniel Limonadi (our TDL thisol) starts waffling on the comm direction -- turns out that 70 degrees might be better than 90. We've just gotten the drive sequence worked out, and this would change it again. Ashitey somehow manages to talk him out of it, and the drive remains intact. Good on ya, Ashitey!

Daniel also proves a tough critic at the walkthrough, asking several difficult questions and in the process uncovering a couple of minor, but real, problems. It makes me realize that while this sort of probing questioning used to be a daily (solly?) occurrence at our reviews, it's now virtually absent. I don't know why the change occurred -- everyone just assumes everyone else knows what they're doing, I guess. Maybe that'll be how we lose the rover.

In any case, the problems Daniel points out are readily fixed, and in the end I go home feeling more confident in the sequence than I would have otherwise. Good on ya, Daniel.