PowerMac Mac OS 9 G4s don't restart anymore after a power failure

[os9rocks]

Hi there,

I'm new to the forum. I'm writing from Milan, Italy. I've searched for a topic on this subject, but can't find any, so I'm writing this. My apologies if I missed something.

I have some 15 RUNNING PowerMacs: they are mostly silver G4s, running on OS 9.2.1. They're running without a glitch from the late nineties, 24/7. There are 2 dns, 2 Eudora mail server and several WebStar servers, running actual services.

One example is MacExchange, the most respected italian site to trade used Apple items. I don't know if I can post a link, just ask if interested (or just add .it at the end).

Coming to the subject, every now and then the WebStar servers go down, mainly due to some kind of attacks. I have four APC devices that I can access remotely to simply cut the juice off and back on. This worked perfectly until earlier this year. The G4s now refuse to restart when the power is cut off and on, as the PRAM batteries were dead, which they're not. I always carry spare batteries, changed them, tested them and everything is fine. Except it doesn't work anymore as expected.

To add to the confusion, 2 G4s are behaving as expected!

It seems that a date has been reached, after which the PRAM mechanism became of no use.

Is anyone of you aware of an explaination or, better yet, a workaround to this?

Thanks in advance for your answers!

Kind regards,
Guido
 

[Mat]

Welcome Guido!

Wow, I am stunned by a Mac OS 9 server farm in 2016! Verry cool.

I have no real idea for you, but I would suspect the PSUs becoming weak? Did you try some of that software outstide that automatically restarts the macs after any power off? Or how did you do it until now that the machines rebooted when you switched powr off and on remotely?

[nanopico]

When you killed power was it just like a switch to the power or did the ups send the signal to the machines and they power down on their own?

[os9rocks]

@nanopico: the APC device is not an UPS; it has 8 power inlets and 8 power outlets which you can manage remotely via Web. When you perform a restart, the APC device cuts the power off, waits a few seconds and then gives power back to the specific outlet. Exactly like a switch. It's physical, not logical. The device is called APC Web/SNMP Management card.

@Mat: thanks! I feel a bit like the notorious japanese soldier, not understanding for decades that the war was over... Of course we have OS X servers as well, but some projects runs swiftly on OS 9, so why change? What software are you suggesting? I understand that if the OS is freezed, the only way to go back in business is physically restarting the Mac. However, if you're not there to push on the power button, the only choice you have is to cut the power and giving it back. However, this works only if the Mac has the appropriate flag checked in the System preferences (Reboot the machine after a power failure). This info is written in the PRAM. It seems that most machines can't access anymore the PRAM after a power failure.

[Nameci]

Have you tried resetting it? via cmd-opt-p-r on boot?

[GaryN]

Coming to the subject, every now and then the WebStar servers go down, mainly due to some kind of attacks.

Could these "attacks" be the cause? Somehow corrupting the PRAM settings? Without detailed info on what kind of attacks and exactly how they go down, this is no more than a "hmmm, I wonder" kind of idea, but if the ones that have been hacked / attacked are the same ones with the symptom and the others are fine, then you have a clue at least.

[DieHard]

Yes... I was definitely confused and I assumed several of the units did not "Power on", but not I see the light... the front power button light that is

[os9rocks]

@DieHard: the Macs work as expected in every other aspect, except the PRAM data (auto-reboot flag, time and such). So, the only light I see is the power button... 

@Gary: I doubt any of the attack could be the cause. I can't remember a single day without tens of them, on every server. The attacks mostly are attempts to gain server control (usually via the php or Apache known vulnerabilities, even if we don't use php nor Apache), some try to cause a classic exploit situation (which is totally unuseful on OS 9). The problem they cause is WebStar freezing, usually due to too many weird requests causing small memory leaks resulting sometimes in a WebStar quit, but more frequently in a system freeze. Nothing fancy, we're not hosting any sensitive data or potential target company on those servers. Furthermore, crashes are spaced erratically many weeks in between.

@Nameci: Yep, I did it several times. Even the three times kind... to no avail.

If you guys have a PowerMac G4, would you try and replicate the issue?

[GaryN]

@Gary: I doubt any of the attack could be the cause. I can't remember a single day without tens of them, on every server. The attacks mostly are attempts to gain server control (usually via the php or Apache known vulnerabilities, even if we don't use php nor Apache),

Call me paranoid, but three systems do not suddenly develop identical faults without a common cause and my gut says that a constant battering by anonymous Ukrainians or whatever might be capable of corrupting something.

I would start by replacing the Energy Saver Control Panel.

The function failure that's causing you so much aggravation boils down to having one little checkbox-on-or-off store one lousy bit properly in PRAM. Since refreshing the PRAM doesn't help, you move to the next place up the line - the Control Panel, to see if it's actually switching that bit. If that doesn't do it…

Reinstall the System

You only have to troubleshoot one of the computers so it shouldn't be all that much trouble. Hopefully, by this point you'll have cured the issue, because if not, the next step is to replace one of the "bad" servers with one of the "good" ones to see if exposure to the elements causes it to fail as well. If that happens, you'll know you have discovered a vulnerability you could do without.

There is always a last resort…

Since the machines will restart via the power buttons, remote controllable relays paralleled across the switches will at least allow you to get the servers back up quickly without having to actually go there. You can use an off-the-shelf internet-connected home automation device added to the router.

OR, you can go old school and find a trustworthy person close by you can call to go push the buttons when needed…

[os9rocks]

@Gary: you might have a point with the ES control panel being corrupted. I'll give it a try. Actually I can remember that re-installing OS 9 was quite frequent (and effective) when something went wrong; however, since we didn't need to do it for several years now, I somehow forgot the thing. I'll try that too on a spare G4.

Also, we discussed internally to install remote controllable relays. It'll be the very last resort, since it requires hours of manual work (testing and installation on at least ten Macs). We might move to OS X with a bit more of work.

BUT I'm not sure I'm ready to quit OS 9, call me a nostalgic... I'm somehow proud of being capable to run modern and complex applications on such old hardware and software. It reveals the very nature of Macs: dependable machines, built for last more than any other computer. Some of them are completely original: can you believe it? A 40 MB HD running for almost twenty years 24/7. I even have a couple of blue&white G3s running WebStar, Lasso and FileMaker... talk about value for money

[ovalking]

I've never actually tried the 'Restart automatically after a power failure' option before... until now.
And this G4 400 (OS 9.2.2) with flat PRAM battery did not restart after I pulled the power at the wall and plugged it back in again.
Now I have to set the clock again!

My P475 has a simple on/off rocker switch - much more reliable!
I recall some other Macs had a power button that you could physically lock into the 'on' position.

[GaryN]

Also, we discussed internally to install remote controllable relays. It'll be the very last resort, since it requires hours of manual work (testing and installation on at least ten Macs). We might move to OS X with a bit more of work.

Hmmm… You described the issue being with "several". If it's actually ten, that is a bit more work.
If I was in your position though, I would definitely work at this some more before giving up on the OS9 that has served (lol) you so well. Moving to OSX can open up a whole new set of vulnerabilities related to all things UNIX - possibly more susceptible to being hacked, or at least crashed, by your anonymous friends.

As I said, try curing just one unit by replacing things step by step.  You might also try just swapping the drives between affected and non-affected units. If a non-affected unit suddenly develops an inability to restart itself, you'll know the system has been corrupted.

[DieHard]

As I said, try curing just one unit by replacing things step by step.  You might also try just swapping the drives between affected and non-affected units. If a non-affected unit suddenly develops an inability to restart itself, you'll know the system has been corrupted.

Excellent Gary !

That is definitely the best and smartest method, I just did a similar thing for a client that was getting a BSOD on a Win 2008 Server and another vendor told him it HAD to be hardware, we put the RAIDED set into an identical BOX and same BSOD, so it was as I had suggested to him that it could be SOFTWARE; and guess what, latest version of Norton Security is causing BSOD on thousands of boxes... random BSOD (Windows is a nightmare, lol). I uninstalled Norton and now... no more BSOD.  Time to go to esat

Anyway, that will definitely narrow down the software or hardware focus

[os9rocks]

Thank you guys. I'll try and come back to let you know. It would be risky to try and swap drives, since most of them are services that can't be let down for long times. But we'll work something out.

Fingers crossed...