My Server Crashed, now what?

4 Steps to help you repair a bad media server

In this Post “My Server Crashed, now what?” –  I’m sharing last week’s experience when called out to a customer site and found myself dealing with this predicament; of how long should I spend on trying to repair the corrupted database or rebuild it from scratch?.

This particular customer had experienced a power outage and tried booting the Avaya Aura Messaging Del R610 Server-Blade multiple times by pushing the power button. The server was getting stuck on the boot-process. The environment consisted of three Dell Blades using RAID technology to mirror its database HDDs, two of the Dell Servers serving as MASs (Message Application Servers) and the defective one acts as a MSS (Message Store Server).

I came up with this 4 steps=

• 1.- How long is too long?
• 2.- Making the decision
• 3.- Tools I use
• 4.- Get to work

1.- How long is too long?

If you are like me, you can get consumed on trying different ways to get the defective system repaired. from running different maintenance commands to trying to re-invent the wheel by coming up with different ideas that you know down deep inside they might not work, but you still want to experiment them. That is why we do this type of work. We love the challenge.

So how long is too long then? – Every customer or situation varies. I personally ask the customer how much down time do I have to bring the servers back on line. Once I have an estimated time frame, I give myself 20% of the total down time to troubleshoot, within this time you should know if your troubleshooting attempts will actually work.

2.- Making the decision

By now you have your system repaired or maybe not. If you need to reload/rebuild your database it is time to do some preliminary work, from getting backups, ISOs, Patches, Licenses, and update your site contact with your results and plans to get the server back up.

3.- Tools I Use

• I have a Windows base computer therefore I need a SSH Client (PuTTy) to be able to access the system through the services port located in the back of the R610.
• Cross-over cable is needed if you are direct-connected to the services port.
• Monitor and Keyboard are a most to get accurate diagnostic during the boot sequence of the Dell R610 Server, this will help you accurately get a real vision of the system’s health and access to the utility server tools, BIOS, etc.
• MD5Summer – this program will be able to check your download and match it the corresponding hex value given by Avaya when downloaded from the support.avaya site.
• DVD-R – Yes you will need a DVD-R blank disc to burn your ISO image.
• ImgBurn – I personally love this program, I am able to burn my ISOs using this app without any issues. they work 100% of the times I’ve tried them.
• Telnet – I use telnet for the initial server configuration connected the services port

4.- Get to Work

these are the 9 steps that I normally take when rebuilding a System Platform Server and VMs.

Now that you have exhausted your resources troubleshooting and tried multiple things to get the old data back, it’s time to go to plan B. Rebuild your server from factory defaults.

1.-Insert your DVD and Power your Server

In my case working with the MSS, I loaded the System Platform 6.2 ISO DVD into the DVD-ROM Drive and reboot the server by pressing Ctrl+Alt-Del or just press the Power button located to the left of the Dell R610.

2.-Check your existing hardware

While booting the server will go though his hardware health routine check, verifying his inventory is working 100%.

3.- Connect your Service Laptop

then it will prompt you “waiting for service laptop” or something alone
those lines, at this point you  want to connect your laptop to the Services Port.

4.- Initial Configuration

Using Telnet go through the initial installation process. (for a complete step by step SP Server configuration visit this link).

5.- Loading Templates

Install and configure your VMs. In the case of AAM make sure your template is Standard or High-Capacity.

6.- Patch updates

7.- Recreate and install licenses

8.- Restore your database

(note, if you are running a RAID configuration all servers will copy their existing database to your new server, so be careful when reloading backups, this might cause a loss of the most recent data written to the MAS database.

9- Verify server roles, license status, server status, alarms and errors.

If the server is booting up to the wrong screen, what do you do?