jamescuarterodotcom

i am totally and utterly fried. the following are just some quick notes on my past 48 hours at work. to make a long story short, my mail server completely flatlined on me yesterday, i ran into every possible pitfall during the restoration process, pushing my stress level to its limit. on the plus side, everything is back up, my backups were good and i learned a lot from this disaster. it might make for good reference material down the line for someone in a similar predicament. oh, and i just wanted to say that msexchange.org was totally invaluable as a resource.

08/30/06 8:00 AM: BSOD on mail server with ntoskrnl.exe error, same thing on reboot, same thing in safe mode. booted from win2K cd, went into recovery console and ran chkdsk. booted once into OS, found a ton of changes to server registry, malware, etc. started to remove but got same BSOD on subsequent reboots. i wonder if this isn’t related to last week’s ranky infection.

08/30/06 10:00 AM: repair from ERD didn’t work, so as a last resort did an in-place install of win2K. setup hung during the ‘registering components’ phase. shift-F10 worked but access to task manager was disallowed. had to go into regedit and set a value (don’t remember offhand) to 0. tried to kill the fixmapi.exe process but binding failed.

08/30/06 12:00 PM: started setting up an extra win2K3 server that i was saving for later. ran updates, added to network, went to install exchange and realized that exchange 2000 isn’t compatible with win2K3 unless you set it up on a win2K server first and then replicate it to the new server.

08/30/06 02:30 PM: took a decommissioned server, put a new HD in it, maxed out the RAM, installed win2K on it.

08/30/06 06:30 PM: took new server home, ran updates, installed SAV 10. in the meantime, i took the HD from the damaged mail server out, hooked it up to my computer and ran SAV on it, root and WINNT/SYSTEM32 directories were crawling with malware/adware. pulled the files and directories i needed off the HD, scanned and burned them to DVD.

08/31/06 07:30 AM: installed exchange (needed to install NNTP service from add/remove programs > windows components > IIS services before messaging could be installed). created a test mailbox but couldn’t send or receive (had a “directory service” error).

08/31/06 09:00 AM: decided to go ahead and restore the last backup in hopes it would fill in the missing settings. could not dismount the public and private storage containers at first. once i was able to dismount, restoring from backup failed multiple times. ‘allow database to be overwritten’ option in storage container properties needed to be cleared. after successful restore from backup (20 min for 2GB over network - not bad!), could not mount new storage containers. determined there was a problem with the log files. tried copying original log files from backup, still no good. realized that my current version of exchange was only SP1, had to update it to the same version i was restoring. after that, still could not mount new storage containers. ran eseutil /r and eseutil /p to try and recover and repair the logs, neither worked. eventually, i just moved the log files out of the folder entirely, leaving only priv1 and pub1 files. ran eseutil /p again, was then able to mount storage containers.

08/31/06 01:30 PM: mailboxes appeared in exchange system manager and users appeared in active directory on exchange server but no connection was being made between the two. the only red x’s next to mailboxes were for users who had been deleted some time ago. as it turns out, RUS was pointed at old PDC. after changing settings to point to the new PDC, i got red x’s for all of the mailboxes and was able to reconnect each account manually (after i’d reinstalled anti-spam gateway and tested its settings, which had been copied from the old server). threw myself down a flight of stairs.