True IT – Storage Stories: 7 (Data Wipe on the wrong machine)
Yea you are right, exactly as the title of this post says, Data Wipe got performed on the wrong machine.
The CE got permission from the customer to perform data wipe on a storage system. The host were retired, storage was ready to be turned off, but part of the procedures, customers typically require that all the data from the drives be cleaned.
The CE took this opportunity to remotely connect into the customers storage system. Thought it will take several hours to finish the process when he would go onsite to physically power off the storage system. Thought he logged into the machine he intended to, he got through to another one. Logically started taking the ports down through soft commands and then within 15 mins kicked off the process for data wipe.
An hour later, a SEV 1 ticket is opened at the customer site with major issues in the storage environment. To his luck he thought that while he is out taking care of the issue, he will also check on the data wipe and physically power off the storage subsystem.
On his way to the customer site, he gets a call from the Level 3 folks from the vendor support team on what they had just found on this storage system, that it was busy doing a data wipe and there was no way to stop it.
The realization set in for the CE…….that he had started a data wipe on the wrong storage system without performing the correct procedures.
Set a corporate wide policy on how storage and server teams can perform certain task onsite and remotely. Set similar procedures with vendor teams as well.
The CE ended up losing his job, though there was no way to recover the data on the storage subsystem except for bringing the backup tapes out.
One could have the best storage execution plan in place to manage the storage environment, but is there a way to avert these exceptional cases.