Google+

Archive

Archive for the ‘Technology’ Category

SUN StorageTek’s RAID 6 Implementation

February 16th, 2009 No comments

Continuing the series on RAID 6 technology with different OEM products; this time around it’s SUN StorageTek version of RAID 6 implementation on 6xxx series arrays.

Here are the links to previous posts related to RAID 6. It covers NetApp’s RAID-DP, Hitachi’s RAID 6, HP’s RAID 6 and technical post on various different RAID technologies.

HP’s RAID 6

NetApp’s RAID–DP

Hitachi’s (HDS) RAID 6

Different RAID Technologies (Detailed)

Different RAID Types

SUN’s Business Case

Similar to HP, Sun is not making a claim on performance with RAID 6; its only claim with this technology implementation on its products is Improved Reliability.

RAID 6 is supported on SUN StorageTek’s 6140, 6580 and 6780 Arrays

SUN StorageTek products traditionally support RAID 0, RAID 1, RAID 3, RAID 5, RAID 1+0 and now RAID 6.

Different Applications that can use RAID 6 technology on SUN StorageTek arrays are Exchange Email, Web Servers, File Servers, Oracle, SAP, SQL and MySQL.

Technology

Here is a small abstract about RAID 6 Technology as discussed in my earlier post.

Technology: Striping Data with Double Parity, Independent Data Disk with Double Parity

Performance: Medium

Overhead: 10% to 50% overhead, with additional drives you can bring down the overhead.

Data Loss: With one drive failure and two drive failures in the same Raid Group no data loss.

Advantages: RAID 6 is essentially an extension of RAID 5 which allows for additional fault tolerance by using a second independent distributed parity scheme (two-dimensional parity). Data is striped on a block level across a set of drives, just like in RAID 5, and a second set of parity is calculated and written across all the drives; RAID 6 provides for an extremely high data fault tolerance and can sustain multiple simultaneous drive failures which typically makes it a perfect solution for mission critical applications.

Disadvantages: Poor Write performance in addition to requiring N+2 drives to implement because of P+Q parity scheme.

The technology (P+Q Parity Algorithm) of Sun’s implementation and HP’s implementation of RAID 6 is similar. To further read about HP’s RAID 6, please use the link below.

HP’s RAID 6

The formula to calculate your usable space with RAID 6 is C * (n – 2), where C is the Size of your smallest drive in the RAID Group and n = number of drives. It is highly recommended all your disk drive sizes are similar.

The largest configuration of RAID 6 can be 257 drives with 2 parity drives and 255 data drives. Usage of more drives will help bring down the RAID 6 overhead since 2 drives are always allocated as parity.

This technology can be implemented with a minimum of 4 drives. The overhead with use of 4 drives in a single RAID Group is 50%. If you run 60 drives in a single RAID group, your overhead might be close to 4% {100 – [100 x 2 (parity) / 60 (drives)]}.

Lower write performance is a unique disadvantage of using RAID 6.

Both Sun StorageTek’s RAID 6 and HP’s RAID 6 offers advantages related to reliability and data protection but no additional benefits with performance, capacity, efficiency, speed, etc.

We will leave it up to the user to decide what RAID technology best fits their environment on SUN StorageTek 6XXX arrays for various different applications.

HP’s RAID 6 (ADG – Advanced Data Guarding)

February 13th, 2009 No comments

Continuing my RAID 6 posts, first HDS’s RAID 6, then NetApp’s RAID-DP and now this time around it’s HP’s RAID-6 ADG.

Some upcoming post would include RAID 6 Technology and its implementation by SUN, IBM and EMC. The final post should be about a comparison of all OEM products and the usage of RAID 6.

Here are the links to previous post related to RAID 6 and data protection

NetApp’s RAID–DP

Hitachi’s (HDS) RAID 6

Different RAID Technologies (Detailed)

Different RAID Types


Will try to keep this post short in terms of overall RAID 6 concepts, rather jump directly into the technical aspects of HP’s RAID-6 ADG (Advanced Data Guarding).

HP’s Business Case with RAID-6 Advanced Data Guarding (ADG)

So Advanced Data Guarding….the name is just perfect…. HP’s pitch to their potential storage customers would include a slide on ADG (I am a
ssuming that is the case).
This cost effective and fault tolerant technology is proprietary to HP and its patented, just cannot find a reference about it on the US PTO’s website.


RAID-6 ADG is supported on the MSA (Modular Smart Arrays) SAN platform.


I believe it’s not supported on any EVA (Enterprise Virtual Array) platforms. No RAID 6 support available on LeftHand Network SAN’s.


With HP XP-24000, HP XP-20000, HP XP-12000 and HP-XP 10000 there is no support for RAID-6 ADG, but there is native support for RAID 6 (dual parity).


HP Storage Products traditionally have support for RAID 0, RAID 1, RAID 1+0, RAID 5 and now RAID-6 ADG. Some LeftHand Network SAN’s support RAID 3 and RAID 4.


The argument from HP is pretty similar to the ones we already discussed with HDS and NetApp in the previous post. The push for RAID 6 at HP comes due to the existence of larger disk size and requirements for fault tolerance to run 24 x 7 x 365 applications.


Since there is an additional parity calculation associated with RAID-6, HP’s recommendation is to use RAID-6 ADG with lower writes and high reads only. If you have an application performing random writes, RAID 6 (ADG) might not be an option for you.


HP’s RAID-6 Advanced Data Guarding (ADG) Technology

Here is a snapshot of how this technology operates.



In the case here, we have 6 disk drives attached on a fiber loop or SCSI bus / controller. Data is striped on Disk 1, Disk 2, Disk 3, Disk 4 and then Parity (P1) and (Q1) are generated and written on Disk 5 and Disk 6. You can assume each data block is 4kb or 8kb in size.

Similarly, as the process continues, the next set of data strips start at Disk 1 then go to Disk 2, Disk 3 and Disk 6, while the parity is written on Disk 4 (P) and Disk 5 (Q). ADG is based on P + Q algorithm to calculate two independent parity sets. Exclusive OR (XOR) is used to calculate the P and Q Parity. The P Parity is exactly like it would be for RAID 5 and Q is calculated based on Error Correcting Code. The Q is then striped across all the disk within the RAID Group.

If a single drive is lost in the Raid Group, data is rebuild using ordinary XOR P (Parity). Also the P and Q are both recalculated for each rebuild block. If a second drive fails during this time, the rebuild takes place using the Q Parity. During these times data is still completely available with a little degradation.

If you do add a spare drive to this configuration, now your raid group can pretty much withstand 3 drive failures before data loss.

This technology can be implemented with a minimum of 4 drives. The overhead with use of 4 drives in a single RAID Group is 50%. If you run 60 drives in a single RAID group, your overhead might be close to 4% {100 – [100 x 2 (parity) / 60 (drives)]}.

The formula to calculate your usable space will be C * (n – 2), where C is the Size of your smallest drive in the RAID Group and n = number of drives. It is highly recommended all your disk drive sizes are similar.

If you are running more than 14 physical drives in a single RAID Group, HP’s recommendation is to use RAID-6 ADG. With 6 drives in a RAID Group, the failure probability is 1E-10. With 60 drives in a RAID Group, the failure probability is 1E-7.

Again HP’s big pitch with RAID-6 ADG is Cost Effectiveness with Fault Tolerance, not really performance.

NetApp’s RAID-DP (Enhanced RAID 6)

February 11th, 2009 4 comments

Continuing the quest on RAID 6, with my previous post related to HDS’s RAID 6, here is a post about NetApp’s RAID-DP (Double Parity – Enhanced RAID 6). In some upcoming posts, we will talk about RAID 6 technology and its usage by EMC and HP. If possible will try to write the final post on comparison between each of the OEM products and how they have leverage the use of RAID 6 technology.


The following are links to my previous posts about RAID Technology

Hitachi’s RAID 6

Raid Technology Continued

Raid Types

NetApp Business Case

Similar to HDS; NetApp’s argument have been about the usage of high capacity disk drives that are FC and SATA (250GB, 300GB, 450GB, 500GB, 750GB and 1TB) which takes quite long time to rebuild and have higher failure rates. During these times the RAID Group might hit a MEDR (Media Errors During Reconstruction) and further cause a complete halt of data rebuild, possibly creating a DL (data loss) situation.


As you know the parity information that is stored on the disk is used to reconstruct the data on the new replaced drive. The time to replace the failed disk plus the time to reconstruct the data can be between 4 hours to 30 hours for these larger drives. It is a high probability, that during these times, there might be a hiccup with a bad sector/block or a legitimate drive failure in the same raid group, which can further cause data loss. The graphs related to the failure probability are in the later part of the post.


With the usage of RAID 6 (RAID-DP), two drive failures in the same RAID group can occur without data loss. NetApp is traditionally known to support RAID 0, RAID 1+0, RAID 4, and now RAID 6 (RAID-DP). The RAID 6 adaptation by NetApp comes in 2006 after HDS’s and HP’s offering to their customers.

Here is a little extract about RAID 4 (Widely used with NetApp)

Technology: Block level parity

Performance: Medium

Data Loss: With one drive failure, no data loss. With multiple drive failures in the same Raid group data loss is imminent.

Advantages: It has the highest Read data transaction rate and with a medium write data transaction rate. Data is stripped on disk creating high efficiency along with a good aggregate transfer rate. Parity is stored on a separate disk.

Disadvantages: Disk failure has medium impact on throughput. Often difficult to rebuild in the event of a disk failure (as compared to RAID level 1) and individual block data transfer rate same as single disk.

Here is an extract about RAID 6 (Remember RAID 6 is not exactly RAID-DP. RAID-DP is an adapted version by NetApp)

Technology: Striping Data with Double Parity, Independent Data Disk with Double Parity

Performance: Medium

Overhead: 20% to 30% overhead, with additional drives you can bring down the overhead.

Data Loss: With one drive failure and two drive failures in the same Raid Group no data loss.

Advantages: RAID 6 is essentially an extension of RAID 4 which allows for additional fault tolerance by using a second independent distributed parity scheme (two-dimensional parity). Data is striped on a block level across a set of drives, just like in RAID 4, and a second set of parity is calculated and written across all the drives; RAID 6 provides for an extremely high data fault tolerance and can sustain multiple simultaneous drive failures which typically makes it a perfect solution for mission critical applications.

Disadvantages: Poor Write performance in addition to requiring N+2 drives to implement because of two-dimensional parity scheme.

Because of the low performance of RAID 6 related to random writes, NetApp has modified the RAID 6 technology and incorporated it into its DATA ONTAP OS as RAID-DP (Double Parity) system. The big pitch from NetApp about Double Parity is, they get better Performance, less overhead and is Cost Effective Capacity Utilization. OEM’s tend to modify RAID technology to better suite their products or enhance it based on factors like speed, rebuild times, efficiency, etc.

RAID-DP is available with NetApp Data ONTAP Operating System version 6.5 and above and is offered across all different NetApp Series of Platforms. There is no added licensing, configuration, special hardware requirements with RAID-DP.

NetApp’s published data loss stats show the following with RAID 5 and RAID-DP.


The data protection offered with RAID-DP is 3800 times better than the closest competition which is RAID 5.

NetApp RAID-DP Technology

RAID-DP is a double parity protection in a RAID group. RAID-DP on NetApp is supported using 14 Data Drives and 2 Parity Disk. A traditional RAID 4 is implemented with horizontal parity structure. On RAID-DP the same principles are used to calculate the parity, the Double Parity (DP) calculations are done diagonally using row components.


With a single disk failure, RAID-DP will treat it as a normal failure and rebuild the new disk giving reconstruction of data a normal priority. With double disks failure, RAID-DP will prioritize the reconstruction of these new disks and finish the process in a shorter duration than a single rebuild.


Read Dave Hitz’s (Vice President of NetApp Engineering Division) Blog about why Double Protection RAID (RAID-DP) doesn’t waste extra disk space.


Consider the following as a simple example of single Parity (RAID 4)



You have data blocks (size possibly 4kb each), writing on 7 separate drives D0, D1, D2, D3, D4, D5, D6. Using simple mathematical equation the parity is generated as an addition of all the data, in this case the elements of a single row are added together to generate the P (Parity) information (follow the color scheme).


The sum of all elements of row 1 (row one) are 23. Now let’s say for example, your D1 drive goes belly up. Using the same equation, the D1 drive is reconstructed as 23 – 1 – 1 – 2 – 8 – 7 – 1 or the mathematical equation P – D6 – D5 – D4 – D3 – D2 – D0 yielding the final result 3.


To make things a little complex let’s look at the actual formula used to generate the Parity.

P = D0 XOR D1 XOR D2 XOR D3 XOR D4 XOR D5 XOR D6

For Row 1 your XOR (Exclusive OR) results are 15.


Now let’s take this same example but let’s add DP (Double Parity) to it and see how that works. Again for the math purposes we will leave the parity calculations based on additions rather than XOR).


The same process as discussed earlier is used to generate the P (Parity) which is the addition of all the elements of a single row D0, D1, D2, D3, D4, D5 and D6.

To generate the DP (Double Parity), diagonally one element from each row is added together. So if you see the above example, to generate DP (1), elements D0 (1), D1 (2), D2 (3), D3 (4), D4 (5), D5 (6), D6 (7) are added together.

The color scheme used above shows how the elements are added to create DP. As you stare at it, it will all make sense….

Note: The bracket above denotes ( ) ROW number.

As pointed earlier, in real life to generate the DP like the P, the row elements are XOR (Exclusive OR) together.

The equation would look like

DP (1) = D0 (1) XOR D1 (2) XOR D2 (3) XOR D3 (4) XOR D4 (5) XOR D5 (6) XOR D6 (7)

For DP (1) our XOR results should yield 13.

As you notice, the DP also includes the P (Parity) drive calculations. But for any given DP, only N numbers of drives are used for generating Double Parity. In short with a 9 drive configuration here, we only used 7 drives to generate the DP. This is because with a double failure, the DP and P will be used to reconstruct the new drives.

Let’s talk about a drive failure with RAID-DP.

Single drive failure, the process would be exactly like the example earlier where parity information is used to reconstruct the new drive.

With 2 drive failures, the use of Row level Parity and Diagonal Parity data is used to reconstruct the new drives. Also ONTAP gives a priority to data rebuild with 2 drive failures in the same RAID Group.

The rebuild process starts at the DP level with 2 drive failures. As it hits a certain Row of DP, it will try to reconstruct the data from DP first and then go to Parity to construct the second drive in a consecutive order. With visual or video that can be explained very easily.

Overhead

Traditionally RAID 6 is known to have 25% to 33% performance overhead with Random Writes. One of NetApp’s arguments to create RAID-DP was to overcome this performance hindrance.


NetApp’s base performance varies between 98% to 100% with RAID-DP.


Disk space overhead for RAID 4 on NetApp is between 18 to 25% depending on the number of drives used in a RAID group. With RAID-DP, the overhead is as low as 7.5% with 28 drive RAID Group. The overhead will increase with fewer drives, where you have 16 drives in a RAID Group.

Additional

If you are using SyncMirror with RAID-DP, 4 drive failures will be allowed before there is any data loss.

If you are currently using RAID 4 with Data ONTAP 7G you can upgrade your volumes to RAID-DP.

A Spare drive is used to create DP in a RAID-DP system.

These days, we do see a lot of customers running RAID 6 with their larger SATA and FC drives. Again I don’t think RAID 6 is the future of storage industry, but it surely is the present of the storage industry and because of it, quite a few OEM’s have jumped on RAID 6 implementation into their products.


Note: One graph above has been extracted from NetApp’s Whitepaper on RAID-DP and Implementation of RAID-DP.

Hitachi's (HDS) RAID 6

February 9th, 2009 2 comments

TO SUBSCRIBE TO STORAGENERVE BLOG

Hitachi (HDS) has been one of the pioneers in implementation of RAID 6 in their storage products. I believe the necessity of RAID 6 at HDS was initially realized back in 2004 with the release of high capacity disk drives and since then they started implementing those in 2005 with the USP’s and then later in the TagmaStore Modular Storage products.

In the next upcoming posts, we will talk about the RAID 6 technology and its usage by different OEMs like HDS, EMC and NetApp. If possible I will try to write the final post on comparison between each of these OEMs and how they have leveraged the use RAID 6.

All the OEM’s tend to modify RAID Technology in their microcode / software / firmware to better fit their product or enhance it based on various factors like speed, rebuild times, etc, prime example will be EMC’s implementation of RAID S with Symmetrix products. NetApp’s implementation of RAID DP with its products.


HDS’s Business Case

RAID 6 is available in Hitachi’s USP, WMS and the AMS disk arrays.

System and Storage Administrators are all very well versed with RAID 5 and has been using it as a standard RAID technology across all Servers and mid tier Storage. With Storage Disk Arrays the need to have RAID configuration is necessary, example RAID 1, RAID 1+0, RAID 3, RAID 5, RAID 6, RAID 10, RAID S, etc.

Hitachi products support RAID 0, RAID 1, RAID 5 and RAID 6.

RAID 5 has been common practice since the last 10 to 15 years. Now the drive sizes during these years varied from 4GB disk to 146GB SCSI or Fiber Disk (which included various different sizes like 4.3GB, 9GB, 18GB, 36GB, 50GB, 73GB and 146GB). These days, seldom you see these size drives, customers are talking about disk sizes that are minimum 300GB (FC or SATA) and go up to 1TB. Over the next 2 to 3 years, we will absolutely see disk sizes that will be between 3TB to 4TB.


Here is an abstract about RAID 5

Technology: Striping Data with Distributed Parity, Block Interleaved Distributed Parity

Performance: Medium

Overhead: 15% to 20% with additional drives in the Raid group you can substantially bring down the overhead.

Minimum Number of Drives: 3

Data Loss: With one drive failure, no data loss. With multiple drive failures in the same Raid group data loss is imminent.

Advantages: It has the highest Read data transaction rate and with a medium write data transaction rate. A low ratio of ECC (Parity) disks to data disks which converts to high efficiency along with a good aggregate transfer rate.

Disadvantages: Disk failure has medium impact on throughput. It also has most complex controller design. Often difficult to rebuild in the event of a disk failure (as compared to RAID level 1) and individual block data transfer rate same as single disk.

RAID 5 also relies on parity information to provide redundancy and fault tolerance using independent data disks with distributed parity blocks. Each entire data block is written onto a data disk; parity for blocks in the same rank is generated on Writes, recorded in a distributed location and checked on Reads.

This would classify as one of the most favorite RAID Technologies of the past.


The rebuild time on drive sizes from 4.3GB to 146GB during off production times can be about 18 to 24 hours, during off production can be close to 4 to 8 hours. There is a risk associated with RAID 5 and having any additional drive failures in the same RAID group.

Let’s say you have a single drive failure in your RAID 5. The vendor picks up the error using the call home feature and dispatches an engineer to come onsite to replace the drive. It’s now 4 hours since the drive has failed. You as a customer haven’t seen any performance impact yet. The drive is replaced and it will take 24 hours to completely sync (rebuild) with (from) its partners in the same RAID group. So now it’s really 28 to 30 hours since your initial drive failure. During this time, if you hit one more roadblock (Read / Write hiccup or a bad sector) in the same RAID group, the data in the RAID group will be lost.

These days the normal drive size is at least 300GB or more. With FC and SATA you can have your drive size variations as 250GB, 300GB, 450GB, 500GB, 750GB and then 1TB being the latest addition. With these larger SATA drives, the rebuild times can go into 30 to 45 hours or in some cases even 100 hours. Now the window where things can really go wrong is much higher. That is one of the reasons quite a few vendors these days have introduced RAID 6.


Here is an abstract about RAID 6

Technology: Striping Data with Double Parity, Independent Data Disk with Double Parity

Performance: Medium

Overhead: 20% to 30% overhead, with additional drives you can bring down the overhead.

Minimum Number of Drives: 4

Data Loss: With one drive failure and two drive failures in the same Raid Group no data loss.

Advantages: RAID 6 is essentially an extension of RAID 5 which allows for additional fault tolerance by using a second independent distributed parity scheme (two-dimensional parity). Data is striped on a block level across a set of drives, just like in RAID 5, and a second set of parity is calculated and written across all the drives; RAID 6 provides for an extremely high data fault tolerance and can sustain multiple simultaneous drive failures which typically makes it a perfect solution for mission critical applications.

Disadvantages: Poor Write performance in addition to requiring N+2 drives to implement because of two-dimensional parity scheme.


Note:

Hitachi does not recommend using RAID 6 drives with high performance applications where extreme random writes are being performed. In some cases, the use of RAID 1 or RAID 1+0 is essential. There is an performance overhead associated with use of RAID 6, we will talk about it later in the post.

Probability of Data Loss with RAID 5 and RAID 6


As you see in the graph, the probability or the percentage of exposure related to RAID 5 double failures is as much as 7.5% while the chance of triple failure in a RAID 6 configuration is 0%. As the drive sizes are increasing, the usage of RAID 6 will become more prominent.


HDS’s Technology


Lets take the above as an example, we have 8 Disk Drives in a USP system.

The D1, D2, D3, D4, D5, D6 represents DATA BLOCKS and P1 and P2 (the Dual Parity).

The data blocks are followed by the parity and then the last parity drive is where the new data blocks starts to write again. With this sequential nature, the vast improvement is seen in the performance of this technology.

To make things a bit more complex and learn this technology lets introduce some mathematical formulas with implemention of RAID 6.


In the above, D0, D1, D2, D3, D4 and D5 are the Data Blocks (Stripes) and P = Calibration data and Q = Secondary Parity

Using mathematical formula’s with the Data Stripes (D0, D1, D2, D3, D4 and D5) and XOR (Exclusive OR), the P (Calibration data) is generated.

P = D0 XOR D1 XOR D2 XOR D2 XOR D4 XOR D5

Q is the product of Coefficent and Data Stripes (D0 through D5) XOR

Q = A0 * D0 XOR A0 * D1 XOR A0 * D2 XOR A0 * D3 XOR A0 * D4 XOR A0 * D5

Typically with one drive failure the P (Calibration Data) is used to Generate or rebuild the new drive, with two drive failures, the P and Q data is used to rebuild the new drive.


Risk

Here is a nice chart the shows the Risk associated with RAID 5 and RAID 6


As times elapse with the drive failure on RAID 5 (with time to respond and rebuild times), the risk associated tends to increase.

With RAID 6 and a drive failure the risk associated tends to stay the same and at 0%.

Based on different Raid Group Size, here is the risk of data loss with rebuild times.

As you can see in both the graphs, the risk associated with RAID 6 is pretty much zero percent.


Overhead

As discussed earlier, there is an additional overhead with usage of RAID 6 vs usage of RAID 5. But the risk associated with using RAID 5 is much higher than the overhead consumption by RAID 6. Here is a graph that shows the overhead associated with RAID 6.

As you see in the graph, the overhead with 6 Data drives and 2 Parity drives is only 25%. If you were running Mirroring or some other variation of RAID 5, the overhead can be between 50% to 25%. So in short even with 2 parity drives the advantages are quite greater with use of RAID 6.

From a performance standpoint, the performance of RAID 5 and RAID 6 is pretty similar when we talk about Random Read, Sequential Read and Sequential Write workloads. There is added penalty when we talk about Random Write workloads, that is because of the two dimensional parity. Compared to RAID 5, RAID 6 takes a 33% Performance hit on Hitachi with Random Write workloads.

To sum up, if you are using high capacity disk drives on your Hitachi Systems and are looking to mitigate failures, it is highly recommended you use RAID 6 on these systems.

RAID 6 is a great technology, may be the technology of present, but the future of RAID will go to a different place. Imagine you have a 20TB drive (reality of 2012 – SATA), how long will it take to rebuild that and what is the risk of triple fault with it.



Note: The graphs above have been obtained from two different documents – Hitachi’s RAID 6 Protection and Hitachi’s RAID 6 Storage Doc.