Tux

...making Linux just a little more fun!

RAID Simulators for Linux

Amit Kumar Saha [amitsaha.in at gmail.com]


Tue, 1 Jan 2008 21:25:22 +0530

Hello!

Does any one here have experience using a RAID Simulator or know of any? Basically, what I am looking for is 'emulating' RAID on a single computer, single hard disk.

Looking forward to some insights

Regards, Amit

-- 
Amit Kumar Saha
Writer, Programmer, Researcher
http://amitsaha.in.googlepages.com
http://amitksaha.blogspot.com

Top    Back


Jim Jackson [jj at franjam.org.uk]


Tue, 1 Jan 2008 17:09:50 +0000 (GMT)

On Tue, 1 Jan 2008, Amit Kumar Saha wrote:

> Hello!
>
> Does any one here have experience using a RAID Simulator or know of
> any? Basically, what I am looking for is 'emulating' RAID on a single
> computer, single hard disk.

check out lvm - logical volume management. It doesn't do all RAID modes but does mirroring. There's an IBM developerworks article at

http://www.ibm.com/developerworks/linux/library/l-lvm/

which might be a bit long in the tooth nowadays, but is ok as an intro.

>
> Looking forward to some insights
>
> Regards,
> Amit
>
> --
> Amit Kumar Saha
> Writer, Programmer, Researcher
> http://amitsaha.in.googlepages.com
> http://amitksaha.blogspot.com
>
> +-+--------------------------------------------------------------------+-+
> You've asked a question of The Answer Gang, so you've been sent the reply
> directly as a courtesy.  The TAG list has also been copied.  Please send
> all replies to tag@lists.linuxgazette.net, so that we can help our other
> readers by publishing the exchange in our monthly Web magazine:
>              Linux Gazette (http://linuxgazette.net/)
> +-+--------------------------------------------------------------------+-+
> _____________________________________________
> TAG mailing list
> TAG@lists.linuxgazette.net
> http://lists.linuxgazette.net/mailman/listinfo/tag
>


Top    Back


Amit Kumar Saha [amitsaha.in at gmail.com]


Wed, 2 Jan 2008 00:11:21 +0530

On 1/1/08, Jim Jackson <jj@franjam.org.uk> wrote:

>
>
>
>
> On Tue, 1 Jan 2008, Amit Kumar Saha wrote:
>
> > Hello!
> >
> > Does any one here have experience using a RAID Simulator or know of
> > any? Basically, what I am looking for is 'emulating' RAID on a single
> > computer, single hard disk.
>
> check out lvm - logical volume management.
> It doesn't do all RAID modes but does mirroring.
> There's an IBM developerworks article at
>
>    http://www.ibm.com/developerworks/linux/library/l-lvm/
>
> which might be a bit long in the tooth nowadays, but is ok as an intro.

Its a good intro on LVM, but actually what I am looking at is simulating a RAID environment on a single Linux box.

Any suggestions?

Thanks, Amit

-- 
Amit Kumar Saha
Writer, Programmer, Researcher
http://amitsaha.in.googlepages.com
http://amitksaha.blogspot.com

Top    Back


Jimmy ORegan [joregan at gmail.com]


Tue, 1 Jan 2008 19:34:42 +0000

On 01/01/2008, Amit Kumar Saha <amitsaha.in@gmail.com> wrote:

> Its a good intro on LVM, but actually what I am looking at is
> simulating a RAID environment on a single Linux box.
>
> Any suggestions?
>

Have you tried qemu?


Top    Back


Francis Daly [francis at daoine.org]


Tue, 1 Jan 2008 19:41:44 +0000

On Wed, Jan 02, 2008 at 12:11:21AM +0530, Amit Kumar Saha wrote:

> On 1/1/08, Jim Jackson <jj@franjam.org.uk> wrote:

Hi there,

> > check out lvm - logical volume management.
> > It doesn't do all RAID modes but does mirroring.
> > There's an IBM developerworks article at
> >
> >    http://www.ibm.com/developerworks/linux/library/l-lvm/
> >
> > which might be a bit long in the tooth nowadays, but is ok as an intro.
> 
> Its a good intro on LVM, but actually what I am looking at is
> simulating a RAID environment on a single Linux box.

I thought it was a useful answer, particularly if you follow the links to the (similarly old) articles on Linux software RAID.

The precise details may not be relevant to your specific distribution and kernel, but the fundamentals of "what is RAID, and why do I want it?" remain.

> Any suggestions?

What do you think you mean by "simulating a RAID environment"?

RAID == Redundant Array of Inexpensive Disks (or something similar).

From a kernel point of view, that means "multiple block devices", which can mean "different disks" or "different partitions on a disk" (or "different partitions on different disks").

So: decide what physical block devices you have available; decide which of the many different types of RAID you want to play with today; and find your distribution's utilities that probably begin "raid" or "md" (multiple disk, or maybe multiple device) to start playing.

Once you recognise that you can partition your /dev/sdb into /dev/sdb1 and /dev/sdb2, then what the RAID part does is to join /dev/sdb1 and /dev/sdb2 into a new logical entity /dev/md0. Outside the RAID config, you ignore /dev/sdb[12], and only use /dev/md0 as a single block device onto which you (typically) create a filesystem, trusting the lower-level code to mirror/stripe/whatever the bits the way you asked it to.

(Granted, using two partitions on a single disk may not be the cleverest for redundancy, resilience, or speed; but once you understand the concept, you are better placed to choose what partitions to use for what part of your simulation.)

LVM can be used to join multiple physical disks or partitions into a single "disk" (volume group); and can then split that again into "partitions" (logical volumes).

And then, if you really want to, you can use the md-utils on those logical volumes as if they were /dev/sdb1 and /dev/sdb2 in the earlier suggestion.

The disadvantage of doing all of that is that you'd have to learn both lvm and md. The advantages are that lvm is more convenient than fdisk in resizing your "partitions" (so you can start over more easily), and that lvm can possibly do what you were wanting to do with RAID without directly involving md at all.

And depending on your interpretation, it is possible that none of this is "simulating RAID", but is "real, live RAID", especially if you are using inexpensive disks in a redundant configuration.

Good luck,

f

-- 
Francis Daly        francis@daoine.org

Top    Back


Jim Jackson [jj at franjam.org.uk]


Tue, 1 Jan 2008 20:59:20 +0000 (GMT)

On Wed, 2 Jan 2008, Amit Kumar Saha wrote:

> On 1/1/08, Jim Jackson <jj@franjam.org.uk> wrote:
>>
>> On Tue, 1 Jan 2008, Amit Kumar Saha wrote:
>>
>>> Hello!
>>>
>>> Does any one here have experience using a RAID Simulator or know of
>>> any? Basically, what I am looking for is 'emulating' RAID on a single
>>> computer, single hard disk.
>>
>> check out lvm - logical volume management.
>> It doesn't do all RAID modes but does mirroring.
>> There's an IBM developerworks article at
>>
>>    http://www.ibm.com/developerworks/linux/library/l-lvm/
>>
>> which might be a bit long in the tooth nowadays, but is ok as an intro.
>
> Its a good intro on LVM, but actually what I am looking at is
> simulating a RAID environment on a single Linux box.
>
> Any suggestions?

yes. LVM. that's why I answered.

LVM doesn't care where the partitions are is managers. So on the one disk you can mirror (one of the raid modes) 2 partitions quite happily. You can stripe etc etc. Of course there will be a performance hit 'cso of the one disk.

Maybe I misunderstood what you meant by "simulating". If so, you'll have to be a lot more specific about what you want to do.


Top    Back


Peter Knaggs [peter.knaggs at gmail.com]


Tue, 1 Jan 2008 16:42:36 -0800

On Jan 1, 2008 7:55 AM, Amit Kumar Saha <amitsaha.in@gmail.com> wrote:

> Hello!
>
> Does any one here have experience using a RAID Simulator or know of
> any? Basically, what I am looking for is 'emulating' RAID on a single
> computer, single hard disk.

Hi Amit,

Well, let's say you want to use a single file from your filesystem, and make it available as multiple linux block devices?

From there, you could use whatever RAID software you like, feeding it these tasty new block devices. Sound tempting?

OK, so let's start with creating the single file in your filesystem. Let's make it blank, of size 1GB:

dd if=/dev/zero of=./somefile bs=1024k count=1024
losetup /dev/loop1 ./somefile
fdisk /dev/loop1
Here, let's create a few 100MB partitions, e.g.:

Disk /dev/loop1: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x68ed631d
 
      Device Boot      Start         End      Blocks   Id  System
/dev/loop1p1               1          10       80293+  83  Linux
/dev/loop1p2              11          20       80325   83  Linux
/dev/loop1p3              21          30       80325   83  Linux
Now you can use the handy script by Christophe Saout http://lkml.org/lkml/2003/7/18/141 (also attached to this email, I needed to change /bin/sh to /bin/bash for it to work on Ubuntu, apparently because /bin/sh is a symlink to a speedier stripped-down shell used for booting called "dash"), to make each of these "partitions" of your file into a linux block device, using device-mapper:
  dmsetup-partitions.sh /dev/loop1
The output would be e.g.

loop1p1 63 160587
loop1p2 160650 160650
loop1p3 321300 160650
and now you have these new block devices to play with:

/dev/mapper/loop1p1
/dev/mapper/loop1p2
/dev/mapper/loop1p3
This method can also be handy for "looking inside" raw backups of flash memory sticks or other kinds of image files which have been hastily dumped using "dd", and have a partition table at the beginning.

For course, this answer might be completely unrelated to what you're looking for with your original question.

Cheers, Peter.


Top    Back


Amit Kumar Saha [amitsaha.in at gmail.com]


Thu, 3 Jan 2008 09:24:14 +0530

On 1/2/08, Jimmy O'Regan <joregan@gmail.com> wrote:

> On 01/01/2008, Amit Kumar Saha <amitsaha.in@gmail.com> wrote:
> > Its a good intro on LVM, but actually what I am looking at is
> > simulating a RAID environment on a single Linux box.
> >
> > Any suggestions?
> >
>
> Have you tried qemu?

Are you telling that I simulate multiple computers using 'qemu' and then combine all their 'virtual HDDs' to form a RAID array.

Correct me, if I am wrong.

Thanks, Amit

-- 
Amit Kumar Saha
Writer, Programmer, Researcher
http://amitsaha.in.googlepages.com
http://amitksaha.blogspot.com

Top    Back


Amit Kumar Saha [amitsaha.in at gmail.com]


Thu, 3 Jan 2008 09:37:16 +0530

On 1/2/08, Francis Daly <francis@daoine.org> wrote:

> And depending on your interpretation, it is possible that none of this is
> "simulating RAID", but is "real, live RAID", especially if you are using
> inexpensive disks in a redundant configuration.

"Real, live RAID"- it is.

Just to put things into better perspective and context, I am working on a project which requires the design of a tool for "Trend analysis of RAID Services.." which will enable better "observability" of systems in a large scale computing infrastructure which will have "live" RAID implemented to ensure more reliable data storage.

Now, since such an infrastructure is absent at my college, my project mentor suggested me that there are "RAID simulators" available which help emulate RAID devices. The simulated RAID environment will then be uses to emulate disk failures, and other dummy RAID operations to enable an "analysis" and hence the design of the application .

I think, i have put my query into better perspective now.

All comments, suggestions are welcome!

Thanks, Amit

-- 
Amit Kumar Saha
Writer, Programmer, Researcher
http://amitsaha.in.googlepages.com
http://amitksaha.blogspot.com

Top    Back


René Pfeiffer [lynx at luchs.at]


Thu, 3 Jan 2008 11:01:03 +0100

On Jan 03, 2008 at 0937 +0530, Amit Kumar Saha appeared and said:

> On 1/2/08, Francis Daly <francis@daoine.org> wrote:
> > And depending on your interpretation, it is possible that none of this is
> > "simulating RAID", but is "real, live RAID", especially if you are using
> > inexpensive disks in a redundant configuration.
>
> "Real, live RAID"- it is.

Real and live RAID will have different implementations, especially for RAID levels such as 4, 5 or 6. You have to be careful picking test systems. A colleague of mine is specialised in data recovery and told me that RAID5 controllers from different vendors have their own way of distributing the data among the disks. This might be an issue when it comes to comparing real and live RAID systems.

> [...]
> Now, since such an infrastructure is absent at my college, my project
> mentor suggested me that there are "RAID simulators" available which
> help emulate RAID devices. The simulated RAID environment will then be
> uses to emulate disk failures, and other dummy RAID operations to
> enable an "analysis" and hence the design of the application.

The easiest way to do this is to use virtualised GNU/Linux systems with multiple virtual disk partition combined by the Linux software RAID. mdadm can simulate drive failures (http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html). This setup misses the fun caused by stubborn hardware, but it's a start. I don't know if this is sufficient for your testing environment.

Best wishes, René.


Top    Back


Kapil Hari Paranjape [kapil at imsc.res.in]


Fri, 4 Jan 2008 09:26:08 +0530

Hello,

On Thu, 03 Jan 2008, Amit Kumar Saha wrote:

> Just to put things into better perspective and context, I am working
> on a project which requires the design of a tool for "Trend analysis
> of RAID Services.." which will enable better "observability" of
> systems in a large scale computing infrastructure which will have
> "live" RAID implemented to ensure more reliable data storage.
> 
> Now, since such an infrastructure is absent at my college, my project
> mentor suggested me that there are "RAID simulators" available which
> help emulate RAID devices. The simulated RAID environment will then be
> uses to emulate disk failures, and other dummy RAID operations to
> enable an "analysis" and hence the design of the application .

As suggested use loopback devices or "real" partitions to create pseudo-disks. You can emulate the failure of one or more disks in a number of different ways:

	a. write directly to some blocks on the block device
	   associated with one of the pseudo-disks.
	b. use blockdev to "suddenly" mark the blockdevice
	   "read-only".
	c. use an external USB disk as one of the elements of the
	   array and un-plug it during operation.
	d. use LVM to interleave a "zero/error" LVM device with
	   actual read/write disk space. Use that as one of
	   your pseudo-disk devices.
	e. suspend your array and replace one of the pseudo-devices
	   with another such device whose data is almost but not
	   entirely identical with the one before. Then resume the
	   array.
and other such innovative techniques!

Have fun,

Kapil. --


Top    Back


Amit Kumar Saha [amitsaha.in at gmail.com]


Fri, 4 Jan 2008 12:31:00 +0530

On 1/3/08, Ren? Pfeiffer <lynx@luchs.at> wrote:

> On Jan 03, 2008 at 0937 +0530, Amit Kumar Saha appeared and said:
> > On 1/2/08, Francis Daly <francis@daoine.org> wrote:
> > > And depending on your interpretation, it is possible that none of this is
> > > "simulating RAID", but is "real, live RAID", especially if you are using
> > > inexpensive disks in a redundant configuration.
> >
> > "Real, live RAID"- it is.
>
> Real and live RAID will have different implementations, especially for
> RAID levels such as 4, 5 or 6. You have to be careful picking test
> systems. A colleague of mine is specialised in data recovery and told me
> that RAID5 controllers from different vendors have their own way of
> distributing the data among the disks. This might be an issue when it
> comes to comparing real and live RAID systems.
>
> > [...]
> > Now, since such an infrastructure is absent at my college, my project
> > mentor suggested me that there are "RAID simulators" available which
> > help emulate RAID devices. The simulated RAID environment will then be
> > uses to emulate disk failures, and other dummy RAID operations to
> > enable an "analysis" and hence the design of the application.
>
> The easiest way to do this is to use virtualised GNU/Linux systems with
> multiple virtual disk partition combined by the Linux software RAID.
> mdadm can simulate drive failures
> (http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html). This setup misses
> the fun caused by stubborn hardware, but it's a start. I don't know if
> this is sufficient for your testing environment.

Thanks, Rene. For the moment, Virtualized GNU/Linux systems is possibly the way to start off my experiments. The Software RAID How-To will be a good starting point, and I plan to use 'qemu' for my works.

Regards, Amit

-- 
Amit Kumar Saha
*NetBeans Community
Docs Coordinator*
Writer, Programmer, Researcher
http://amitsaha.in.googlepages.com
http://amitksaha.blogspot.com

Top    Back