Subject: RE: SCSI and EIDE

From: Al Youngwerth <[email-masked]>
Date: Wed, 14 Aug 1996 09:58:58 -0600

Sorry, I just had to get in my two bits about this...

I believe the logic for reordering SCSI IO requests really does belong in the drive. With regard to the performance concerns of the drive's CPU speed for making decisions about the next IO to process: an average seek is 10ms and average latency (time waiting for the drive to spin around to the right sector) is 5.5ms (time for one revolution of a 5400rpm drive divided by 2); on average, the drive's CPU has 15.5ms to think about how to order IOs received while the current IO is being processed. Lots-o-time for your average 16-bit embedded controller.

The drive's CPU can always make better decisions about reordering requests because it knows the details of the drive's parameters and its current state. A drive's CPU can reorder requests based upon seek AND latency; the system's CPU cannot.

Given equal drive parameters, the IO throughput (what matters for multi-tasking OSs) for a SCSI-2 system should be better than EIDE. Where SCSI really begins to shine in IO throughput is using disk block striping (RAID 0) using two or more SCSI disks. You can place 7 disks on a single SCSI bus and come no where near saturating the SCSI bus bandwidth with the random IOs that Linux generates. If one SCSI drive gives you 60 IOs/sec of throughput, seven of those drives gives you 420 IOs/sec of throughput. I only wish it was that easy to increase my CPU performance!

Al Youngwerth
[email-masked]

----------
From: 	Rob Browning[SMTP:[email-masked]]
Sent: 	Tuesday, August 13, 1996 6:19 PM
To: 	Debian User List
Subject: 	Re: SCSI and EIDE

Sherwood Botsford <[email-masked]> writes:

> I agree that you can do this in the OS, but I don't think that it
> *should* be done in the OS.
> 0.	In general smarts should be at the point they are used. We had a
> VAX that was about as speedy as a 12 MHz 286 with 287 co-processor.
> However, that vax could handle 8 simultaneous 19 KB terminal lines.
> How?  Smart serial cards.  
> 	The CPU should do those tasks that are either too general for
> dedicated hardware, or that don't happen often enough to take up much
> time.
> 1.	The OS shouldn't have to care about the layout of the disk.

Not trying to start a flame war here, but if the overhead to the cpu
is minimal, which I would guess it is for seek algorithms, you are
often better off doing it with the cpu.  Mainly because changing the
algorithms, fixing bugs, trying out new approaches, etc. can happen
with a simple kernel recompile, and improvment's in CPU speed (which
happen all the time, as opposed to the somewhat slower embedded
controller market) immediately translate to improvements in algorithm
speed.  Now if the disks just had some flash ROM and a 386 (or
equivalent that gcc knows about) on board, things might be different.
Either that or a couple of cheap 386's on the motherboard for stuff
like this...

I'm mostly just speculating.  I don't know that much about drive
hardware.

--
Rob

---

From: Rob Browning <[email-masked]>
Date: 14 Aug 1996 11:50:43 -0500

> Sorry, I just had to get in my two bits about this...

No problem.

> The drive's CPU can always make better decisions about reordering =
> requests because it knows the details of the drive's parameters and
> its = current state. A drive's CPU can reorder requests based upon
> seek AND = latency; the system's CPU cannot.

I see your point.  I was mostly just voicing a general preference
about avoiding embedded systems.  In this case using the embedded
control might be warranted, but in general, embedded systems are a
real pain to deal with, and most people can't do much to fix them when
they're broken.

--
Rob

---

From: Ricardo Kleemann <[email-masked]>
Date: Wed, 14 Aug 1996 13:28:01 -0700

Good point about RAID0.

I know linux's implementation of RAID0 is with the MD program. Do you 
know if that's fully functional, and if so, is there a Debian package for 
it? How is it used?

---

From: Rob Browning <[email-masked]>
Date: 15 Aug 1996 09:43:55 -0500

> know if that's fully functional, and if so, is there a Debian package for 

I'm using it on two machines in the lab.  One has two identical
Quantum Fireballs, and the other has a Fireball and a Western Digital
Caveat (ahem Caviar).  On each machine most of the partitions are
striped, but at least one is not.  It works fine.

You need to compile a kernel with bultiple device/RAID0 support and
you need to install the debian mdutils package.  Then read the stuff
in /usr/doc/mdutils.  That's it.  The md package doesn't care if the
drives are the same type, or even on the same kind of bus.  That's the
advantage to a software implementation.  Granted I would expect
performance to be best when you have 2 identical drives on the same
kind of bus...

--
Rob

---

From: Ricardo Kleemann <[email-masked]>
Date: Thu, 15 Aug 1996 10:03:40 -0700

That's wonderful!

Now will Linux implement anything greater than RAID0? 
Would you say your performance is significantly increased with striping?

How many drives can be striped?

---

From: Rob Browning <[email-masked]>
Date: 15 Aug 1996 13:03:44 -0500

> How many drives can be striped?

That I don't know.  I'm not sure there's a (small) limit.  I didn't
find one in a cursory glance of the manpage for mdadd, or in a
similarly cursory look at the stuff in /usr/doc/mdutil.  md just
groups a number of physical disk partitions into one logical one,
/dev/md*.  Then you treat /dev/md* as the actual device, and you can
mkswap it, mke2fs it, whatever.  It doesn't state any limitations on
the number of partitions that can be added to a /dev/md*.  Note though
that with RAID0, you can't add more partitions after you have data on
an md device, and you have to create the device in exactly the same
fashion every time.  Normally this happens at boot from /dev/mdtab.

--
Rob

---

From: Rob Browning <[email-masked]>
Date: 15 Aug 1996 16:52:50 -0500

> Would you say your performance is significantly increased with striping?

It also currently handles RAID1, and linear; higher levels have been
temporarily disabled because they are still too buggy.

I don't know about performance.  The machines certainly *seem* faster,
but I didn't do any real testing.  I would have done hdparm -t, but it
turned out it wouldn't work for md devices.

--
Rob

---

From: Todd Tyrone Fries <[email-masked]>
Date: Fri, 16 Aug 1996 08:28:04 -0500 (CDT)

> > Now will Linux implement anything greater than RAID0? 
It does.  Raid1 is in development.  It does partial mirroring using about 1/3rd
of the disk space for 'backup' data.

> > Would you say your performance is significantly increased with striping?
I have an ncr53c815 card with two 312mb XT 1986 scsi-1 SLOW maxtor drives.
They definately perform much better after being raid0'ed.  If there would
be a point at which the data read from a drive was faster than the cpu could
crunch the raid0 'virtual' device, it would be slower than no raid at all.  I
personally don't see how any setup could be this 'disk fast' and 'cpu slow'.

>From my experience it is significantly faster.  It is literally (for me)
doubling the number of read-heads that can be reading or writing data at a
given time.

> > How many drives can be striped?

You can have 4 raid devices. /dev/md0 - /dev/md3.  I am not aware of a
limitation on the number of 'block' devices that can be grouped under each
raid device.  I would also be surprised if you were not able to increase the
numbers if you needed to.  Reminds me of ppp/slip in it's early stages.  A
default limit of 4.  Then they raised it to 16, and finally to 
on-the-fly-creation of devices up to a default limitation of 256, with an
easy #define to change to increase this limitation.  I see no reason why the
raid devices will not follow this pattern.

> md just groups a number of physical disk partitions into one logical one,
> /dev/md*.
I'm being picky, but no, it groups block devices.  I couldn't get it to work
with a loopback device, so perhaps right now block device = hard drive.  But
it shouldn't have to.

One happy raid0 user.
--
Todd Fries .. [email-masked]

---

From: Ricardo Kleemann <[email-masked]>
Date: Fri, 16 Aug 1996 12:55:50 -0700

Hi guys,

starting a new thread about the subject... ;-)

I'm still trying to get a grip on this...

Few questions:
1. can raid0/raid1 be done on either scsi or ide or both?
2. what's the difference between raid0 and raid1?
3. what exactly does it do? Does it mirror data accross multiple devices, 
and if so, how does it maintain the data in all devices?
4. How does one go about "creating" an md device? Would it automatically 
mirror a "non-md" drive into the multiple devices?

Thanks!
Ricardo

---

From: Rob Browning <[email-masked]>
Date: 16 Aug 1996 17:34:45 -0500

> 1. can raid0/raid1 be done on either scsi or ide or both?

The linux md driver is software, so it doesn't care what the
underlying block device technology is.  You just give it block-device
partitions to turn into a RAID device.  These partitions can (at the
least) be on SCSI or IDE devices.  It dosnt' matter to the md driver.

>From /usr/doc/mdutils/README:

 Raid0 does a classic (and rather efficient) striping on disks
 (i.e. contiguous blocks on the md device are spread across real
 devices).  It gives rather good performances on SCSI disks, specially
 with concurrent disk access.  There's no limitation on disks sizes
 (i.e. sizes can be different, md will cope with this).
 
 Raid1 adds mirroring to raid0 striping.  Note that it is not complete
 yet (no rebuild tools, error trapping is incomplete).  It's also known
 to be rather slow when writing. 

>From the same source:

 Since MD pre0.31, RAID-[15] have been removed from the main
 distribution, because of pathological unstability... and for the
 kernel integration of linear and RAID-0 modes.

> mirror a "non-md" drive into the multiple devices?

Get the debian mdutils package install it, and read the docs.

--
Rob

---

From: Todd Tyrone Fries <[email-masked]>
Date: Fri, 16 Aug 1996 22:24:24 -0500 (CDT)

> mirror a "non-md" drive into the multiple devices?

Please read the manpage for the 'mdcreate' utility.  It answers all of these
questions.  That said, I'll attemt to exlain it in my own words.

raid0/raid1/linear are currently functional raid personalities that can be
used for any block device.  floppy+ide+scsi+zipdrive+whatever.  I don't know
if you could use a ramdisk, but I wouldn't be surprised.  ANY BLOCK DEVICE.

So, for my computer, I have:
/dev/hda hda1(1mb)  hda2(90mb)  hda3(50mb) hda4(60mb)
/dev/sda sda1(45mb) sda2(265mb) sda3(7mb)
/dev/sdb sdb1(45mb) sdb2(265mb) sdb3(7mb)

Long story on the reasoning behind this, but anyway, /etc/mdtab contains:

   /dev/md0 raid0,4k,0,72e5f713 /dev/sdb1 /dev/hda6 /dev/sda1
   /dev/md1 raid0,4k,0,7b8942ce /dev/sdb2 /dev/sda2

Which gets me an output of 'df' as follows:

   LightHouse~:$ df
   Filesystem     1024-blocks  Used Available Capacity Mounted on
   /dev/hda2          89183   79028     5550     93%   /
   /dev/md0          149794  131671    18123     88%   /home
   /dev/md1          515289  458539    30134     94%   /usr

(the 50mb partition on ide is swap).

Notice I am creating a /usr partition that is bigger than any one of my
drives alone, and also am using ide + scsi on the /home partition.

[ a note to those who would suggest I'm crazy to swap on ide when I have
  scsi-->  my ide is 2x faster than my scsi disks because they are VERY old ]
[ a note to those who wonder about the 2 unused 7mb partitions on my scsi
  disks--> the 2nd scsi disk has problems reading the last 7mb of the drive
  and causes lockups.  So I decided to partition the drives around the 'bad'
  blocks. ]

Anyway, raid (or multiple device, md) in linux has 3 personalities availble
at this time:
   linear = appending all the block devices together.  So if you had
             md0 = /dev/hda1 + hdb3 + sda, the blocks would be accessed
              in this order:

              /dev/hda1 = { block1, block2, block3 }
              /dev/hdb3 = { block4, block5 }
              /dev/sda  = { block6, block7, block8 }

   raid0 = software disk striping.  So if you had
             md0 = /dev/hda1 + hdb3 + sda, the blocks would be accessed
               in this order:

              /dev/hda1 = { block1, block4, block7 }
              /dev/hdb3 = { block2, block5 }
              /dev/sda  = { block3, block6, block8 }

   raid1 = software disk striping + recovery; I am not certain of the block
             layout but it is similar to raid0 with not as much physical space
             available due to some data redundancies

So, when I was booting off /dev/hda2 on my machine, I simply typed:

mdcreate raid0 /dev/md0 /dev/sdb1 /dev/hda6 /dev/sda1
mdcreate raid0 /dev/md1 /dev/sdb2 /dev/sda2

And now, upon bootup, I simply type:
mdadd -ar

Which activates the md devices.  At this point in time I am able to fsk and
mount the /dev/md* just as if they were hd*'s or sd*'s.  They are a block
device, like any other.

Honestly, you will get the idea when you toy around and start planning to do
your own raid'ing.  Especially when you grab the md utils and compile them,
while they compile you can read the man pages and you should have no problem.

It is actually fairly simple to use.
--
Todd Fries .. [email-masked]