<H2>Drobo-Utils: Drobo: API, CLI, & Dashboard for linux</H2>
So you bought a Drobo and you want to use it under linux.
Great. Plug it in and Go! Really, it's just a normal disk in a lot of ways. <P> [ Gallery ] [ Download - Where to get it]
<P>
Contents
- Project -- development home page. source code there as a download too...
- README - Documentation]
- Group - Discussion Forum
- Drobo - Who makes them
- Project - SourceForge Project
- DEVELOPERS Developer docs
- DroboSpace - Vendor forum
- CHANGES.txt - What's in each version
If you have questions, you can try the google groups or just post to drobo-utils-devel@sourceforge.net, but for now, it's just me... my email works too... Peter.A.Silva@gmail.com (help welcome!)
Drobo-utils is a set of linux tools to query and manage Data Robotics Drobo storage systems. If you fire up droboview, it should look pretty familiar to those who have seen the dashboard on other operating systems. Droboview is built on a little programmer interface which can be installed on the system and used by other applications as well.
For experienced Linux hands, there is a command line interface, drobom, which offers the same functionality as droboview. For real hackers, fire up a python interpreter, 'import Drobo', help(Drobo), and you are off to the races... Command-line access is also bundled into a set of improvements to a standards droboshare called 'Droboshare Augmented Root File System' (DARFS)
Drobo utils installation.
Drobo-utils was developed on pre-release version of Kubuntu (Hardy, Intrepid, and now Jaunty) Any similarly recent distro ought to do.
To get drobo-utils running, you need packages something like (these are ubuntu packages, names may vary on other distros):
python -- interpreter for python language parted -- partitioner, usually included with the distro.
To get a complete list, it is best to use a shell window to grep in the Debian package control file (which defines what the dependencies are for the build system):
peter@pepino% grep Depend debian/control
Build-Depends: debhelper (>= 5), python2.5-dev, python-docutils
Depends: ${shlibs:Depends}, ${misc:Depends}, parted
peter@pepino%
On ubuntu, it would typically look like so: Open a shell window. Enter the following package installation commands:
% sudo aptitude install python-qt4 parted % sudo aptitude install debhelper python2.5-dev % sudo aptitude install python-docutils
If you have received a pre-built binary package,then you only need the first line. If you want to build from source, then you need the second line. The third line install what you need to build documentation.
On redhat/fedora distros, it would more likely be 'yum' instead of 'aptitude' and some of the package names will change. A typical difference is that packages for developers have the -devel suffix on Redhat derived distributions, instead of the -dev favoured by debian derived ones.
here is an example from fedora 7 (courtesy of help4death on the google group):
% yum install python % yum install PyQt4 % yum install python-devel
NOTE: if X or QT is missing, it will only disable the GUI. Line mode will work without issues. the package should work fine on headless servers using only the command line.
Point your browser at: http://sourceforge.net/project/showfiles.php?group_id=222830 where current packages are available. after downloading a .deb, it is simply a matter of:
dpkg -i drobo-utils-<version>.deb
done!
See DEVELOPER.txt
Assuming you have all of the above parts, in the directory where you downloaded the source, you should be able to invoke the command line interface as follows:
drobom status
see if something sensible happens... on my system with a drobo the following happens:
% sudo drobom status /dev/sdz /drobo01 100% full ( ['Red alert', 'Bad disk', 'No redundancy'], 0 ) %
Note: drive changed to sdz to avoid copy/paste errors.
very scary, but my drobo is in bad shape right now... you should just get [] as a status, which means there is nothing wrong. To get all kinds of information on your drobo, try 'drobom info.' You can then invoke it with no arguments at all which will cause it to print out a list of the commands available through the command line interface.
Once the command line stuff that is working, and assuming you have python-qt4 installed, try:
% droboview
which should start a GUI for each drobo attached to your machine, that you have permission to access (depends on the setup, usually USB devices on desktops are accessible to users, so you can see them.
See DEVELOPERS.txt
See DEVELOPERS.txt
One can use the Format tab of the GUI to partition the device and create a single file system for a given LUN.
NOTE: mke2fs takes a very long time to run, on the order of ten minutes per Terabyte. the display format button just turns red while the format is in progress,and you have to wait until it finishes. Have not determined a method to monitor progress yet. other file systems are much more quickly created, so less of an issue.
I actually prefer to use the system tools manually, as described below:
Drobos with firmware 1.1.1 or later work well under linux with ext3. You can, of course set up an NTFS or HPS+ or FAT32 if you really want, but it seems actively counter-intuitive on Linux. Have not tested HPS, but ntfs-3g worked fine initially. However, unless you are going to physically move the disk to between systems, the native (ext3) format has many advantages. The ´coffee is hot´ disclaimer is necessary at this point:
WARNING: THE FOLLOWING 4 LINES WILL ERASE ALL DATA ON YOUR DROBO! WARNING: NO, IT WILL NOT ASK ANY QUESTIONS! WARNING: ASK YOURSELF, before you start: ARE YOU SURE? WARNING: AFTER THE SECOND LINE, YOU ARE TOAST. WARNING: BEST TO BACKUP YOUR DATA BEFOREHAND...
If you didn't use the GUI, Here is what you have to type:
# drobom -d /dev/sdz format ext3 PleaseEraseMyData You asked nicely, so I will format ext3 as you requested if you are really sure, go ahead and do: sh /tmp/fmtscript # cat /tmp/fmtscript #!/bin/sh parted /dev/sdz mklabel gpt parted /dev/sdz mkpart ext2 0 100% parted /dev/sdz print; sleep 5 mke2fs -j -i 262144 -L Drobo01 -m 0 -O sparse_super,^resize_inode /dev/sdz1
The above sets up the drobo as one big partition, with a label that says it ought to contain an ext2 file system. If you want an NTFS file system, then write ´ntfs´ in place of ext2. The next step is to add the file system into the partition. while parted's are instantaneous, the mke2fs takes a while, just have a little patience, it´ll be fine.
sh -x /tmp/fmtscript
(If you want an ntfs file system, then mkntfs -f -L Drobo01 /dev/sdz1 ought to work too... )
On my system the process looked like this:
root@alu:~# parted -i /dev/sdz
GNU Parted 1.7.1
Using /dev/sdz
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart ext2 0 100%
(parted) quit
root@alu:~# fdisk /dev/sdz
GNU Fdisk 1.0
Copyright (C) 1998 - 2006 Free Software Foundation, Inc.
This program is free software, covered by the GNU General Public License.
This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.
Using /dev/sdz
Command (m for help): p
Disk /dev/sdz: 2199 GB, 2199020382720 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdz1 1 267350 2147488843 83 Linux
Command (m for help): q
root@alu:~# mke2fs -j -i 262144 -L Drobo01 -m 0 -O sparse_super,^resize_inode /dev/sdz1
mke2fs 1.40.8 (13-Mar-2008)
Filesystem label=Drobo01
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
8388608 inodes, 536870886 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
16384 block groups
32768 blocks per group, 32768 fragments per group
512 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
root@alu:~#
root@alu:~# mount /dev/sdz1 /mnt
To find Drobo's on a system, drobo-utils queries all the attached devices for indications it is made by Data Robotics. These strings change from product to product. If your (new model) of Drobo is not detected, then run the command line interface with the hardware detection debugging output turned out. like so:
# drobom -v 16 status examining: /dev/sda id: (0, 0, 0, 0, 'ATA ') rejected: vendor is ATA (not from DRI) examining: /dev/sdb id: (2, 0, 0, 0, 'ATA ') rejected: vendor is ATA (not from DRI) examining: /dev/sdc id: (8, 0, 0, 0, 'Drobo ') rejected: vendor is Drobo (not from DRI) returning list: [] No Drobos discovered
Here you see that the vendor string is 'Drobo' which was not a known vendor string at the time this example was run. so then try:
# drobom -s Drobo status
as in, take the unknown vendor string and feed it as -s option to tweak detections of drobom.
See DEVELOPERS.txt
LUN is an abbreviation of 'Logical UNit'. The origin of the term is SCSI terminology. When RAID units became too large for support in the past, and were sub-divided to present smaller units the operating system. The default LUNSIZE on Drobos is 2 TiB (adjustable using the tools.) If more disk space (after allowing for parity/redundancy) than LUNSIZE is installed in a unit, Drobo will show a second (or even third) LUN. Each LUN shows up in Linux as a separate disk (examples if the first LUN shows up as /dev/sde, the next will be /dev/sdf, then /dev/sdg.)
If you think you should see multiple LUNS and you don't, you might have a look at some kernel settings: make sure that scsi_mod kernel module is loaded, make sure /sys/module/scsi_mod/parameters/max_luns is > 1.
Droboview will start up one GUI per drobo, regardless of the number of LUNS. If asked to format, all LUNS for the device will be formatted.
- linux tools which aren't 2TB ready... to exceed 2 TB, you need to:
- -- use GPT partitions, which aren´t supported by older fdisk
- versions. Tools based on libparted work fine, mostly.
- -- gparted fails, and seems to have a 1 TB limit on devices.
- (bug #524948 reported to bugzilla.gnome.org) It's just the GUI, as libparted is fine, and other tools based on it still work.
Drobo-utils depends on the linux generic scsi layer. I suspect that there is just a basic ethernet connection now, and you a few additional driver layers set up before it will work. You need to configure the iscsi driver to recognize the device. Lemonizer on the Google Group 2009/05/16 reported good luck with:
I had to manually configure the ip of the dbpro from the Drobo Dashboard on my macbook to do this as I'm not sure how to get the portal ip for iscsiadm. In my case it was 192.168.2.80 port 3260 and I'll use that ip in the example below
If everything went well, your dbpro should show up under /dev. Also check /var/log/messages to confirm that the iscsi device connected successfully.
After that, drobo-utils should be able to detect the Drobo and manage it over ethernet.
(source: http://groups.google.com/group/drobo-talk/browse_frm/thread/453e02e105e9b41?hl=en )
Some people reported data corruption. This link claims to fix one such issue: http://www.drobospace.com/forum/thread/13951/Dropped-iSCSI-connections/?page=2#24792
Upgrading firmware is pretty self-explanatory in the GUI. the first time you press the Update button, it checks to see if a new firmware is available. If it there is newer firmware, it offer to upgrade, with suitable prompts. Similarly, the line mode interface has two commands to deal with firmware, fwcheck will tell you if an upgrade is required. the fwupgrade will do the job. It takes a few minutes, and prints a status you you can see how it is progressing. Have patience:
root@pepino:/home/peter/drobo/drobo-utils/trunk# drobom fwupgrade validateFirmware start... Magic number validated. Good. 484 + 2937552 = 2938036 length validated. Good. CRC from header: 4260378881, calculated using python zlib crc32: 398201869 yeah, the header CRCs do not match. For now they never do ... ignoring it. CRC for body from header: 1852877921, calculated: 1852877921 32 bit Cyclic Redundancy Check correct. Good. validateFirmware successful... writeFirmware: i=484, start=484, last=2938036 fw length= 488 . wrote 32768 bytes... total: 33252 wrote 32768 bytes... total: 66020 . . . wrote 32768 bytes... total: 2720228 wrote 32768 bytes... total: 2752996 wrote 32768 bytes... total: 2785764 wrote 32768 bytes... total: 2818532 wrote 32768 bytes... total: 2851300 wrote 32768 bytes... total: 2884068 wrote 32768 bytes... total: 2916836 wrote 21200 bytes... total: 2938036 writeFirmware Done. i=2938036, len=2938036 root@pepino:/home/peter/drobo/drobo-utils/trunk#
when it's done, you can check if it worked using:
root@pepino# drobom status /dev/sdf - 00% full - (['New firmware installed'], 0)
If the status is like that, then do:
root@pepino:/home/peter/drobo/drobo-utils/trunk# drobom shutdown
lights will flash etc... wait until Drobo goes dark. Wait another five seconds, then un-plug the USB / connector.
Plug it back in, and wait 10 seconds. it should start up with the latest firmware available for your drobo.
The drobom commands, like DRI's dashboard, will normally get the latest and greatest firmware and upgrade. If you have the need, you can load arbitrary firmware from the CLI with fwload command.
Those worried about safety of using this software should know: it was developed with assistance from the vendor (Data Robotics Inc.), and in every case, based on vendor documentation, and with at least encouragement, if not outright support. For each release, a QA.txt file is built, demonstrating the functionality tests run. There are multiple checksum verifications built into the firmware upgrade process, so it is next to impossible to brick a drobo using the tools. Drobo-utils verifies firmware checksums before attempting to upload the image to the device, and the device checks the firmware against the checksums as well. New firmware is loaded into an alternate location from the currently active one, and if activation of the new firmware fails, the drobo will simply boot the old one.
On the other hand, common sense rules do apply. Setting the LUN size, or re-formatting a Drobo will erase all your data whether you do it on Linux or any other operating system. These are power tools, and they can do some damage to your data if used without proper care. For example, the reliability of any storage unit does not reduce the need for backups it only makes doing them easier. A Drobo is an excellent place to put backups, but not a substitute for them. Backups are the only way to address error 18 (the number of inches in front of the keyboard the source of the issue lies.) and no storage unit can protect against fire or flood.
Drobo has been tested with every old firmware version. Any Drobo should be upgradable to modern firmware using the dashboard.
- NOTE: really need at least 1.1.1 to use Linux & ext3.
- just use the tools to upgrade your firmware ASAP.
- 1.01 - very old... bad idea to install this, need to write
- a script to get out, because it isn't in the revision table. not much works except firmware upgrade.
1.0.2 - works ok in CLI And GUI to view, and upgrade firmware.
- 1.0.3 - GUI and CLI work OK, can upgrade firmware.
- Used ntfs3g for a few months under ubuntu 7.10 Linux. Used 2 TB LUN, with 1.5 TB of physical space available.
- 1.1.0 - dashboard works no issues.
- from this point, you don't seem to need to unplug the USB connector to complete the upgrade.
- firmware prior to here deals badly with ext3.
- 1.1.1 - 1.2.4 works without issues.
- ('name' not supported by firmware)
1.3.0 - works without issues.
droboview isn't suited to run continuously for long periods, as it has a memory leak... total foot print starts out at 32M with a 15 MB resident set size, of which 10 MB are shared, so only about 4M of real memory consumed. but the RSS grows at about 2MB/hour.
29m 11m S 1 2.9 9:44.50 droboview
best to restart it daily, or use it when necessary, but not leave it on for days.
After you resize luns, droboview gets confused, you need to exit and restart.
We have a report that dumping diagnostics does not work over firewire. Work-around: connect via USB.
who did what:
Peter Silva: wrote most all of it. Chris Atlee: the proper debian packaging. Brad Guillory: some help with diagnostics and patches. Joe Krahn: lots of inspiration. Andy Grover: some elegance cleanups.
version 9999, somewhen
copyright:
Drobo Utils Copyright (C) 2008,2009 Peter Silva (Peter.A.Silva@gmail.com) Drobo Utils comes with ABSOLUTELY NO WARRANTY; For details type see the file named COPYING in the root of the source directory tree.
This software is copyright under GPL. See near end file for details...
Random useful bits for developers. (aka notes to myself ;-)
Project Source code is managed using a git repository provided by sourceforge.net. Git ( http://git-scm.com/ ) provides a fully distributed development model, so one can exchange patches arbitrarily among developers. On the sf.net repository, the 'master' branch is the final integration target for future releases.
the following checks out the master branch of the source code tree and puts it in the drobo-utils subdirectory of the current tree.
git clone git://drobo-utils.git.sourceforge.net/gitroot/drobo-utils/drobo-utils (read-only)
git clone ssh://username@drobo-utils.git.sourceforge.net/gitroot/drobo-utils/drobo-utils drobo-utils (to be able to patch)
Before doing Any commits, ensure that the author fields are appropriately set. One can obtain patches applied to the branch with git pull, and commit patches for distribution with git push.
To ensure the correct author on patches created, make sure to set your author settings. Something like the following is appropriate if you use a single identity for all your code contributions:
boule% git config --global user.name "firstname lastname" boule% git config --global user.email "developer@sourceforge.net" boule%
Others may wish for the identity to be associated with each project.
Sometimes, when there are issues, the correction gets checked in, but there hasn't been time to do a full release process. If you really need the fix, then your only option is to get it from the developers' git repository. If you are told 'it is in git', that refers to the git source code management system, and the server for that is at sourceforge.net. How to get it:
# git clone ssh://developer@drobo-utils.git.sourceforge.net/gitroot/drobo-utils/drobo-utils mine # cd mine # go into the source directory you downloaded # ./drobom status # try it out... # python setup.py install # install it in system places... # drobom status # try it out... # git pull # get any changes made since the git clone was done. # python setup.py install # install the changes in system places.
Sample checkout of a stable version. To view available branches:
% git branch -r origin/HEAD -> origin/master origin/master origin/noC origin/peter origin/r0.2.1 origin/r0_3_3 origin/r0_4_0 origin/r0_5_0 origin/r0_6_0
Where a version is something like r0_4_0. then you can pick anyone to work with:
% git branch r0_4_0 -r origin/r0_4_0 Branch r0_4_0 set up to track remote branch r0_4_0 from origin. % git checkout r0_4_0 Switched to branch 'r0_4_0' %
When you use git to get a tree, it keeps copies of metadata to be able to track changes. If you want a copy that is contains no git cruft, rm -rf .git in the root of the source tree.
If you are mixing downloaded packages and source installs, check out the next section for gotchas.
The 'setup.py' script, mentioned in the previous section, is a convention from the distutils python packaging system. distutils installation is slightly different from installation from debian packages. There doesn't seem to be a distutils way to remove a package. touch all the files, do an installation, then manually remove the files it installed.
drobo-utils has been picked up for inclusion in debian. The "real" packaging for debian packages is kept in a separate tree, and maintained by debian developers.
The debian/ setup puts stuff in /usr/sbin while setup.py puts things in /usr/bin. Python install does not install man pages either, which the dpkg takes care of. The libs are placed differently too. haven't reviewed for other conflicts, least confusing to use one or the other method on a system.
(if you do distutils install, then remove the debian package via: dpkg --purge drobo-utils)
- 1 - make a branch
- # assuming you have a local repository... git branch <branch> # creates the branch, from the cwd (ought to be master) git checkout <branch> # switches current dir to the branch.
- 2 - Stamp the branch with version
- vi CHANGES.txt # complete change manifest for release vi debian/changelog # copy manifest from txt, add signature. vi setup.py # edit version vi Drobo.py # edit VERSION
3 - Build packages for testing & Install them. (see separate recipe.)
- 4 - Run QA.
- Record results of release tests in the branch (QA.txt) As new tests are created, modify QA.txt on trunk for to keep references for the next release.
QA.txt is a quality assurance log. The version on the trunk of the releases indicates the QA procedure to be applied to the next version during the release process. Since a branch is created for each release, the version of QA.txt acts as a quality log for that release. so one can do an git checkout, or git export to get the quality log for any release (QA.txt introduce in version 0.4.0)
Assumes you have installed the Build dependencies:
# obtain a fresh tree % git clone ssh://peter_silva@drobo-utils.git.sourceforge.net/gitroot/drobo-utils/<version> drobo-utils-<version> % cd drobo-utils-<version> % rm -rf .git # get rid of Git cruft, yielding a raw source tree. % chmod 755 debian/rules # I dunno why the permissions are wrong... # this debian/ config is just for non-distro packages. # builds for debian and Ubuntu. % dpkg-buildpackage -rfakeroot % cd .. # rename it for whatever distro is appropriate... % mv drobo_utils_0.3.3-1_i386 --> droboutils_0.3.3-1_i386_ubunutuIntrepid.deb # rebuild the source tar because it will have the 'debian' link in it. % cd drobo-utils-0.99.9 % rm debian % cd .. % tar -czvf drobo-utils-0.3.3-1.tgz drobo-utils-0.99.9
apply QA tests. as per QA.txt recording results there.
use the restructured text tools (from the python-docutils package.) to build things using:
% make doc
Have a look at Makefile for how that works. update the web site:
% scp README.html <user>,drobo-utils@web.sourceforge.net:htdocs
Tasks which would be cool to get done, in no particular order:
Option setting... especially for DroboPro IP Address & netmask.
CentOS support. Fedora is likely fine, but should build a VM with CentOS on it, and build an rpm. RHEL has an ancient python (2.4?) with no ctypes by default, so could be interesting.
GUI - handle when Drobo disappears...
partition type should pre-select in Format dialog...
memory consumption on the GUI. still chews up 4 MB per hour...
STANDARD menu entries for droboview...
-- there should be some sort of service integrated with dbus-hal -- not sure that a GUI is needed at all. but if it is, it should be a DBUS client. -- how does authentication work in DBUS?
of interest: -- python bindings for parted... http://dcantrel.fedorapeople.org/pyparted/
Revision date: 2009/05/19
copyright:
Drobo Utils Copyright (C) 2008,2009 Peter Silva (Peter.A.Silva@gmail.com) Drobo Utils comes with ABSOLUTELY NO WARRANTY; For details type see the file named COPYING in the root of the source directory tree.
version 9999, somewhen
copyright:
Drobo Utils Copyright (C) 2008 Peter Silva (Peter.A.Silva@gmail.com) Drobo Utils comes with ABSOLUTELY NO WARRANTY; For details type see the file named COPYING in the root of the source directory tree.
last revised: October 28th, 2009
copyright:
Drobo Utils Copyright (C) 2008,2009 Peter Silva (Peter.A.Silva@gmail.com) Drobo Utils comes with ABSOLUTELY NO WARRANTY; For details type see the file named COPYING in the root of the source directory tree.