migrating data from linux to opensolaris and zfs with a mirror setup

What I am trying to do
I am trying to backup my data (music, movies, my companies files, perso data) to a stable storage location with some mirroring – I have commodity hardware like most you guys have – I am not willing to loose them because of a faulty hardware.
Right now I have on my main box 2×1.5TB disks for storage and a root disk of 200GB, used for my different operating systems (linux, opensolaris and DragonflyBsd).
I bought the 2nd 1,5TB to backup my main 1.5TB disk (hence this article ;-) ).

Why choosing Hard drive storage
I ruled out dvd/cd backups because:

• too much data to back-up (more than one TB), taking of that many disks brings in other problems (like space …). I am a backup junky, I backup everything and multiple times :-(
• MTBF of burned dvd/cd can be very low (at least the discs I buy …), see this faq to understand how variable the life of a burned dvd is and this thread that confirm my past experience with burned cd/dvd (I lost 5 out of 100 cd after 4 years).
• regularly checking for damaged dvd is time consuming: I am not willing to spend hours/days flipping disks around to see if all my collection is ok. The machine can do that on my behalf with hard disks (zfs scrub command or manual md5sums to compare both disks to the original md5sum stored somewhere – hardcore but still easier than manually feeding my dvd toaster …)

Why zfs
I am a long time user of the xfs filesystem: everytime I setup a linux box, this is the filesystem I use (except for /boot, I use ext2). Of course xfs does not provide mirroring or raid out of the box, but it is a very reliable filesystem.
As for zfs, I was impressed by this article from eric lowe’s blog. After that I was curious, my main issue with storage is how to prevent data rot. There are no perfect solution, but zfs is a nice step in the right direction; I like this quote (see the article) from Kevin McAleer, as it sums up perfectly why I did choose zfs:

Finally, I choose to use the ZFS file system because I know that it checksums every read and write to the filing system, ensuring that my data is as it was when it was written to disk.

The nice plus is that it has some very nice documentation like this one from solaris internals.

why mirroring instead of a raid configuration
That’s the simplest setup, easy to understand, easy to maintain, easy to export since each disk is complete as opposed to raid. I don’t think a mirror is better than a raid setup: it just suits my needs better than a raid would do (at least for now).
An alternate configuration, a raid setup is described here.

Why opensolaris
I use open technologies/standards and open source code in general for as many things as I can. I try to avoid any vendor lock-in and I do not trust any software that does not have its code available and that I can rebuild (yes I probably have paranoia).
Linux meet these requirements as well as opensolaris (even better though), so why solaris in the end ? Mainly because zfs is part of it natively. Linux has a zfs port available, through fuse, but it has performance issues.
FreeBSD was serious candidate, I hesitated a long time between opensolaris and FreeBSD …
I am a unix developer, I use c and c++, opensolaris has the gcc tool chain as FreeBSD has and also the sun studio compilers that I have to use from time to time for my day to day job …
A totaly arbitray choice for some of you, I know, but this is my setup :-).

Seting up the box for opensolaris and zfs
First things first: dowloading opensolaris, I choosed the 2008/11 version for 32/64 bits.
I installed it on my box without any major issue. Instalation instructions can be found at the sun website here.
I said no major issues, I did not say no issues at all ;-)

• I run linux, when I installed opensolaris I found out that it wiped out my mbr, and I was not able to choose to boot linux. No big deal for me: I went to the grub console and booted it from here; I then updated grub and reinstalled it. This can be a little bit too complicated if you are not used to this kind of dance, so do as the doc says:

Note – Before you install the OpenSolaris OS on a system that is running the Linux OS, save a copy of the menu.lst file. The contents of the GRUB menu.lst file dictate what is displayed in the GRUB menu when you boot the system. You will need to update the GRUB menu after the installation. For further information, see menu.lst file specifics at x86: Booting a Solaris System with GRUB.

• disk partitioning did not work under the installer for my disks. I ended up using fdisk by hand: I dedicated a full primary partition to solaris with fdisk, and that was it.

I choosed to use zfs for the whole disk, as such when booting into solaris:

ppi@leo:~$ zpool list
NAME      SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
rpool      54G  10.1G  43.9G    18%  ONLINE  -

will show you the rpool, the root zfs pool which is where / is mounted. So far so good, now we need to setup our storage policy: the zfs mirror.

listing the storage devices
The first step is to create a ‘pool’ i.e. a storage pool. Members of the pool are devices, hard disks. Current devices one the system can be listed using the ‘format’ command:

root@leo:/root# format
Searching for disks...

The device does not support mode page 3 or page 4,
or the reported geometry info is invalid.
WARNING: Disk geometry is based on capacity data.

The current rpm value 0 is invalid, adjusting it to 3600

The device does not support mode page 3 or page 4,
or the reported geometry info is invalid.
WARNING: Disk geometry is based on capacity data.

The current rpm value 0 is invalid, adjusting it to 3600

The device does not support mode page 3 or page 4,
or the reported geometry info is invalid.
WARNING: Disk geometry is based on capacity data.

The current rpm value 0 is invalid, adjusting it to 3600
done

c7t2d0: configured with capacity of 1397.21GB
c7t5d0: configured with capacity of 1397.21GB
c9t0d0: configured with capacity of 465.74GB

AVAILABLE DISK SELECTIONS:
       0. c7t0d0
          /pci@0,0/pci1458,b002@11/disk@0,0
       1. c7t2d0
          /pci@0,0/pci1458,b002@11/disk@2,0
       2. c7t5d0
          /pci@0,0/pci1458,b002@11/disk@5,0
       3. c9t0d0
          /pci@0,0/pci1458,5004@13,2/storage@3/disk@0,0
Specify disk (enter its number):

This will list the devices available on your system. for example, my two 1,5TB disks are c7t2d0 and c7t5d0. How do I know that ? At the prompt I selected c7t2d0 by selecting 1:

selecting c7t2d0
[disk formatted]
No Solaris fdisk partition found.

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> inquiry
Vendor:   ATA
Product:  ST31500341AS
Revision: SD1B

I did the same thing for the second disk:

format> disk 2
selecting c7t5d0
[disk formatted]
No Solaris fdisk partition found.
format> inquiry
Vendor:   ATA
Product:  ST31500341AS
Revision: CC3G

setting up the zpool and creating filesystems
Now we know which disks should be part of the mirror pool: c7t5d0 and c7t2d0.
We can go on and create the pool … well no we cannot. one of the 1,5TB drives is formatted as XFS and contains the data to be mirrored, we cannot setup the pool as a mirror for now. We need to migrate the data to the other disk, reformat this one as zfs, since opensoalris don’t know how to handle xfs and then setup the mirror.
Hopefully setting up the mirror after this initial copy is very easy with zfs. So for now we create a regular zfs pool with one drive only:

zpool create tank c7t5d0p0

This will create the zpool ‘tank’ and add device c7t5d0p0 to it.
We need now to create a zfs filesystem on it:

zfs create tank/media
...
zfs create tank/backup
...

tank/media will store all my multimedia files and tank/backup will store all my other files, like digital personal data, copy of the invoices made to my clients, my emails …
These filesystems are mounted on /, so you can easily access them using /tank/media and /tank/backup.

We are now ready to start the migration of the linux data to opensolaris/zfs.

Migrating the data from linux to solaris
The linux system is installed on the same box, that rules out networking to replicate the data. As such I will have to use another disk, a smaller disk to migrate the data from linux to opensolaris.
A usb disk of 320 GB will do the trick (i have a spare disk and an usb enclosure so …).

Now the real thing: we need to have linux and opensolaris to be able to exchange data and as such they need to mount/unmount the same filesystem… And off course they do not share any reliable filesystem (I mean a reliable implementation), there are patches/software to access ntfs and ext2 filesystem.I tried them and the machine hang the various times I used them.

I checked then the zfs implementation for linux, expecting the same overall quality as the ext2 and ntfs filesystems.
I must say that I was surprised to come upon a very stable zfs port for linux (I downloaded the trunk version, not the 0.4 which has some bugs).

	hg clone http://www.wizy.org/mercurial/zfs-fuse/trunk
	cd trunk/src
	scons
	sudo scons install

Do not forget to add /usr/local/bin to your PATH variable so you can type:

	sudo zfs-fuse

That will initialize the zfs code for linux. You can now use zfs commands under linux :-)

zpool create tempdev /dev/sdd
...
zfs create tempdev/mass
...

And then copy from the 1.5TB to the 320 GB – while you grab a coffe. It take sometime, but I do not care: the machine is working, not me.

It worked quite well, minus the performance – 10 MB/s at best – anyhow it was good enough for my purpose: migrating my data from linux to opensolaris.

Once you finished the copy from linux, we need to copy it to opensolaris. We need to find the usb drive and mount the linux zfs filesystem. To check, under opensolaris, where is the zfs pool you created under linux, run the command ‘zpool import’:

root@leo:~# zpool import
  pool: tempdev
    id: 13948735862038308700
 state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
config:

	tempdev     ONLINE
	  c9t0d0p0  ONLINE

It’s like Magic. ZFS complains about the version of zfs used to create the pool, but we don’t care, it can handle it.
To add the pool to the system, we try to run (see the doc):

zpool import tempdev

If that doesn’t work, try with the -f option.
At this point you are now able to copy your +300GB data to the new 1,5 TB disk of yours.

I did this all dance i.e. copy/erase/import/copy a bunch of time. I was not in a hurry – I wanted a functionnal mirror and had some time available to play with zfs.

add mirroring to the pool
Once all the data has been copied over, it’s time to setup the mirroring i.e. using the second 1,5TB to be a replica. This is the easiest part, you just have to add this new device (you need to remove all partitions from it before) :

zpool attach tank c7t5d0p0 c7t2d0p0

where c7t5d0p0 is the tank disk that already exists and c7t2d0p0 is the new disk to be added to the pool. The pool will automatically be promoted to a mirror, and zfs will start re-silvering the disk i.e. copying data to the replica.

Dead simple. I loved it – the culmination of the whole thing :-)

You can check the state of the pool with ‘zpool status tank’, when it is finished you will have:

root@leo:~# zpool status tank
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  mirror      ONLINE       0     0     0
	    c7t5d0p0  ONLINE       0     0     0
	    c7t2d0p0  ONLINE       0     0     0

errors: No known data errors

Quit a journey: a long but easy ride.
ZFS fits perfectly in my new setup, I can now go to sleep happy since I know my data is in good hands. One can do many. many more things with it, I barely scratched the surface and only cared about my immediate needs: having a set of working and trusted mirrored disks :-).

Share, and enjoy.