I kept it simple and easy to modify/extend for my own needs: I open some messages snippets in vim and I want to get a decoded version of the message right away, selecting the line I was on and calling a program to dump the decoded buffer. Very useful when I am dealing with rare tags or when I was not doing some FIX for a while.

Well, this little code does just that. I added also a section where I can add the local dictionary used by my employer (very helpful).

the code is provided with a BSD licence i.e. yes you can put/use it in commercial code.

compiled/edited on a linux box, but it should be pretty easy to port to whatever platform you want run it.

it is simple to compile:

c++ fixdecode.cpp -g -o a.out


it is simple to run:

./a.out < ExampleFixMessages.txt
——————————————————————————-
83 bytes have been retrieved ’8=FIX.4.19=6135=A49=BRKR56=INVMGR98=034=152=20000426-12:05:08108=3010=143′
[   8]               BeginString ‘FIX.4.1′
[   9]                BodyLength ’61′
[  35]                   MsgType ‘A’
[  49]              SenderCompID ‘BRKR’
[  56]              TargetCompID ‘INVMGR’
[  98]             EncryptMethod ’0′
[  34]                 MsgSeqNum ’1′
[  52]               SendingTime ’20000426-12:05:08′
[ 108]                HeartBtInt ’30′
[  10]                  CheckSum ’143′
-> OK 83 B
…

Under vi I just select line and apply a.out to it, and it dumps this helpful message.

Enjoy and share

I intend to fix bugs reported to me on this code but I do not intend to add any new feature for now.
This code will be maintained but won’t get any new feature from me. At some point this may change but for now, this is it.

If somebody sends bug reports or want me to fix a bug quickly, you better send me a very good report and a method to reproduce the bug :-)

sudo yum install glibc-devel.i386

In this post I’ll be talking about procmail, fetchmail and mutt. Three programs to read and backup emails … Wait, three programs just to read emails !? None sens !
Well, actually it makes perfect sens: to read an email on your computer with your favorite email client, you need to follow these steps:

1. grab the email from your email provider (connect and download)
2. filter/process is the email (in which mailbox do we store it – possibly the spam mailbox, or your regular inbox)
3. read it

For each of these steps, the following jargon is used:

• when you download your email, you are using a mail transfer agent i.e. MTA
• when filtering/processing it, you are using a mail delivery agent i.e. MDA
• and finally when reading it, you are using a mail user agent i.e. MUA

Weird ? Not really if you know the UNIX philosophy: each program does one
thing, but it does it well (Yeah, I know: sounds like a real cliche).

When you use thunderbird or outlook, you are using a MTA, MDA and MUA at the
same time. I will not talk about mail submission agent
MSA that is in charge of sending the email you wrote: this is out of scope :-).

Ok, now let’s go back to the purpose of this post: backing up my gmail account.

the big plan

So, given the previous steps described above, I will:

• use fetchmail as my MTA to connect to gmail and retrieve my emails
• use procmail as my MDA to store the content in the various mailboxes
• read it on mutt, yes mutt is my MUA

step 1: retrieving mail with fetchmail

configuring my gmail account

google provides access to my email using POP3 or IMAP.

I will retrieve my email using POP3, google provides interesting features for it. The one I am interested in is located at settings -> forward and pop/imap -> POP download.
Here I select “Enable POP for all mail (even mail that’s already been downloaded)”: it will allow me to download all my emails, including the ones I already did read.
WARNING: be sure to select “When messages are accessed with POP – keep gmail’s copy in inbox” otherwise, gmail will remove your downloaded email
from your account – which is not my intent.

I also select, in the same page, enabling IMAP, so I can read my emails from everywhere using the IMAP protocol which allows me to view my emails from
anywhere while keeping them on the server i.e. I browse them, when I finish reading my mail, it’s still there on the server: be sure to check IMAP wikipedia entry.

I download them only for backup purpose and use POP3 for that.

Do not forget to save the changes you made to your google account …

dealing with gmail server security

Access to my account is encrypted (a good thing) hence I need to setup some security before beeing able to download some emails.
SSL as used by google, requires you to download some credentials from a certificate authority; it is used to authenticate google server i.e. making sure you really are connecting to google and not to some random (and possibly malicious) server.

First, I need to retrieve the certificate of the CA used by google:

# wget -O Equifax_Secure_Certificate_Authority.pem https://www.geotrust.com/resources/root_certificates/certificates/Equifax_Secure_Certificate_Authority.cer

and convert it to something that can actually be used by our beloved ssl tools:

# openssl x509 -in Equifax_Secure_Certificate_Authority.pem -fingerprint -subject -issuer -serial -hash -noout

I now store it where they can latter be referenced and checked. Since I am always setting up workstations when moving from one workplace to another, I store that kind of information in the UNIX account that has been given to me:

# mkdir ~/.certificates
# mv Equifax_Secure_Certificate_Authority.pem ~/.certificates

Now I make them available to the various programs I use:

# c_rehash ~/.certificates

On opensolaris c_rehash is not available (I don’t know why). It’s a perl script and I found a version using google code here.

cut and paste this code into a file named c_rehash and execute it:

# cd ~
# perl c_rehash .certificates
Doing .certificates/
Equifax_Secure_Certificate_Authority.pem => 594f1775.0

This is it for the security part, SSL credentials are now ready to be used by fetchmail.

configuring fetchmail for POP access

my gmail account is something like:

• user ‘vodoom@gmail.com’
• pop3 server is ‘pop.gmail.com’

my local settings are:

• user on my machine is ‘ppereira’
• my google certificate is located at ‘~/.certificates’ (see previous step)
• mda to be used is ‘procmail’ (more on that later)

The corresponding fetchmail configuration (to be stored in ~/.fetchmailrc):

set daemon 600
poll pop.gmail.com with proto POP3 and options no dns
user 'vodoom@gmail.com' there is 'ppi' here
options keep ssl sslcertck sslcertpath '~/.certificates'
mda '/usr/bin/procmail -d %T'

For more details about these options, see fetchmail’s manual.

configuring procmail for delivery

spam filtering ?

Note that there is no spam filtering configuration here: since this setup is only for my gmail account, I rely on gmail spam filter to take care of those.
If you need a spam filtering setup, you may want to have a look at http://wiki.apache.org/spamassassin/UsedViaProcmail.

managing my mailing lists

I use the TO_ recipe rule of procmail: I do not use the list-id header, it is more covenient for me as explained in http://www.ii.com/internet/robots/procmail/qs/#alt2TO.
Procmail recipes are matching rules associated with a mailbox (yes, I am over simplifying).
For example, here is a simple recipe:

:0:
* ^TO_zfs-discuss@opensolaris.org
osol-zfs

:0: is the delivery options, in my case I want locking when accessing to the inbox (that’s the second ‘:’)

* ^TO_zfs-discuss@opensolaris.org
means that we want to match messages that were sent to zfs-discuss@opensolaris.org
One can use regular expressions and special matching rules e.g. ^From to create a matching rule.

osol-zfs tells procmail to store this email in the mailbox ‘osol-zfs’

here are my recipes to match the mailing lists I am subscribed to:

:0:
* ^TO_zfs-discuss@opensolaris.org
osol-zfs

:0:
* ^TO_perf-discuss@opensolaris.org
osol-perf

:0:
* ^TO_opensolaris-code@opensolaris.org
osol-code

:0:
* ^TO_opensolaris-announce@opensolaris.org
osol-announce

:0:
* ^TO_indiana-discuss@opensolaris.org
osol-indiana

:0:
* ^TO_ogb-discuss@opensolaris.org
osol-board

:0:
* ^TO_eeepc-discuss@opensolaris.org
osol-eeepc

:0:
* ^TO_crypto-discuss@opensolaris.org
osol-crypto

:0:
* ^TO_xvid-devel@xvid.org
xvid-devel

:0:
* ^TO_users@crater.dragonflybsd.org
dragonfly-bsd

All those recipes are stored in their own file: rc.mailing-lists.
This file will be included from a ‘master’ file, the main .procmailrc file.
The .procmailrc file looks like:

# Directory for storing procmail configuration and log files
# You can name this variable anything you like, for example
# PROCMAILDIR, or don't set it (but then don't refer to it!)
PROCMAIL_DIR=$HOME/procmail
SHELL=/bin/sh

# LOGFILE should be specified ASAP so everything below it is logged
LOGFILE=$PROCMAIL_DIR/procmail.log

# where are the various mailboxes stored ?
# note that this is *not* the location of the system email
# this is where I want my mailboxes to be stored by procmail i.e. the
# destination after filtering
MAILDIR=$HOME/mail

# recipes/filters are included from here
INCLUDERC=$PROCMAIL_DIR/rc.mailing-lists
INCLUDERC=$PROCMAIL_DIR/rc.banking

Now that procmail is setup, time to retrieve some emails:

# fetchmail -vk
File /export/home/ppi/.fetchmailrc must have no more than -rwx--x--- (0710) permissions.

oops I forgot to setup the proper permissions for the .fetchmailrc file,
let’s fix that:

# chmod 710 .fetchmailrc

Running it again, and failing again for a very different reason, from my logfile in ~/fetchmail/fetchmail.log:

fetchmail: starting fetchmail 6.3.8 daemon
fetchmail: 6.3.8 querying pop.gmail.com (protocol POP3) at Thu Jan 28 00:06:43 2010: poll started
fetchmail: getaddrinfo("pop.gmail.com","pop3s") error: service name not available for the specified socket type
fetchmail: Try adding the --service option (see also FAQ item R12).
fetchmail: POP3 connection to pop.gmail.com failed: Bad file number
fetchmail: 6.3.8 querying pop.gmail.com (protocol POP3) at Thu Jan 28 00:06:43 2010: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: sleeping at Thu Jan 28 00:06:43 2010 for 600 seconds

hum, fetchmail is looking for the port number corresponding to the ‘pop3s’ service. AFAIK, on my opensolaris box I don’t have such service configured:

# grep -i pop3s /etc/services
#

To connect to gmail I have to manually setup use the port number
corresponding to ‘pop3s’, as suggested by fetchmail in the log:

'fetchmail: Try adding the --service option (see also FAQ item R12).'

so here I go again:

# fetchmail -vk --service 995

now from my fetchmail log I can see:

fetchmail: 6.3.8 querying pop.gmail.com (protocol POP3) at Fri Jan 29 00:15:07 2010: poll started
fetchmail: Trying to connect to 74.125.93.111/995...connected.
fetchmail: Issuer Organization: Google Inc
fetchmail: Issuer CommonName: Google Internet Authority
fetchmail: Server CommonName: pop.gmail.com
fetchmail: pop.gmail.com key fingerprint: 92:73:17:4C:34:4B:68:F7:B2:17:71:42:0D:7F:9F:33
fetchmail: POP3< +OK Gpop ready for requests from 70.31.248.56 8pf4635265qwj.1
fetchmail: POP3> CAPA
fetchmail: POP3< +OK Capability list follows
fetchmail: POP3< USER
fetchmail: POP3< RESP-CODES
fetchmail: POP3< EXPIRE 0
fetchmail: POP3< LOGIN-DELAY 300
fetchmail: POP3< X-GOOGLE-VERHOEVEN
fetchmail: POP3< UIDL
fetchmail: POP3< .
fetchmail: POP3> USER vodoom@gmail.com
fetchmail: POP3< +OK send PASS
fetchmail: POP3> PASS *
fetchmail: POP3< +OK Welcome.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 354 21544383
fetchmail: POP3> LAST
fetchmail: POP3< -ERR Not supported
fetchmail: Not supported
fetchmail: POP3> UIDL
fetchmail: POP3< +OK
fetchmail: POP3< 1 GmailIdfbe952ed20bfe97
fetchmail: POP3< 2 GmailIdfc26b09f74e3775
fetchmail: POP3< 3 GmailIdfc6f980243d2a13
fetchmail: POP3< 4 GmailId1043d10888f8ed70
...
fetchmail: POP3< 354 GmailId10e7b3433c301f94
fetchmail: POP3< .
fetchmail: 354 messages for vodoom@gmail.com at pop.gmail.com (21544383 octets).
fetchmail: POP3> LIST 1
fetchmail: POP3< +OK 1 1694617
fetchmail: POP3> RETR 1
fetchmail: POP3< +OK message follows
fetchmail: reading message vodoom@gmail.com@pop.gmail.com:1 of 354 (1694617 octets)
...

Yes ! This is working :-)

So now my gmail account is backed up. From time to time I run my fetchmail script, and it downloads the new emails that are not yet backed up.
If like me you have thousands of emails waiting to be download, be patient, as emails are not downloaded in a single connection: each connection will download hundreds of emails, fetchmail will have to reconnect to download the rest. This is fine, that’s why I setup the daemon mode in fetchmail :-).

Ok now I have to access my emails with IMAP, but thats another post :-)

enjoy and share.

ok some context first.

I am using vmware for 3 years at work (in my two previous jobs), and I must say it worked flawlessly all this time.

But lately it started to “freeze” when compiling some hot stuff under the vm. It also became more and more a resource hog.

So I started to look for some alternatives, and of course virtualbox from sun showed up. I gave it a try and was hooked up: it is fast, and handled very well the various OS and jobs I throwed at it.

So I thought it is ready to replace vmware at work (and it is, with some more work).

I use a windows box (for corporate reasons etc.), and edit/compile code in virtualbox vm running opensolaris (for efficiency reasons :-)). The changes I make are later on compiled on build machine.

I use a X display fo 1680×1050 32 bpp – and yes took me some time to get this setup on vmware.

I wanted to have the same setup on virtualbox and … At first I failed: virtualbox gfx card did not have the VESA mode I wanted and defaulted to 1280×1024, with a lot of unused space left on my monitor :-(

I finally found a solution, and I am posting it here so I will not forget it later on: I used VBoxManage – read the section “custom VESA modes” in the virtualbox user manual.

On your windows box, lauch the command line prompt and go to the folder where virtualbox has been installed, in my case:

C:\Program Files\Sun\VirtualBox

then add the missing VESA mode fro your vm, in my case the vm is called ”adam”:

VBoxManage setextradata "adam" "CustomVideoMode1" "1680x1050x32"

Restart your vm and *magic* my login screen is 1680×1050.

Do not forget to edit your xorg.conf file to actually setup this mode. Here is mine in case you need an example (this is a valid file for samsung monitor model 2243bw, for other monitors you MUST edit the HorizSync/VertRefresh values or comment them out) :

Section "Monitor"
Identifier "samsung-2243bw"
HorizSync 30-81
VertRefresh 50-85
Modeline "1680x1050" 147.14 1680 1784 1968 2256 1050 1051 1054 1087 -HSync +Vsync
EndSection
Section "Screen"
Identifier "Screen 1"
Monitor "samsung-2243bw"
DefaultDepth 24
Subsection "Display"
Depth 8
Modes "1680x1050" "1280x1024" "1024x768" "800x600" "640x480"
ViewPort 0 0
EndSubsection
Subsection "Display"
Depth 16
Modes "1680x1050" "1280x1024" "1024x768" "800x600" "640x480"
ViewPort 0 0
EndSubsection
Subsection "Display"
Depth 24
Modes "1680x1050" "1280x1024" "1024x768" "800x600" "640x480"
ViewPort 0 0
EndSubsection
EndSection
Section "ServerLayout"
Identifier "Simple Layout"
Screen "Screen 1"
EndSection
Section "InputDevice"
Identifier "VBoxMouse"
Driver "vboxmouse"
Option "CorePointer"
EndSection

Enjoy and share.

why compiling it !?

I am actually interested to build mplayer on opensolaris, with xvid and x264 support.

First things first: download mplayer sources from their website, they have an svn repository one can use to download the latest version:

$ svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer

If you do not have subversion, try to install it using the following command:

$ pfexec pkg install SUNWsvn

you should have now a directory called mplayer, it has the sources we want in there. Mplayer uses the GNU autotools, let’s configure the sources:

$ ./configure
Detected operating system: SunOS
Detected host architecture: i386
Checking for gcc version ... 3.4.3
Checking for host cc ... gcc
Checking for cross compilation ... no
Checking for CPU vendor ... AuthenticAMD (15:107:2)
Checking for CPU type ...  AMD Athlon(tm) 64 X2 Dual Core Processor 4200+
Checking for kernel support of mmx ... yes
Checking for kernel support of mmxext ... yes
Checking for kernel support of 3dnow ... yes
Checking for kernel support of 3dnowext ... yes
Checking for kernel support of sse ... yes
Checking for kernel support of sse2 ... yes
Checking for kernel support of cmov ... yes
Checking for mtrr support ... yes
Checking for GCC & CPU optimization abilities ... k8
Checking for byte order ... little-endian
...
...
Config files successfully generated by ./configure  !

Install prefix: /usr/local
Data directory: /usr/local/share/mplayer
Config direct.: /usr/local/etc/mplayer

Byte order: little-endian
Optimizing for: k8

Languages:
Messages/GUI: en
Manual pages: en

Enabled optional drivers:
Input: dvdnav(internal) ftp tv libdvdcss(internal) dvdread(internal) vcd smb network
Codecs: libavcodec(internal) qtx xanim win32 faad2(internal) libmpeg2(internal) liba52(internal) mp3lib(internal) speex tremor(internal)
Audio output: sdl mpegpes(file)
Video output: sdl pnm jpeg png mpegpes(file) cvidix yuv4mpeg md5sum tga
  Disabled optional drivers:
    Input: vstream pvr radio tv-teletext tv-v4l2 tv-v4l1 tv-dshow live555 nemesi cddb cdda dvb
    Codecs: libschroedinger libdirac x264 xvid libdv libamr_wb libamr_nb real faac musepack libdca libtheora toolame twolame libmad liblzo gif
    Audio output: sun alsa openal jack pulse nas esd arts oss v4l2 ivtv dxr2
    Video output: zr zr2 v4l2 ivtv dxr3 dxr2 vesa gif89a fbdev svga caca aa ggi xmga mga xvidix winvidix opengl 3dfx dga vdpau xvmc xv x11 dfbmga directfb bl xvr100 tdfx_vid wii s3fb tdfxfb

Hum, not so good. The summary says we are missing a bunch of codecs I am interested in: x264 xvid libtheora.
Some output drivers are also missing: xv x11 opengl.

I am so used to get mplayer from linux that I almost forgot how it was to build everything from source …
But there also some good things about that: I will bee able to have some tweaking for my cpu (at least that’s my hope).

compiling xvid

So now, we need to get the missing codecs/drivers and compile them so mplayer can use them. First let’s get xvid:

$ wget http://downloads.xvid.org/downloads/xvidcore-1.2.1.tar.gz

Using the SUN compiler to compile xvid is not supported, from the FAQ:

Q: Are there instructions available how to compile Xvid with my compiler?
A: Have a look at the install readme file that comes along with Xvid. It describes how to use the Xvid build scripts to compile Xvid with gcc under Linux or MinGW/CygWin. Xvid can also be compiled with Microsoft Visual C++ compiler using the provided project files included in the build directory. For other compilers you’re pretty much on your own. It should be not very difficult however to make the code compile with a different compiler.

Ok so let’s try it, otherwise why having the source code :-) ?

compiler to be used

The compiler that I want to use is the sun compiler, version on my opensolaris box:

    $ cc -V
    cc: Sun Ceres C 5.10 SunOS_i386 2009/03/06
    usage: cc [ options] files.  Use 'cc -flags' for details

assembler to be used for xvid

If available, I want to use assembly code provided by xvid. xvid uses nasm to assemble the code. Time to get it:

$ wget http://www.nasm.us/pub/nasm/releasebuilds/2.06rc11/nasm-2.06rc11.tar.gz

then we compile and install it:

    $ gtar xzf nasm-2.06rc11.tar.gz
    $ cd nasm-2.06rc11
    $ CC=/usr/bin/suncc CFLAGS='-xtarget=native' ./configure
    checking for prefix by checking for nasm... /usr/local/bin/nasm
    checking for gcc... /usr/bin/suncc
    checking for C compiler default output file name... a.out
    checking whether the C compiler works... yes
    ...

Note that the compiler used is suncc, the c compiler from sun and not gcc.
When compiling some warnings are given:

cc: Warning: illegal option -d=c99

For now I will ignore them – the option was dropped anyway. It’s now time to install nasm:

    $ pfexec make install
    mkdir -p /usr/local/bin
    /usr/bin/ginstall -c nasm /usr/local/bin/nasm
    /usr/bin/ginstall -c ndisasm /usr/local/bin/ndisasm
    mkdir -p /usr/local/share/man/man1
    /usr/bin/ginstall -c -m 644 ./nasm.1 /usr/local/share/man/man1/nasm.1
    /usr/bin/ginstall -c -m 644 ./ndisasm.1 /usr/local/share/man/man1/ndisasm.1

cool, now let’s go back to xvid.

tweaking the xvid source

Since I want to be able to work on this source tree from time to time, I’ll track this directory content using mercurial.

    $ hg init
    $ hg add

let’s give it a first try, let’s go to the build directory:

    $ cd build/generic
    $ CC=/usr/bin/suncc CFLAGS='-xtarget=native' ./configure
    ....
    $ make
    make: Fatal error in reader: Makefile, line 82: Badly formed macro assignment

So … chances are that we have a gnu makefile here:

gmake
  D: =build
  C: ./decoder.c
"../../src/portab.h", line 439: warning: no explicit type given
"../../src/portab.h", line 439: syntax error before or at: void
"../../src/portab.h", line 447: warning: no explicit type given
"../../src/portab.h", line 447: syntax error before or at: int64_t
"../../src/global.h", line 63: warning: no explicit type given
"../../src/global.h", line 63: syntax error before or at: int
"../../src/global.h", line 70: warning: no explicit type given
"../../src/global.h", line 70: syntax error before or at: int
"../../src/global.h", line 234: warning: no explicit type given
"../../src/global.h", line 234: syntax error before or at: uint32_t
"../../src/image/image.h", line 41: cannot have void object: __inline
"../../src/image/image.h", line 42: syntax error before or at: image_null

It was a gnu makefile, hence gmake took care of it. The source files use some gcc extension, like __inline. There is a c99 keyword for it, that does pretty much the same thing: inline.
We reconfigure the package:

$ CC=/usr/bin/suncc CFLAGS='-D__inline=inline -xtarget=native' ./configure

hum … Too many warnings:

"../../src/encoder.c", line 1085: warning: integer overflow detected: op "<<"
"../../src/encoder.c", line 1112: warning: integer overflow detected: op "<<"
"../../src/encoder.c", line 1150: warning: integer overflow detected: op "<<"
...
"../../src/xvid.c", line 105: warning: assignment type mismatch:
	pointer to void "=" pointer to function(int) returning void
...
"../../src/bitstream/mbcoding.c", line 1848: warning: initializer will be sign-extended: -1
"../../src/bitstream/mbcoding.c", line 1859: warning: initializer will be sign-extended: -1
"../../src/bitstream/mbcoding.c", line 1895: warning: initializer will be sign-extended: -1
...

overflow is scary … I’ll need to check the code and see whats going on.

The linker problem can be solved by installing the following patch found here.

Edit from May 13th 2009: I finally added a patch for SUN compiler to get xvid to compile without too many warnings, you can find it here

Why choosing Hard drive storage
I ruled out dvd/cd backups because:

• too much data to back-up (more than one TB), taking of that many disks brings in other problems (like space …). I am a backup junky, I backup everything and multiple times :-(
• MTBF of burned dvd/cd can be very low (at least the discs I buy …), see this faq to understand how variable the life of a burned dvd is and this thread that confirm my past experience with burned cd/dvd (I lost 5 out of 100 cd after 4 years).
• regularly checking for damaged dvd is time consuming: I am not willing to spend hours/days flipping disks around to see if all my collection is ok. The machine can do that on my behalf with hard disks (zfs scrub command or manual md5sums to compare both disks to the original md5sum stored somewhere – hardcore but still easier than manually feeding my dvd toaster …)

Why zfs
I am a long time user of the xfs filesystem: everytime I setup a linux box, this is the filesystem I use (except for /boot, I use ext2). Of course xfs does not provide mirroring or raid out of the box, but it is a very reliable filesystem.
As for zfs, I was impressed by this article from eric lowe’s blog. After that I was curious, my main issue with storage is how to prevent data rot. There are no perfect solution, but zfs is a nice step in the right direction; I like this quote (see the article) from Kevin McAleer, as it sums up perfectly why I did choose zfs:

Finally, I choose to use the ZFS file system because I know that it checksums every read and write to the filing system, ensuring that my data is as it was when it was written to disk.

The nice plus is that it has some very nice documentation like this one from solaris internals.

why mirroring instead of a raid configuration
That’s the simplest setup, easy to understand, easy to maintain, easy to export since each disk is complete as opposed to raid. I don’t think a mirror is better than a raid setup: it just suits my needs better than a raid would do (at least for now).
An alternate configuration, a raid setup is described here.

Why opensolaris
I use open technologies/standards and open source code in general for as many things as I can. I try to avoid any vendor lock-in and I do not trust any software that does not have its code available and that I can rebuild (yes I probably have paranoia).
Linux meet these requirements as well as opensolaris (even better though), so why solaris in the end ? Mainly because zfs is part of it natively. Linux has a zfs port available, through fuse, but it has performance issues.
FreeBSD was serious candidate, I hesitated a long time between opensolaris and FreeBSD …
I am a unix developer, I use c and c++, opensolaris has the gcc tool chain as FreeBSD has and also the sun studio compilers that I have to use from time to time for my day to day job …
A totaly arbitray choice for some of you, I know, but this is my setup :-).

Seting up the box for opensolaris and zfs
First things first: dowloading opensolaris, I choosed the 2008/11 version for 32/64 bits.
I installed it on my box without any major issue. Instalation instructions can be found at the sun website here.
I said no major issues, I did not say no issues at all ;-)

• I run linux, when I installed opensolaris I found out that it wiped out my mbr, and I was not able to choose to boot linux. No big deal for me: I went to the grub console and booted it from here; I then updated grub and reinstalled it. This can be a little bit too complicated if you are not used to this kind of dance, so do as the doc says:

Note – Before you install the OpenSolaris OS on a system that is running the Linux OS, save a copy of the menu.lst file. The contents of the GRUB menu.lst file dictate what is displayed in the GRUB menu when you boot the system. You will need to update the GRUB menu after the installation. For further information, see menu.lst file specifics at x86: Booting a Solaris System with GRUB.

• disk partitioning did not work under the installer for my disks. I ended up using fdisk by hand: I dedicated a full primary partition to solaris with fdisk, and that was it.

I choosed to use zfs for the whole disk, as such when booting into solaris:

ppi@leo:~$ zpool list
NAME      SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
rpool      54G  10.1G  43.9G    18%  ONLINE  -

will show you the rpool, the root zfs pool which is where / is mounted. So far so good, now we need to setup our storage policy: the zfs mirror.

listing the storage devices
The first step is to create a ‘pool’ i.e. a storage pool. Members of the pool are devices, hard disks. Current devices one the system can be listed using the ‘format’ command:

root@leo:/root# format
Searching for disks...

The device does not support mode page 3 or page 4,
or the reported geometry info is invalid.
WARNING: Disk geometry is based on capacity data.

The current rpm value 0 is invalid, adjusting it to 3600

The device does not support mode page 3 or page 4,
or the reported geometry info is invalid.
WARNING: Disk geometry is based on capacity data.

The current rpm value 0 is invalid, adjusting it to 3600

The device does not support mode page 3 or page 4,
or the reported geometry info is invalid.
WARNING: Disk geometry is based on capacity data.

The current rpm value 0 is invalid, adjusting it to 3600
done

c7t2d0: configured with capacity of 1397.21GB
c7t5d0: configured with capacity of 1397.21GB
c9t0d0: configured with capacity of 465.74GB

AVAILABLE DISK SELECTIONS:
       0. c7t0d0
          /pci@0,0/pci1458,b002@11/disk@0,0
       1. c7t2d0
          /pci@0,0/pci1458,b002@11/disk@2,0
       2. c7t5d0
          /pci@0,0/pci1458,b002@11/disk@5,0
       3. c9t0d0
          /pci@0,0/pci1458,5004@13,2/storage@3/disk@0,0
Specify disk (enter its number):

This will list the devices available on your system. for example, my two 1,5TB disks are c7t2d0 and c7t5d0. How do I know that ? At the prompt I selected c7t2d0 by selecting 1:

selecting c7t2d0
[disk formatted]
No Solaris fdisk partition found.

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> inquiry
Vendor:   ATA
Product:  ST31500341AS
Revision: SD1B

I did the same thing for the second disk:

format> disk 2
selecting c7t5d0
[disk formatted]
No Solaris fdisk partition found.
format> inquiry
Vendor:   ATA
Product:  ST31500341AS
Revision: CC3G

setting up the zpool and creating filesystems
Now we know which disks should be part of the mirror pool: c7t5d0 and c7t2d0.
We can go on and create the pool … well no we cannot. one of the 1,5TB drives is formatted as XFS and contains the data to be mirrored, we cannot setup the pool as a mirror for now. We need to migrate the data to the other disk, reformat this one as zfs, since opensoalris don’t know how to handle xfs and then setup the mirror.
Hopefully setting up the mirror after this initial copy is very easy with zfs. So for now we create a regular zfs pool with one drive only:

zpool create tank c7t5d0p0

This will create the zpool ‘tank’ and add device c7t5d0p0 to it.
We need now to create a zfs filesystem on it:

zfs create tank/media
...
zfs create tank/backup
...

tank/media will store all my multimedia files and tank/backup will store all my other files, like digital personal data, copy of the invoices made to my clients, my emails …
These filesystems are mounted on /, so you can easily access them using /tank/media and /tank/backup.

We are now ready to start the migration of the linux data to opensolaris/zfs.

Migrating the data from linux to solaris
The linux system is installed on the same box, that rules out networking to replicate the data. As such I will have to use another disk, a smaller disk to migrate the data from linux to opensolaris.
A usb disk of 320 GB will do the trick (i have a spare disk and an usb enclosure so …).

Now the real thing: we need to have linux and opensolaris to be able to exchange data and as such they need to mount/unmount the same filesystem… And off course they do not share any reliable filesystem (I mean a reliable implementation), there are patches/software to access ntfs and ext2 filesystem.I tried them and the machine hang the various times I used them.

I checked then the zfs implementation for linux, expecting the same overall quality as the ext2 and ntfs filesystems.
I must say that I was surprised to come upon a very stable zfs port for linux (I downloaded the trunk version, not the 0.4 which has some bugs).

	hg clone http://www.wizy.org/mercurial/zfs-fuse/trunk
	cd trunk/src
	scons
	sudo scons install

Do not forget to add /usr/local/bin to your PATH variable so you can type:

	sudo zfs-fuse

That will initialize the zfs code for linux. You can now use zfs commands under linux :-)

zpool create tempdev /dev/sdd
...
zfs create tempdev/mass
...

And then copy from the 1.5TB to the 320 GB – while you grab a coffe. It take sometime, but I do not care: the machine is working, not me.

It worked quite well, minus the performance – 10 MB/s at best – anyhow it was good enough for my purpose: migrating my data from linux to opensolaris.

Once you finished the copy from linux, we need to copy it to opensolaris. We need to find the usb drive and mount the linux zfs filesystem. To check, under opensolaris, where is the zfs pool you created under linux, run the command ‘zpool import’:

root@leo:~# zpool import
  pool: tempdev
    id: 13948735862038308700
 state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
config:

	tempdev     ONLINE
	  c9t0d0p0  ONLINE

It’s like Magic. ZFS complains about the version of zfs used to create the pool, but we don’t care, it can handle it.
To add the pool to the system, we try to run (see the doc):

zpool import tempdev

If that doesn’t work, try with the -f option.
At this point you are now able to copy your +300GB data to the new 1,5 TB disk of yours.

I did this all dance i.e. copy/erase/import/copy a bunch of time. I was not in a hurry – I wanted a functionnal mirror and had some time available to play with zfs.

add mirroring to the pool
Once all the data has been copied over, it’s time to setup the mirroring i.e. using the second 1,5TB to be a replica. This is the easiest part, you just have to add this new device (you need to remove all partitions from it before) :

zpool attach tank c7t5d0p0 c7t2d0p0

where c7t5d0p0 is the tank disk that already exists and c7t2d0p0 is the new disk to be added to the pool. The pool will automatically be promoted to a mirror, and zfs will start re-silvering the disk i.e. copying data to the replica.

Dead simple. I loved it – the culmination of the whole thing :-)

You can check the state of the pool with ‘zpool status tank’, when it is finished you will have:

root@leo:~# zpool status tank
  pool: tank
 state: ONLINE
 scrub: none requested
config:

	NAME          STATE     READ WRITE CKSUM
	tank          ONLINE       0     0     0
	  mirror      ONLINE       0     0     0
	    c7t5d0p0  ONLINE       0     0     0
	    c7t2d0p0  ONLINE       0     0     0

errors: No known data errors

Quit a journey: a long but easy ride.
ZFS fits perfectly in my new setup, I can now go to sleep happy since I know my data is in good hands. One can do many. many more things with it, I barely scratched the surface and only cared about my immediate needs: having a set of working and trusted mirrored disks :-).

Share, and enjoy.

I wrote this after someone asked me why cont int** cannot be assigned an int**, which seemed a valid conversion. I pointed him to the c++ faq like, section 18 item 17, but It was pretty terse:

The reason the conversion from Foo** → const Foo** is dangerous is that it would let you silently and accidentally modify a const Foo object without a cast:

class Foo {
public:
void modify(); // make some modify to the this object
};

int main()
{
const Foo x;
Foo* p;
const Foo** q = &p; // q now points to p; this is (fortunately!) an error
*q = &x; // p now points to x
p->modify(); // Ouch: modifies a const Foo!!
...
}


It was hard for him to see why such thing was possible, after all, int* to a const int* is a valid assignment.

If you are comfortable with the const keyword, go directly to the “messing with it” section :-)

OK for now let’s go back to the const keyword itself: it allows a c++ object to be considered as read-only (well, almost …).
The various code snippets were compiled/tested with:

linuxbox:~$ c++ --version
c++ (Ubuntu 4.3.2-1ubuntu12) 4.3.2
...


syntax first

Most of the confusion when using this keyword comes from how c++ declares variable, the syntax used: machine friendly but definitely not human friendly.

Easy things first:

const int one = 1;
const char letter_a = 'a';


These declaration are straight forward, let’s raise the bar:

const int *p = &one;
const int two = 2;
p = &two;

so far so good: p is pointer to a const int. Both addresses of one and two can be assigned to p. Of course the following is not allowed, since p points to a read-only int:

p = &one;
*p = 2; /* error - pointed object cannot be changed */

now let’s have:

const int * const p = &one;

now p is a pointer to a const int plus p is read-only too:

p = &two; /* error: p cannot be changed */

g++ diagnostic is:

linuxbox:~$ c++ const.cpp
const.cpp: In function ‘int main()’:
const.cpp:11: error: assignment of read-only location ‘* p’

Note how the second const is declared after the star to qualify p as being const it self. This is the trick when reading const declaration (oh and yes, const int one = 1; and int const one = 1; are the same declarations), the star is used as a separator to whom the const is applied to.

We can promote a regular object to a const one:

int a = 1;
int* p = &a;
const int* q = p; // a cannot be modified through q


messing with it

Now it’s time to get ugly, the following promotion won’t work:

const int a = 1;
int *p = 0;
const int ** err = &p;

g++ diagnostic is:

linuxbox:~$ c++ const.cpp
const.cpp: In function ‘int main()’:
const.cpp:20: error: invalid conversion from ‘int**’ to ‘const int**’

and we are like “ugh ?”. And yes, it is very fortunate that the compiler can catch that…
Why ? Otherwise we could change const objects and have undefined behavior without realizing it. Let’s say the previous example is legal:

const int a = 1;
int *p = 0;
const int ** err = &p;
/* chaos from here */
*err = &a;
*p = 2;


The type of *err is const int*, hence this assignment is legal:
*err = &a;

and we know that err = &p;, and that as such *err == p …
So we have *err == p and *err = &a, two different types…

Trying to change the value of a through err won’t work, err has the right type:
**err = 2 /* failure: err is const int**, it cannot change a */
BUT we can do it through p now, p has the right type and the promotion allowed us to make it point to the const int a:
*p = 2; /* legal: p has the right type, we lost the trace of constness of a */
We just allowed p to point to const int a and short circuit the type system.

That’s why this type of conversion is not allowed: we don’t want to be able to write such thing without the c++ type system catching it. Now go grab a coffee, you deserve it ;-)

Enjoy and share ;-)

updated: Feb 18th, 2009 – added picnick firefox extension

updated: Dec 1st, 2009 – added firefox tweak for session store

Each time I start a new contract or job, I go through the very same ritual: I spend some time tweaking my workstation.

This is a really boring process, and it takes a lot of energy every-time I do it. This is because I spend too much time doing it by heart, instead of having my tweaks written somewhere I could re-use them (I now have a plain old tar file that I untar on my new workstation every time I get a new contract).

This post should fix this.

Here are my configuration files, firefox tweaks and other tips when I setup my work environment.

shell environment

I don’t really care which shell is available, but I try to make sure that screen is there. If not, I try to install it ASAP. I have a really simple .screenrc file:

        # terminfo and termcap for nice 256 color terminal
        # tell screen how to set colors. AB = background, AF=foreground
        termcapinfo xterm 'Co#256:AB=\E[48;5;%dm:AF=\E[38;5;%dm'

        # erase background with current bg color
        defbce "on"

        # I need ths status line of screen - much easier to navigate and use
        hardstatus on
        hardstatus alwayslastline
        hardstatus string "%{.bW}%-w%{.rW}%n %t%{-}%+w %=%{..G} %H
        %{..Y} %d/%m/%Y %C%a "

I have a very simple shell rc file, currently a .bashrc:

        # yes I use gpg, and by default this is the key that should be used
        export GPGKEY=E495D229

        # I am a bit fanatic on what my prompt should lokks like
        function jobcount {
            jobs | wc -l | tr -d " "
        }
        export PS1='$(jobcount):\u@\h:\w\$ '

        # less is much better than more
        export PAGER=/usr/bin/less

text editor

vim is the editor I use, corresponding configuration file (~/.vimrc). From time to time I switch to plain old vi on some unix systems like IBM AIX or SUN SOLARIS, nvi on BSDs or elvis on linux. I do switch to these editors when vim is configured in gui mode, and/or when  it is to slow to get a decent console mode.

My .vimrc file is here. Its associated color scheme was inspired from moria and adapted so it can be used in a 256 color terminal, this is moria256.

email configuration

I use the following email architecture:

• mutt as my MUA, powerful (regex friendly)
• fetchmail as my MTA
• procmail as my MDA

corresponding configuration files are: muttrc, fetchmailrc and procmailrc. The imap gmail configuration file is here.

So far this setup has been very solid, mutt is a gem. No smtp client there, I use mutt for now but may switch to exim in a near future (no dsn with mutt for now).

firefox extensions

colorzilla: pickup colors from any web page

palette grabber: creates a color palette from a web page

webdeveloper: popular extension for web developers

foxytunes: listen to music in firefox

FireFTP: add ftp navigation to firefox

delicious: bookmarks from any browser

Dictionary Search: provides a very easy way to look-up  (or translate) a word in a Web page or an email

ScribeFire: this is a full-featured blog editor that integrates with your browser and lets you easily post to your blog

google toolbar: if you are a google user, I am sure you already know the infamous toolbar.

flashblock: disallow flash to play automatically – a real time saver

Evernote: replacement of google reader since as 2009, development on it has been stopped.

Read it later: saves a web page so you can read it later on.

download helper: downloads any video from a web page

pronounce: how to pronounce a word

picnick: edit images with a right click

firefox tweaks

disable close button on tabs: set to value 3

disable pre-fetching of web pages: set to false

sets the amount of memory firefox should use: set to 33 MB

spellcheck every text area: set to 2

turns off the rarely-used Go button at the end of the address bar: set to true

set browser.sessionstore.interval to 300,000, see dru lavigne post about it

Don’t get me wrong: I am not saying it’s a fad or something like that, I am just saying that it’s not something new at all, SMP machines exists for many
years now.
I started working with this kind of machine in 1997, a SGI machine with 16 processors. Old stuff that becomes new stuff, again …
Since then, when working for banking or financial companies, this has always been the motto: let’s do it faster. Add more ram, more cpus, more disks.
And, yes please, use them as efficiently as possible: I paid the big bucks for the hardware, the software must use it  (Nowadays the companies willing to spend the most time and money on multi-core programming will probably be the gaming companies).

So … We have more computing power now and we want to use it efficiently, we want to keep these cpus busy.
Of course this is not easy, in order to get a faster program, having more than one cpu does not make automatically any of your applications faster. One must choose the right technology in order to achieve the most efficiency out of these new multi-core cpus.

Multi-threading is the most common chosen technology to try to use the multiple cores available on your $600 desktop or +10,000$ on your brand new server.
But multi-threading is not the only technology that does exist to maximize your computing efficiency nor it is the right technology to solve every problem: multi-processing is also a good choice under certain circumstances.

So you want to optimize or develop a new program that would scale when in presence of multiple cores/cpus, here are the steps I follow when trying it:

First things first: check the operating system the program is supposed to run on.

• why: not all operating systems are created equal when dealing with multiple cpus. Even if your preferred OS claims to support multiple cores it may not be able to use them efficiently i.e. where are the  GL contention point in your OS ?

• example: if your new application is a networking app and you think that multiple cpus will make it faster you should check if your OS implements a BGL around networking APIs.

• conclusion: check your OS internals regarding areas where you expect multiple core to speed-up your application. usually I use SOLARIS and Linux, I tend to avoid MAC OSX and other UNIX like HP-UX and IBM AIX. I do not use windows boxes, hence I cannot make any comment on them.

I will not speak of correct time-sharing and scheduling in presence of multiple cores since all these OS have good support for those.

choose between multi-threading and multi-processing.
As I previously said, multi-threading is not always the right choice for your multi-core applications. This always a shock to people I talk with, since for them multi-threading is the only answer to multi-core programming.
So here are a description of situations where multiple processes may be a better choice than multiple threads in a single process:

• robustness, multiple threads share the same address space, hence if one generates a fault, the whole application may crash. Multi-thread applications do not run in complete isolation, so if certain threads of your program MUST stay alive no matter what can happen to the other threads, multi-threading may be a bad choice. If restarting the program upon failure is not an issue, you can safely ignore the robustness requirement.
• integrity, when designing multiple processes application, each process has its own address space so if one process goes crazy, it cannot corrupt the other processes data, it can only corrupt the shared data between processes. On the other hand, if a thread goes crazy it can corrupt the other threads data since there is no memory protection between threads. This is particularly true when a parent spawns a child process, the child process does whatever computation it needs, and then terminate. If any corruption occurs when doing the computation, it just goes away when the child dies: a new process is spawn to handle a new computation without being affected the previous child fault. There is no such safety net with threads. Corruption made by one thread is program wide, regardless of its lifetime.

Now a description of when a multi-threading application is more desirable than a multi-process one:

• CPU bound applications, video/sound/images processing, numerical simulations (finite elements, molecules interactions)
• high bandwidth message passing, this is when multiple threads exchanges messages for processing at very high rate i.e. the processing done on the various messages is very lightweight.
• cache friendly code, this is related to CPU bound applications
• model driven, multiple threads just make your coding easier: 1 request to be processed by 1 thread, like network connections etc. (yep this is naive e.g. see C10K problem, high performance server design)

Still not decided between multi-threaded and multi-process application ?

Measure the time necessary for the computations, if this amount of time is really small i.e. lightweight computation, compare it to the time taken to increment/decrement a semaphore between 2 processes. If they are in the same order of magnitude, you probably be more satisfied with multi-threading
Otherwise, multi-processing is a very viable choice. Multi-processing is a good choice when the computation takes an order of magnitude more time than the time spent exchanging data.

reduce contention
See ddj article – that’s what will kill your application (regardless of multi-threading or multi-processing).

• reduce code size that is protected by mutexes/semaphores i.e. a whole numerical simulation between two threads is a bad idea

• reduce the necessity for locking, fastest way of sharing data between two threads ? No sharing at all !!! if you are in a master/slave processing model, try to have the master make a copy of the data to be protected by the mutex/semaphore and feed the copy to the various slaves. Copying the shared data can be way much cheaper than competing for a lock (especially when the number of threads increase).

• try to use dirty-reads, do you really need to know the exact value of a shared data i.e. the value of a counter.
  Example: you have to remove items from a cache when the item is expired. The item expires at 11:00, the current timestamp value is protected by a mutex. Locking can be avoided, since reading 11:00 or 11:01 (it has been updated by another thread while reading it) does not really mater: as soon as the timestamp is greater than 11:00 deletion can be performed. Your counter/timestamp is quite big ? See if Lamport concurrent reading and writing of clocks algorithm applies to you (it’s simple and effective).
• do not forget that memory is not cheap to access, check some machine architecture docs to educate yourself on these low levels issues

If dealing with a CPU bound application, beware of trashing: that beast will decrease your performance while maintaining your various cores busy

If dealing with a CPU bound application, maximize the use of your cpu architecture. This is quite a last resort action, but it can be very effective.
For example, I had to a given computation on a powerpc processor. The computation involved multiple floating point operations. The powerpc is a beautiful beast, you have multiple floating point engines when doing the calculations (in my case 4 if I remember correctly).

The trick was to make the compiler produce code that use all of them.

The  following code and discussion are not about loop unrolling (yes it is closely related), here we try to avoid stalling the cpu pipeline by using the multiples floating point units.

original code (and simplified code):

float* a = new float[1000];
float* b = new float[1000];
float* c = new float[1000];
// most cpu time was spent in there...
for ( int i = 0; i < 1000; ++i )
{
    c[i] = 0;
    for ( int j = 0; j < 1000; ++j )
    {
        c[i] += a[j] * b[j];
    }
}

updated loop:

for ( int i = 0; i < 1000; ++i )
{
    float partial_sum = 0;
    for (int j = 0; j < 1000; j += 4)
    {
        // load data into the various processor registers
        float a1 = a[j];
        float a2 = a[j+1];
        float a3 = a[j+2];
        float a4 = a[j+3];
        float b1 = b[j];
        float b2 = b[j+1];
        float b3 = b[j+2];
        float b4 = b[j+3];

        // compute all sums in isolation, so they can be mapped
        // independantly to the various floating point engines
        float s1 = a1 * b1; // FPU engine 1
        float s2 = a2 * b2; // FPU engine 2
        float s3 = a3 * b3; // FPU engine 3
        float s4 = a4 * b4; // FPU engine 4

        s1 += s2; // FPU engine 1
        s3 += s4; // FPU engine 2

        partial_sum += s1 + s3; // FPU engine 1
    }
    c[i] = partial_sum;
}

This is longer but much more efficient and related to multi-threading: you are using the concurrency offered by your processor at the operation level: s1, s2, s3, s4 are computed in parallel, then s1 and s3 are computed in parallel again. This is very good, and most of the time, neglected source of concurrency for your programs.

The previous code, the updated loop, will probably be faster even on classic x86 chips with only 1 FPU, reason being that placing data in registers (explicitly) will be faster than retrieving the data from the cache.

Forget about the fact that this is a powerpc processor and there are multiple FPU engines etc.
What we really did here, was to provide the compiler with lot of context, on how the operation should be done and handled. After that the compiler can work on the optimizations, register allocations etc. Always provide your compiler a maximum of context on the computation to be performed.

You are not alone, seek advices and help
The conclusion to this post is: think twice before writing a multi-core application, check the os, your design and implementation with these previous notes before writing any line of code. Get some help from old schools SMP programmers, get them to review the design and get them to give you some tips. Multi-core programming is hard, really hard.

That’s all folks, for now. I’ll probably make a few other posts about multi-core programming, I have yet to decide the material to write on ;-)

Enjoy and share.