Saturday, July 19, 2008

New Hard Drive - Determining Best Parameters for ext3 File System

I've install a new hard drive on my laptop. Got a 320 GB Western Digital Passport and followed instructions here to extract the 320 GB SATA hard drive and replace my old 100 GB hard drive. I ended up mangling a plastic fastener in the hard drive enclosure like the author of the instructions did. But it still closes fine and now holds the old hard drive.

EDIT: Another guide (with pictures!) for disassembling this external drive. Might be better to follow this page.

In moving contents of the old hard drive to the new, I decided to stick with ext3. I've read JFS is faster, but I really like being able to access my linux partitions while using Windows. Also, this thread seems to suggest that ext3 is better at recovering from disk errors.

I want good parameters for partitions and file systems on the new drive. So, using tools like "df" and "tune2fs", I found out some statistics on the files residing on my old partitions.

I use two non-swap partitions for linux. A partition for "/home" and a partiton for everything else ("/"). "/" partition used 4.2 GB with 209893 items (files, directories, symbolic links...). In the "/home" partition, 61681 items took up 30.8 GB. From this, I've determined that "/" partition will be 20 GB and "/home" will be 140 GB on the new hard drive.

On the "/" partition, 42% (1,250,630/3,166,498) of blocks and 14% (209,301/1,609,344) of inodes were used. On the "/home" partition, 88% (10,061,413/12,016,356) of blocks and 2% (86,576/6,087,360) of inodes were used. I imagine the disparity in blocks and inodes used for "/home" results from large files I stored there. As both partitions used 4 KiB blocks, the ratio of bytes to inode were 8,059 for "/" and 8.085 for "/home". Clearly, past usage shows "/home" partition could do with larger bytes/inode ratio. As the laptop will continue to be used exclusively by me, I feel it's safe to change the ratio to 10,000 for "/" and 200,000 for "/home" without worry of running out of inodes.

Also, I'll set inode size to 256 in both partitions. 256 byte inodes improve performance for some applications and allow smooth upgrade to ext4. While older kernels won't be able to use the file system and grub had trouble booting on ext3 partitions with 256 byte inodes. But I'm not concerned as I won't use older kernels and grub has been patched. Ext2 IFS, the Windows driver I had been using to access the ext3 partitions, doesn't seem to support this setting so I may go with Crossmeta or ext2fsd as latest version of those drivers should support 256 byte inodes.

To determine the best blocks size, I ran some "find" commands to find out the distribution of the items based on their size:

ubuntu@ubuntu:/media/disk-1$ sudo find | wc -l
209893
ubuntu@ubuntu:/media/disk-1$ sudo find -size +1k | wc -l
125940
ubuntu@ubuntu:/media/disk-1$ sudo find -size +2k | wc -l
101253
ubuntu@ubuntu:/media/disk-1$ sudo find -size +3k | wc -l
88251
ubuntu@ubuntu:/media/disk-1$ sudo find -size +4k | wc -l
58733
ubuntu@ubuntu:/media/disk-1$ sudo find -size +5k | wc -l
53168
...


Resulting data:

Item Distirbution by Size
size"/""/home"
All20989386567
≤1KiB8395317953
>1KiB, ≤2KiB246875951
>2KiB, ≤3KiB130024934
>3KiB, ≤4KiB295186902
>4KiB, ≤5KiB55652536
>5KiB, ≤6KiB42101860
>6KiB, ≤7KiB33941429
>7KiB, ≤8KiB28951232
>8KiB, ≤9KiB26091054
>9KiB, ≤10KiB2787955
>10KiB, ≤11KiB2285767
>11KiB, ≤12KiB2133878
>12KiB, ≤13KiB1793803
>13KiB, ≤14KiB1509852
>14KiB, ≤15KiB1313942
>15KiB, ≤16KiB13881110
>16KiB, ≤32KiB1069614588
>32KiB, ≤64KiB793810158
>64KiB, ≤128KiB39243666
>128KiB20977997


More than half the files in "/" partition were 2 KiB or smaller. In the "/home" partition, however, only around 28% fell into this category and only 41% were smaller than 4KiB.

I ran the commands again with "-type d" option to find size distribution of directories. Because both of the old partitions used 4 KiB block sizes, the size attributed to the directories are all multiples of 4 KiB. This explains how the distribution for both partitions seem to deviate from the decreasing pattern at 3 KiB to 4 KiB range. Most of the directories uses a single 4 KiB block and causes the spike.

Though "/" may benefit from smaller block size, 4 KiB doesn't seem bad. I'll leave that as is.

I've decided to preserve some of other configuration from the old file systems. So following are the resulting mkfs commands I'll be running:

For "/", mkfs.ext3 -c -b 4096 -i 10000 -I 256 /dev/sda3

For "/home", mkfs.ext3 -c -b 4096 -i 200000 -I 256 /dev/sda4

Other options specified in the "/etc/mke2fs.conf" file seemed good enough (sparse_super,filetype,resize_inode,dir_index). Hopefully all goes well.

UPDATE: Output from running mkfs is below. Might come in handy for data recovery.

ext3 -c -b 4096 -i 200000 -I 256 /dev/sda4;
mke2fs 1.40-WIP (14-Nov-2006)
Warning: 256-byte inodes not usable on older systems
warning: 380 blocks unused.

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
2000176 inodes, 4882432 blocks
244140 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
149 block groups
32768 blocks per group, 32768 fragments per group
13424 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Checking for bad blocks (read-only test): done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
mke2fs 1.40-WIP (14-Nov-2006)
Warning: 256-byte inodes not usable on older systems
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
691488 inodes, 33709204 blocks
1685460 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1029 block groups
32768 blocks per group, 32768 fragments per group
672 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Checking for bad blocks (read-only test): done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.


UPDATE2: After doing some reading, I've decided to create an ext2 boot partition. It seems the only sure-fire way to have grub boot and have the other partitions be ext3 with 256 byte inodes.