Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Saturday, July 19, 2008

New Hard Drive - Determining Best Parameters for ext3 File System

I've install a new hard drive on my laptop. Got a 320 GB Western Digital Passport and followed instructions here to extract the 320 GB SATA hard drive and replace my old 100 GB hard drive. I ended up mangling a plastic fastener in the hard drive enclosure like the author of the instructions did. But it still closes fine and now holds the old hard drive.

EDIT: Another guide (with pictures!) for disassembling this external drive. Might be better to follow this page.

In moving contents of the old hard drive to the new, I decided to stick with ext3. I've read JFS is faster, but I really like being able to access my linux partitions while using Windows. Also, this thread seems to suggest that ext3 is better at recovering from disk errors.

I want good parameters for partitions and file systems on the new drive. So, using tools like "df" and "tune2fs", I found out some statistics on the files residing on my old partitions.

I use two non-swap partitions for linux. A partition for "/home" and a partiton for everything else ("/"). "/" partition used 4.2 GB with 209893 items (files, directories, symbolic links...). In the "/home" partition, 61681 items took up 30.8 GB. From this, I've determined that "/" partition will be 20 GB and "/home" will be 140 GB on the new hard drive.

On the "/" partition, 42% (1,250,630/3,166,498) of blocks and 14% (209,301/1,609,344) of inodes were used. On the "/home" partition, 88% (10,061,413/12,016,356) of blocks and 2% (86,576/6,087,360) of inodes were used. I imagine the disparity in blocks and inodes used for "/home" results from large files I stored there. As both partitions used 4 KiB blocks, the ratio of bytes to inode were 8,059 for "/" and 8.085 for "/home". Clearly, past usage shows "/home" partition could do with larger bytes/inode ratio. As the laptop will continue to be used exclusively by me, I feel it's safe to change the ratio to 10,000 for "/" and 200,000 for "/home" without worry of running out of inodes.

Also, I'll set inode size to 256 in both partitions. 256 byte inodes improve performance for some applications and allow smooth upgrade to ext4. While older kernels won't be able to use the file system and grub had trouble booting on ext3 partitions with 256 byte inodes. But I'm not concerned as I won't use older kernels and grub has been patched. Ext2 IFS, the Windows driver I had been using to access the ext3 partitions, doesn't seem to support this setting so I may go with Crossmeta or ext2fsd as latest version of those drivers should support 256 byte inodes.

To determine the best blocks size, I ran some "find" commands to find out the distribution of the items based on their size:

ubuntu@ubuntu:/media/disk-1$ sudo find | wc -l
209893
ubuntu@ubuntu:/media/disk-1$ sudo find -size +1k | wc -l
125940
ubuntu@ubuntu:/media/disk-1$ sudo find -size +2k | wc -l
101253
ubuntu@ubuntu:/media/disk-1$ sudo find -size +3k | wc -l
88251
ubuntu@ubuntu:/media/disk-1$ sudo find -size +4k | wc -l
58733
ubuntu@ubuntu:/media/disk-1$ sudo find -size +5k | wc -l
53168
...


Resulting data:

Item Distirbution by Size
size"/""/home"
All20989386567
≤1KiB8395317953
>1KiB, ≤2KiB246875951
>2KiB, ≤3KiB130024934
>3KiB, ≤4KiB295186902
>4KiB, ≤5KiB55652536
>5KiB, ≤6KiB42101860
>6KiB, ≤7KiB33941429
>7KiB, ≤8KiB28951232
>8KiB, ≤9KiB26091054
>9KiB, ≤10KiB2787955
>10KiB, ≤11KiB2285767
>11KiB, ≤12KiB2133878
>12KiB, ≤13KiB1793803
>13KiB, ≤14KiB1509852
>14KiB, ≤15KiB1313942
>15KiB, ≤16KiB13881110
>16KiB, ≤32KiB1069614588
>32KiB, ≤64KiB793810158
>64KiB, ≤128KiB39243666
>128KiB20977997


More than half the files in "/" partition were 2 KiB or smaller. In the "/home" partition, however, only around 28% fell into this category and only 41% were smaller than 4KiB.

I ran the commands again with "-type d" option to find size distribution of directories. Because both of the old partitions used 4 KiB block sizes, the size attributed to the directories are all multiples of 4 KiB. This explains how the distribution for both partitions seem to deviate from the decreasing pattern at 3 KiB to 4 KiB range. Most of the directories uses a single 4 KiB block and causes the spike.

Though "/" may benefit from smaller block size, 4 KiB doesn't seem bad. I'll leave that as is.

I've decided to preserve some of other configuration from the old file systems. So following are the resulting mkfs commands I'll be running:

For "/", mkfs.ext3 -c -b 4096 -i 10000 -I 256 /dev/sda3

For "/home", mkfs.ext3 -c -b 4096 -i 200000 -I 256 /dev/sda4

Other options specified in the "/etc/mke2fs.conf" file seemed good enough (sparse_super,filetype,resize_inode,dir_index). Hopefully all goes well.

UPDATE: Output from running mkfs is below. Might come in handy for data recovery.

ext3 -c -b 4096 -i 200000 -I 256 /dev/sda4;
mke2fs 1.40-WIP (14-Nov-2006)
Warning: 256-byte inodes not usable on older systems
warning: 380 blocks unused.

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
2000176 inodes, 4882432 blocks
244140 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
149 block groups
32768 blocks per group, 32768 fragments per group
13424 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Checking for bad blocks (read-only test): done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
mke2fs 1.40-WIP (14-Nov-2006)
Warning: 256-byte inodes not usable on older systems
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
691488 inodes, 33709204 blocks
1685460 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1029 block groups
32768 blocks per group, 32768 fragments per group
672 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Checking for bad blocks (read-only test): done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.


UPDATE2: After doing some reading, I've decided to create an ext2 boot partition. It seems the only sure-fire way to have grub boot and have the other partitions be ext3 with 256 byte inodes.

Wednesday, February 6, 2008

Video Playback Problem Fixes

Have been putting up with bad color in video playback. I would adjust the "Hue" setting in the media player but the colors were always off. Turns out the culprit is ATI's video driver. That'd explain why video applications had this problem but Flash-based video applications didn't. Flash must not use video card acceleration for video playback. Not on my computer, at least.

The exact problem is that a bug in ATI's driver swaps U and V color coordinates in YUV color space.

The forum thread about the problem suggests the following solution:

launch gstreamer-properties from terminal
change the video output plugin to custom
change the video output pipeline to:

ffmpegcolorspace ! video/x-raw-yuv,format=(fourcc)YV12 ! xvimagesink


Yet another problem with video playback I've been having is that some files will have a green band on top when played in VLC. I thought it was a deinterlacing problem since some deinterlace option changed the size of the green band. But There was an online discussion about this also. The fix is to change VLC preferences to use a differnt video output module. Using X11 video output module instead of the "Default" option got rid of the green band. As a bonus, this change also got rid of an artifact where fast-changing image would show that the video was being rendered in two steps triangles.

These problems counted two in the ever-shrinking list of reasons to have Windows around. Not many more left. I think I can safely delete my Windows partition soon.

Saturday, November 17, 2007

OpenWRT WPA Gives Me Problems... and WEP Works!

So I thought WPA was working perfectly. But then it started dropping out on me every now and then. I can't find a pattern to it. But I did leave the SSH connection to the router open on my desktop and the error message "wl0: Invalid argument" was displayed repeatedly.

Sometimes the connection get reestablished after a while. Sometimes I get impatient and reboot the router.

So I gave the instructions for WEP setup on OpenWRT Kamikaze one more try, except this time instead of using my old WEP key I followed all the steps and generated a new key as instructed. And now 128-bit WEP works! I'll have to see if it's stable but I'm hopeful.

Also, "Encryption key:<too big>" still appears when I run "iwconfig". Maybe 128-bit key is too big to display in what they allocated for the field to use? I dunno.

Wednesday, November 14, 2007

OpenWRT WEP Trouble

I had some trouble with enabling 128-bit WEP encryption on OpenWRT Kamikaze 7.09 (kernel 2.4). I'd run iwconfig and see "Encryption key:<too big>" but could not figure out how the key could be too big as I entered into "/etc/config/wireless" file, under "key" option, a 26-digit hexadecimal number as the WEP key.

As to be expected someone else ran into the same problem and discussed it in OpenWRT Forum and references a link with the solution. The solution did not work for me, though.

So I followed the instructions here to use WPA instead of WEP. It's working great. Don't know why WEP wouldn't work.

Afterthought: The router I am using is Linksys WRT54GL v1.1. I didn't try WPA2.

Sunday, November 11, 2007

OpenWRT

Installed OpenWRT on my wireless router (Linksys WRT54GL). Default firmware from Linksys wouldn't do DHCP and static IP allocation at the same time. At least I couldn't find a way to do it.

Default installation of OpenWRT does not come with an HTTP server so all configuration is done over SSH sessions. I could always install a web server with configuration manager but it works well as it is.

Spent a good chunk of my time with configuration. One problem I had was that I was following instructions for configuring a previous version - White Russian - rather than the current - Kamikaze. Information on getting Kamikaze to do what you want seems rather fragmented. Might be because it's new.

Plan to get it to update DynDNS domain and forward some ports to my desktop.

The desktop was given a static IP by configuring DHCP on the router. I found a discussion about setting up and troubleshooting static IP addresses. I configured dnsmasq by adding MAC-IP address mapping in /etc/ethers file (and made it readable by user "nobody").

Other links:
Configuring DHCP and DNS on OpenWRT