Ext3 as root filesystem?

[Andy Green]

Hi folks -

Next problem. Using current Fedora kernel. Custom initrd has the ext3 and
jbd modules, installs them, and is able to mount an ext3 partition, all
inside the initrd. That's great.

At the end of the initrd script, Linux wants to mount the root filesystem
again, okay, BUT the Redhat kernel does not have ext3 compiled in, its as a
module. This shouldn't matter because the initrd has loaded the jbd and ext3
modules. Yet despite this the kernel is unable to mount the same ext3
filesystem that the initrd was, apparently because ext3 was not compiled in
and is 'not on its list'. It instead mounts it as ext2 and then blows up
later in /etc/rc.sysinit when that tries fsck on it. The kernel commandline
rootfs=ext3 has no effect.

Any clues as to how Redhat manage to mount ext3 root filesystems or what is
wrong with my picture?

- -Andy

[Leonard den Ottolander]

Hi Andy,

Post by Andy Green
At the end of the initrd script, Linux wants to mount the root filesystem
again, okay, BUT the Redhat kernel does not have ext3 compiled in, its as a
module. This shouldn't matter because the initrd has loaded the jbd and
ext3 modules. Yet despite this the kernel is unable to mount the same ext3
filesystem that the initrd was, apparently because ext3 was not compiled in
and is 'not on its list'.

I don't think the second stage will use the modules from the initrd.

The module should be loaded on demand. Are you sure it is available under
/lib/modules/<version>/kernel/fs/? You might want to run depmod -a,
although that should be called by the init scripts as well.

Bye,
Leonard.

[Andy Green]

On Saturday 06 December 2003 13:32, Leonard den Ottolander wrote:

Hi Leonard -

Thanks for your reply :-)

Post by Leonard den Ottolander
I don't think the second stage will use the modules from the initrd.
Are you sure it is available under
/lib/modules/<version>/kernel/fs/? You might want to run depmod -a,
although that should be called by the init scripts as well.

I confess in the initrd I had copied them into /lib "to make things simple".
After your email I improved this to keep the same /lib/modules/<ver>/...
structure in the initrd, and I ran depmod with -a -b <fake root> on both the
initrd and the root filesystem at generation time, and the modules are now to
be found in the same place in the main root filesystem and the initrd.
(Depmod is done in /etc/rc.sysinit but not until after the fsck that is
blowing up).

In the meanwhile while trying things in /etc/fstab I managed to get the
/etc/rc.sysinit to fail to do the fsck properly, this means that the error is
skipped temporarily, and I find to my delight I do get through to a login
prompt and then into bash. So I am close.

Unfortunately even with the depmods and the improved directory structure for
the modules in the initrd, the kernel still insists on pulling in the ext3
root fs as ext2. The messages I am seeing just before init starts up are

EXT2-fs warning (device nbd(43,0)): ext2_read_super: mounting ext3 filesystem
as ext2
VFS: Mounted root (ext2 filesystem)
Trying to move the old root into /initrd ... okay
Freeing unused kernel memory: 120k freed

Having ext3 is important, because the root filesystem is on another machine
using the Network Block Device, it can be expected to experience rough
treatment.

Post by Leonard den Ottolander
The module should be loaded on demand.

What prompts the kernel to try to load the ext3 modules? If I try
rootfstype=ext3 the kernel seems to understand this to mean that the initrd
is ext3 (it is in fact ext2 since that's how the Fedora one is formatted).

Since this is all based on the Fedora initrd, I must be missing something or
doing something (else) stupid, because Fedora has no trouble coming up on
ext3 on my laptop HDD here :-(

- -Andy

[Andy Green]

Post by Andy Green
In the meanwhile while trying things in /etc/fstab I managed to get the
/etc/rc.sysinit to fail to do the fsck properly, this means that the error
is skipped temporarily, and I find to my delight I do get through to a
login prompt and then into bash. So I am close.

Another piece of information, after I come up in bash with my ext3 root
filesystem mounted as ext2 by the init script, I did an lsmod:

Module Size Used by Not tainted
ext3 65092 0 (unused)
jbd 47404 0 [ext3]
via-rhine 14192 1
mii 3736 0 [via-rhine]
nbd 15716 2

All of these modules were inserted by the initrd... ?8-(

-Andy

[Andy Green]

Post by Andy Green
In the meanwhile while trying things in /etc/fstab I managed to get the
/etc/rc.sysinit to fail to do the fsck properly, this means that the error
is skipped temporarily, and I find to my delight I do get through to a
login prompt and then into bash. So I am close.

Another piece of information, after I come up in bash with my ext3 root
filesystem mounted as ext2 by the init script, I did an lsmod:

Module Size Used by Not tainted
ext3 65092 0 (unused)
jbd 47404 0 [ext3]
via-rhine 14192 1
mii 3736 0 [via-rhine]
nbd 15716 2

All of these modules were inserted by the initrd... ?8-(

-Andy

[Leonard den Ottolander]

Hi Andy,

Post by Andy Green
I confess in the initrd I had copied them into /lib "to make things simple".

These kind of actions are (usually) unnecessary and should be avoided
unless you have a specific reason to do so. Ext3 is the file system of
choice on Red Hat and Fedora, so things are set up to "just work".

Post by Andy Green
After your email I improved this to keep the same /lib/modules/<ver>/...
structure in the initrd, and I ran depmod with -a -b <fake root> on both the
initrd and the root filesystem at generation time, and the modules are now
to be found in the same place in the main root filesystem and the initrd.

Are you using a standard Red Hat kernel? Then both the initrd and the
module layout should work as is. If you compiled your own all you should
have to do is a mkinitrd. Path layout should be usable as well.

Post by Andy Green
EXT2-fs warning (device nbd(43,0)): ext2_read_super: mounting ext3
filesystem as ext2

Are you sure you actually transformed the file system from ext2 to ext3
(tune2fs -j -c 0 -i 0 /dev/<partition>)? What does tune2fs -l
/dev/<partition> show you?

Post by Andy Green
Having ext3 is important, because the root filesystem is on another machine
using the Network Block Device, it can be expected to experience rough
treatment.

Using ext3 is also useful to avoid file system checks during boot.
Although you could achieve this by just issueing tune2fs -c 0 -i 0
/dev/<partition> on an ext2 partition. Of course you shouldn't be using an
ext2 file system in that state for too long to avoid file system errors.

Post by Andy Green
What prompts the kernel to try to load the ext3 modules?

The mount command itself, so the fact that the file system is described as
ext3 in /etc/fstab.

Post by Andy Green
Since this is all based on the Fedora initrd, I must be missing something or
doing something (else) stupid, because Fedora has no trouble coming up on
ext3 on my laptop HDD here :-(

I am sorry I have to agree with you on this ;-) . Ext3 should "just work".

Bye,
Leonard.

[Andy Green]

Post by Leonard den Ottolander
These kind of actions are (usually) unnecessary and should be avoided
unless you have a specific reason to do so. Ext3 is the file system of
choice on Red Hat and Fedora, so things are set up to "just work".

Its clear you're right.

Post by Leonard den Ottolander
Are you using a standard Red Hat kernel? Then both the initrd and the
module layout should work as is. If you compiled your own all you should
have to do is a mkinitrd. Path layout should be usable as well.

Yes, it is 2.4.22-1.2129.nptl. The kernel is taken straight from the binary
kernel RPM... I install it into the root filesystem and then copy it and some
modules over into the initrd filesystem.

This is my linuxrc modified from the Fedora one:

#!/bin/nash
echo
echo Mounting /proc filesystem
mount -t proc /proc /proc
echo Loading nbd.$KERNMODULESUFFIX module
insmod
/lib/modules/$KERNELVERSION/unsupported/drivers/block/nbd.$KERNMODULESUFFIX
echo Loading mii networking module" >> $RHINITMOUNT/linuxrc
insmod /lib/modules/$KERNELVERSION/kernel/drivers/net/mii.$KERNMODULESUFFIX
echo Loading via-rhine networking module
insmod
/lib/modules/$KERNELVERSION/kernel/drivers/net/via-rhine.$KERNMODULESUFFIX
echo Loading ext3 modules
insmod /lib/modules/$KERNELVERSION/kernel/fs/jbd.$KERNMODULESUFFIX
insmod /lib/modules/$KERNELVERSION/kernel/fs/ext3.$KERNMODULESUFFIX
echo getting network settings from DHCP
/bin/dhcpcd -t 5 -h tinycat eth0
echo Looking for nbd server on $4:$5
/bin/nbd-client $4 $5 /dev/nd0
echo 0x2b00 > /proc/sys/kernel/real-root-dev
echo End of linuxrc

Post by Leonard den Ottolander

Post by Andy Green
EXT2-fs warning (device nbd(43,0)): ext2_read_super: mounting ext3
filesystem as ext2

Are you sure you actually transformed the file system from ext2 to ext3
(tune2fs -j -c 0 -i 0 /dev/<partition>)? What does tune2fs -l
/dev/<partition> show you?

I used mkfs.ext3 to creat the filesystem, so I did not perform this
transformation action. I am able to mount the resulting filesystem as ext3
via loopback on my laptop, for example.

tune2fs 1.34 (25-Jul-2003)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 1c6a4d57-c00d-4635-bb41-d7f0b0a2bfbd
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal filetype sparse_super
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 87720
Block count: 350000
Reserved block count: 0
Free blocks: 123263
Free inodes: 57972
First block: 1
Block size: 1024
Fragment size: 1024
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 2040
Inode blocks per group: 255
Filesystem created: Sat Dec 6 14:35:44 2003
Last mount time: Sat Dec 6 15:02:32 2003
Last write time: Sat Dec 6 15:02:35 2003
Mount count: 1
Maximum mount count: 31
Last checked: Sat Dec 6 15:02:32 2003
Check interval: 15552000 (6 months)
Next check after: Thu Jun 3 16:02:32 2004
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 11b6998a-a3d6-46f9-8da3-e54fc35c7438

Post by Leonard den Ottolander
The mount command itself, so the fact that the file system is described as
ext3 in /etc/fstab.

Having the following generated into /etc/fstab (on both the initrd and root
filesystems, just in case):

/dev/nd0 / ext3 defaults
none /dev/pts devpts gid=5,mode=620 0 0

gets me the following from the /etc/rc.sysinit

Setting hostname localhost: [OK}
Checking root filesystem
/dev/nd0 is mounted. e2fsck: Cannot continue, aborting.
[FAILED]
then it drops to a repair shell.

Post by Leonard den Ottolander
I am sorry I have to agree with you on this ;-) . Ext3 should "just work".

No doubt there is some stupid thing I am doing... as usual I am operating at
or beyond what I know how to do :-) its the only way to learn new things...
but even so this is being quite painful...

- -Andy

[Leonard den Ottolander]

Hi Andy,

Post by Andy Green
Having the following generated into /etc/fstab (on both the initrd and root
/dev/nd0 / ext3 defaults
none /dev/pts devpts gid=5,mode=620 0 0
gets me the following from the /etc/rc.sysinit
Setting hostname localhost: [OK}
Checking root filesystem
/dev/nd0 is mounted. e2fsck: Cannot continue, aborting.
[FAILED]
then it drops to a repair shell.

How did you create the initrd? Note that /initrd usually is just the
directory where /boot/initrd-<version> (which is a gzipped file system)
gets mounted. Are you building an initrd in this directory?

Looking at the above messages it seems your problem is that / gets mounted
rw, which is why fsck can't check the file system. You need to mount it ro
on boot, so fsck can check it (or actually just establish it is an ext3
file system and skip that step), and remount it rw.

Bye,
Leonard.

[Andy Green]

Post by Leonard den Ottolander
Looking at the above messages it seems your problem is that / gets mounted
rw, which is why fsck can't check the file system. You need to mount it ro
on boot, so fsck can check it (or actually just establish it is an ext3
file system and skip that step), and remount it rw.

Hi Leonard -

Yes, you were right about needing ro on the kernel commanline, thank you.

I finally solved the ext3 boot problem a couple of days ago and am describing
the resolution here for future searchers.

The key thing was to never return from nash, instead to use the nash exec
command to replace nash with /sbin/init from the mounted root partition. So
the concept is you come up in nash from your initrd, load any modules you
need, including jbd and ext3 from the initrd filesystem, then mount your root
filesystem and pivot-root into it. THEN do not exit nash, but use the exec
nash command to spawn init in place of nash. The boot then proceeds
properly.

In the struggle to find the above solution, I discovered that the
real-root-dev stuff in the redhat initrd nash script is deprecated.

- -Andy

[Andy Green]

Post by Leonard den Ottolander
Looking at the above messages it seems your problem is that / gets mounted
rw, which is why fsck can't check the file system. You need to mount it ro
on boot, so fsck can check it (or actually just establish it is an ext3
file system and skip that step), and remount it rw.

Hi Leonard -

Yes, you were right about needing ro on the kernel commanline, thank you.

I finally solved the ext3 boot problem a couple of days ago and am describing
the resolution here for future searchers.

The key thing was to never return from nash, instead to use the nash exec
command to replace nash with /sbin/init from the mounted root partition. So
the concept is you come up in nash from your initrd, load any modules you
need, including jbd and ext3 from the initrd filesystem, then mount your root
filesystem and pivot-root into it. THEN do not exit nash, but use the exec
nash command to spawn init in place of nash. The boot then proceeds
properly.

In the struggle to find the above solution, I discovered that the
real-root-dev stuff in the redhat initrd nash script is deprecated.

- -Andy

[Andy Green]

Post by Leonard den Ottolander
These kind of actions are (usually) unnecessary and should be avoided
unless you have a specific reason to do so. Ext3 is the file system of
choice on Red Hat and Fedora, so things are set up to "just work".

Its clear you're right.

Post by Leonard den Ottolander
Are you using a standard Red Hat kernel? Then both the initrd and the
module layout should work as is. If you compiled your own all you should
have to do is a mkinitrd. Path layout should be usable as well.

Yes, it is 2.4.22-1.2129.nptl. The kernel is taken straight from the binary
kernel RPM... I install it into the root filesystem and then copy it and some
modules over into the initrd filesystem.

This is my linuxrc modified from the Fedora one:

#!/bin/nash
echo
echo Mounting /proc filesystem
mount -t proc /proc /proc
echo Loading nbd.$KERNMODULESUFFIX module
insmod
/lib/modules/$KERNELVERSION/unsupported/drivers/block/nbd.$KERNMODULESUFFIX
echo Loading mii networking module" >> $RHINITMOUNT/linuxrc
insmod /lib/modules/$KERNELVERSION/kernel/drivers/net/mii.$KERNMODULESUFFIX
echo Loading via-rhine networking module
insmod
/lib/modules/$KERNELVERSION/kernel/drivers/net/via-rhine.$KERNMODULESUFFIX
echo Loading ext3 modules
insmod /lib/modules/$KERNELVERSION/kernel/fs/jbd.$KERNMODULESUFFIX
insmod /lib/modules/$KERNELVERSION/kernel/fs/ext3.$KERNMODULESUFFIX
echo getting network settings from DHCP
/bin/dhcpcd -t 5 -h tinycat eth0
echo Looking for nbd server on $4:$5
/bin/nbd-client $4 $5 /dev/nd0
echo 0x2b00 > /proc/sys/kernel/real-root-dev
echo End of linuxrc

Post by Leonard den Ottolander

Post by Andy Green
EXT2-fs warning (device nbd(43,0)): ext2_read_super: mounting ext3
filesystem as ext2

Are you sure you actually transformed the file system from ext2 to ext3
(tune2fs -j -c 0 -i 0 /dev/<partition>)? What does tune2fs -l
/dev/<partition> show you?

I used mkfs.ext3 to creat the filesystem, so I did not perform this
transformation action. I am able to mount the resulting filesystem as ext3
via loopback on my laptop, for example.

tune2fs 1.34 (25-Jul-2003)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 1c6a4d57-c00d-4635-bb41-d7f0b0a2bfbd
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal filetype sparse_super
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 87720
Block count: 350000
Reserved block count: 0
Free blocks: 123263
Free inodes: 57972
First block: 1
Block size: 1024
Fragment size: 1024
Blocks per group: 8192
Fragments per group: 8192
Inodes per group: 2040
Inode blocks per group: 255
Filesystem created: Sat Dec 6 14:35:44 2003
Last mount time: Sat Dec 6 15:02:32 2003
Last write time: Sat Dec 6 15:02:35 2003
Mount count: 1
Maximum mount count: 31
Last checked: Sat Dec 6 15:02:32 2003
Check interval: 15552000 (6 months)
Next check after: Thu Jun 3 16:02:32 2004
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 11b6998a-a3d6-46f9-8da3-e54fc35c7438

Post by Leonard den Ottolander
The mount command itself, so the fact that the file system is described as
ext3 in /etc/fstab.

Having the following generated into /etc/fstab (on both the initrd and root
filesystems, just in case):

/dev/nd0 / ext3 defaults
none /dev/pts devpts gid=5,mode=620 0 0

gets me the following from the /etc/rc.sysinit

Setting hostname localhost: [OK}
Checking root filesystem
/dev/nd0 is mounted. e2fsck: Cannot continue, aborting.
[FAILED]
then it drops to a repair shell.

Post by Leonard den Ottolander
I am sorry I have to agree with you on this ;-) . Ext3 should "just work".

No doubt there is some stupid thing I am doing... as usual I am operating at
or beyond what I know how to do :-) its the only way to learn new things...
but even so this is being quite painful...

- -Andy

[Leonard den Ottolander]

Hi Andy,

Post by Andy Green
Having the following generated into /etc/fstab (on both the initrd and root
/dev/nd0 / ext3 defaults
none /dev/pts devpts gid=5,mode=620 0 0
gets me the following from the /etc/rc.sysinit
Setting hostname localhost: [OK}
Checking root filesystem
/dev/nd0 is mounted. e2fsck: Cannot continue, aborting.
[FAILED]
then it drops to a repair shell.

How did you create the initrd? Note that /initrd usually is just the
directory where /boot/initrd-<version> (which is a gzipped file system)
gets mounted. Are you building an initrd in this directory?

Looking at the above messages it seems your problem is that / gets mounted
rw, which is why fsck can't check the file system. You need to mount it ro
on boot, so fsck can check it (or actually just establish it is an ext3
file system and skip that step), and remount it rw.

Bye,
Leonard.

[Andy Green]

On Saturday 06 December 2003 13:32, Leonard den Ottolander wrote:

Hi Leonard -

Thanks for your reply :-)

Post by Leonard den Ottolander
I don't think the second stage will use the modules from the initrd.
Are you sure it is available under
/lib/modules/<version>/kernel/fs/? You might want to run depmod -a,
although that should be called by the init scripts as well.

I confess in the initrd I had copied them into /lib "to make things simple".
After your email I improved this to keep the same /lib/modules/<ver>/...
structure in the initrd, and I ran depmod with -a -b <fake root> on both the
initrd and the root filesystem at generation time, and the modules are now to
be found in the same place in the main root filesystem and the initrd.
(Depmod is done in /etc/rc.sysinit but not until after the fsck that is
blowing up).

In the meanwhile while trying things in /etc/fstab I managed to get the
/etc/rc.sysinit to fail to do the fsck properly, this means that the error is
skipped temporarily, and I find to my delight I do get through to a login
prompt and then into bash. So I am close.

Unfortunately even with the depmods and the improved directory structure for
the modules in the initrd, the kernel still insists on pulling in the ext3
root fs as ext2. The messages I am seeing just before init starts up are

EXT2-fs warning (device nbd(43,0)): ext2_read_super: mounting ext3 filesystem
as ext2
VFS: Mounted root (ext2 filesystem)
Trying to move the old root into /initrd ... okay
Freeing unused kernel memory: 120k freed

Having ext3 is important, because the root filesystem is on another machine
using the Network Block Device, it can be expected to experience rough
treatment.

Post by Leonard den Ottolander
The module should be loaded on demand.

What prompts the kernel to try to load the ext3 modules? If I try
rootfstype=ext3 the kernel seems to understand this to mean that the initrd
is ext3 (it is in fact ext2 since that's how the Fedora one is formatted).

Since this is all based on the Fedora initrd, I must be missing something or
doing something (else) stupid, because Fedora has no trouble coming up on
ext3 on my laptop HDD here :-(

- -Andy

[Leonard den Ottolander]

Hi Andy,

Post by Andy Green
I confess in the initrd I had copied them into /lib "to make things simple".

These kind of actions are (usually) unnecessary and should be avoided
unless you have a specific reason to do so. Ext3 is the file system of
choice on Red Hat and Fedora, so things are set up to "just work".

Post by Andy Green
After your email I improved this to keep the same /lib/modules/<ver>/...
structure in the initrd, and I ran depmod with -a -b <fake root> on both the
initrd and the root filesystem at generation time, and the modules are now
to be found in the same place in the main root filesystem and the initrd.

Are you using a standard Red Hat kernel? Then both the initrd and the
module layout should work as is. If you compiled your own all you should
have to do is a mkinitrd. Path layout should be usable as well.

Post by Andy Green
EXT2-fs warning (device nbd(43,0)): ext2_read_super: mounting ext3
filesystem as ext2

Are you sure you actually transformed the file system from ext2 to ext3
(tune2fs -j -c 0 -i 0 /dev/<partition>)? What does tune2fs -l
/dev/<partition> show you?

Post by Andy Green
Having ext3 is important, because the root filesystem is on another machine
using the Network Block Device, it can be expected to experience rough
treatment.

Using ext3 is also useful to avoid file system checks during boot.
Although you could achieve this by just issueing tune2fs -c 0 -i 0
/dev/<partition> on an ext2 partition. Of course you shouldn't be using an
ext2 file system in that state for too long to avoid file system errors.

Post by Andy Green
What prompts the kernel to try to load the ext3 modules?

The mount command itself, so the fact that the file system is described as
ext3 in /etc/fstab.

Post by Andy Green
Since this is all based on the Fedora initrd, I must be missing something or
doing something (else) stupid, because Fedora has no trouble coming up on
ext3 on my laptop HDD here :-(

I am sorry I have to agree with you on this ;-) . Ext3 should "just work".

Bye,
Leonard.

[Andy Green]

Hi folks -

Next problem. Using current Fedora kernel. Custom initrd has the ext3 and
jbd modules, installs them, and is able to mount an ext3 partition, all
inside the initrd. That's great.

At the end of the initrd script, Linux wants to mount the root filesystem
again, okay, BUT the Redhat kernel does not have ext3 compiled in, its as a
module. This shouldn't matter because the initrd has loaded the jbd and ext3
modules. Yet despite this the kernel is unable to mount the same ext3
filesystem that the initrd was, apparently because ext3 was not compiled in
and is 'not on its list'. It instead mounts it as ext2 and then blows up
later in /etc/rc.sysinit when that tries fsck on it. The kernel commandline
rootfs=ext3 has no effect.

Any clues as to how Redhat manage to mount ext3 root filesystems or what is
wrong with my picture?

- -Andy

[Leonard den Ottolander]

Hi Andy,

Post by Andy Green
At the end of the initrd script, Linux wants to mount the root filesystem
again, okay, BUT the Redhat kernel does not have ext3 compiled in, its as a
module. This shouldn't matter because the initrd has loaded the jbd and
ext3 modules. Yet despite this the kernel is unable to mount the same ext3
filesystem that the initrd was, apparently because ext3 was not compiled in
and is 'not on its list'.

I don't think the second stage will use the modules from the initrd.

The module should be loaded on demand. Are you sure it is available under
/lib/modules/<version>/kernel/fs/? You might want to run depmod -a,
although that should be called by the init scripts as well.

Bye,
Leonard.