The Journey to Journaling

pack your fs' diary seperately

From Licht Bülb

Answered By Dolavimus the platypus, Pretzel, Virtual Beer, Konqi, Tuxedo T. Herring, Swirl, corncob Pipe, the Scissors, Amanda the Panda

Hi Gang.

I'm fiddling on my laptop again (almost all things are working) trying to get s-video out working. In the course of this fiddling I'm raking wildly in the bios which sometimes screws up the display (and I am still able to login via ssh) or the machine freezes. Then I have to "push the button" and - since I was forced to use ext2 (ext3 accesses the hd every 5 secs -> spindown and therefore power saving impossible) I have to wait quite some time for the fsck to finish with my 10 GB root system (yeah, now I know why to have multiple partitions..). So,

does any of you know a jounaling fs which plays nice with laptops? I googled a bit, read stuff, but didn't find anything about it. I think I remember someone here in the TAG saying that reiserfs had some patch to play nice? Can someone confirm that?

Cheers and TIA

[Dolavimus] I've been using rfs on my laptop since new two yrs ago with good results. Although I haven't done anything in particular to address hd spin down for power economy.

[Amanda] I have actually used ext3 with a recently installed Debian "testing" distribution. The hdd access can be "reduced" by installing "noflushd".

noflushd doesn't work with any j-fs. It says so on the web page.

[Amanda] However, I confess that I haven't really had a chance to fully examine this issue. As regards partitioning the disk the following works well: /, /usr, /boot as ext2 mounted read-only. Of /tmp, /var and /home which need to be writable only /home really is usually large enough to require journalling ( and sometimes /var ).

I have tried several things over time: only one root and a small boot, full monty with all dirs on seperate partitions and some things in between. The prob with several partition: when you need some large space (naturally) none is there on a single partition. Across two or more there would be enough... Disadvantage of only one /: you can not unmount anything beforehand if you know you gonna crash the system now... And a crypto-fs is hard to make then too.

[Pretzel] I think the very idea of a journaling filesystem makes "play[ing] nice" impossible. Journaling filesystems have to access the hard drive on every write. More accurately, they have to access the journal device each filesystem write.

Well, it might be necessary for the j-fs to write to it's journal every fs access, but ext3 writes to hdd every 5 secs, regardless of fs access or not.

[Pretzel] I think most journaling filesystem in Linux have an option for journaling device which is normally on-disk but can be on any block device, at least with ext3 and reiserfs. Some non-volatile memory would do nicely, but on a laptop, I think the chances of being able to do this are almost nil.

I (as a not-knower) would have two ideas: a compact flash card in the pcmcia slot and - RAM! In case of the RAM the journal would then be written to hdd when also a normal hdd access takes place. True, this would be bad if a system outage would occur without the journal having been rewritten to disk but I would take these chances... In case of CF, dunno if you can plug them in straight away or if you need an adapter, but 16MB are really cheap anmd if you can save some power (-> time) with it...

[Pretzel] Doing it in RAM would effectivly make a journaling fs useless. What would be the point then? That's the same effect as using a non-journaled filesystem.

Well, the journal would get written to disk with the data. If you use noflushd then writes of the system (logging etc.) get postponed or get written to RAM and then get written to hdd if a normal write (user initiated) occurs. So, I dunno exactly how noflushd does this but when it redirects the writes to ram the journal entries (if they need to be made in that case) should be written to ram too.

[Scissors] This isn't so much "writtem to RAM" like scribbled in a ramdisk - it's more like being hidden in the RAM of a caching controller. That's all noflushd does, is allow some buffering at the filesystem-driver level. So if something really does have content for the disk - and yes, that includes its journals - it's either got to hit the disk eventually, or you get to bear the risk that something might fail before it does.

But the whole point of having a journal is to have it still be present after a reboot event made something which normally isn't volatile space, lose its cookies. Having a journal that isn't allowed to do its job just complicates matters. Ergo, it shouldn't be put on volatile RAM.

[Pretzel] Another possiblily compiling a kernel with magic sysrq support, if the machice isn't totally frozen, you could do an emergency Sync/Unmount/Reboot.

OK, now I have to admit it: HOW THE HECK DOES THIS WORK? I read the stuff in /usr/src/linux/docs/ but as much as I gathered Alt-Print d should do something? It doesn't in my case. I compiled this in (if it's only the Magic SysRq key in Kernel hacking) some time (not in my current kernel I see now) but then (I tested it) it didn't work.

[Pretzel] Worked for me. (I say worked because at the moment I don't have it compiled it.) Try Alt-Sysrq-<magic sysrq command> (all three at the same time.) I don't remember if it needed all three at once or not. sysrq probably only says "print screen" on some keyboards.

Yeah, the prob was all at the same time... Now it works.

[Scissors] At least part of the confusion is with SysRq - on some keyboards the SysRq lives as a subfunction on another key. Thus for such keyboards you'd also need the extra key that invokes the secondary keycode. Fn maybe.

When it works then you should be able to (for instance) press Z and get a little help list. In fact any character that doesn't do something is supposed to show the little list. What I'm not clear on is "get outta sysrq mode"...

I've sometimes seen a console get into a state where it would respond to Magic SysRq, but it couldn't get out of that mode anymore. So I hope you have some spare virtual consoles, if you are just using it to settle something simpler than "telinit 6 doesn't work".

[Pretzel] So, in summary: No.

A No? <shout> I DON'T ACCEPT NO "NO" ! </shout> (yes, I've been in the army..)

[Beer] Why not plug one of these USB memory sticks and keep the journals there? That way the immediate access is only required for the USB device and not for the actual HD.

Cool idea, hadn't thought of this one, although I recently bought one! <me stupid>

[Beer] Which one? I've seem that Sony ads -- but they tell a lot about some strange cruft with user mangerment and cryptosoftware which suggest custom WinXX drivers to my suspicious ears.

The cheapest one I could find. It was 49 Euro, super slim and works like a charm. I think I will use gpg to encrypt files on the thing and simply put a win gpg version on the stick too.

[Beer] If I ask for a drive give me drive and not an encryption device where I can't trust the encryption anyway....

True, true...

[Beer] Might still need patches, but if the journal keeping actually needs only access to the journmal file and is not accessing the HD where the data are finally going anyway -- It might even work with existing journal/kernel code.

Then on the other hand you want journaling for some testing period -- why is power saving durting that time so important? You could always switch back to unjournaled if power saving is important. ext2/3 in that case would be easiest as a cleanly unmounted ext3 is mountable as ext2 without problems. The switching between ext3 and ext2 might be done by boot options I guess. (can you ask in a init.d script for boot options ?)

Well, the case is pressing just now in this testing period. I just rebooted and had a full fsck (with Ctrd-d at the end) just now and I really need a j-fs NOW. But I like the idea of j-fs's generally so I would like to keep it after the fiddling too. You are right that ext2|3 would work now but I am a little bit burnt with that since I had ext3 in the very beginning, installed the system (a lot) and then wanted to switch to ext2 (with removing the journal) according to manual. But the ext-tools weren't current at that time in testing|sid so this broke (!) the fs with FULL reinstall (some bad things occured at the same time leading to this). But ext2|3 would be an option.

[Beer] I never tried to completely switch back to ext2 and delete the journals, but I did mount ext3 as ext2 and nothing bad happened. I made sure they are clean so.

On Laptop power save is only a real issue on battery -- then the sudden crashes should not be that frequent unless you insist of fiddling around in kernel space on the train.... So for some normal usage running on battery and ext2 might be sufficient, but when you are on mains power you can switch back to ext3. There must be a way to make this decission in lilo by some option which is evaluated in init.d/boot*

Or check if you are on battery and make the decision based on that. Frequent sleep/resume inbetween are then not that practical so.

[Konqi] For those living in Germany: Aldi has an 128 MB USB memory stick for 49.99 Euro next week, I read in an Heise announcement.

And for those who want to know something about mine: http://www.computer-cash-carry.de/ccc/index-ns.html see Festplatten IDE, there at the bottom (all german, just for the curious)

[Tux] Any of you guys want to write up an article on what it takes to get one working, or is it simple enough to describe in a 2-cent tip? I'd be very curious.

modprobe sd_mod
modprobe usb-storage
mount /dev/sda1 /mnt/usbstick

[Tux] Wow. Mondo cool. I love Linux.

I think that there's a USB memory stick in my near future...

[Swirl] I wonder if what I have on my website is sufficient for a 2-cent tip? I can re-write it, if it's considered too half-assed. Here's the relevant excerpt:

USB on Linux seems a bit... different, especially since I'm not really used to devfs-type things.

[Beer] The only thing I ever tried was a digital camera (Olympus C100 zoom) and it needed a kernel patch (usb-driver) to be properly recognised. Apart from that the camera works as a disk as expected. I mount /dev/sda1, it will be rw right on the first mount, no need for remounting.

So if you really have to go through all this remounting, you can't fdisk the thing into partitions, etc. It really seems strange.

I do remember that zip-drive "problem" -- the win programs to access it partition it to use sda4 instead of sda1. So this might all be very specific to the hardware and drivers used (or used the first time the disk was partitioned).

But basically these memory sticks seem to work, then I'll keep my eyes open for one.

[Swirl] I've been figuring out how best to deal with it on my laptop, since my wife gave me a 32 MB Easy Disk "memory stick", a cute little plastic thing on a keychain fob.

I'm still working it out, but it looks like I need the usb-uhci and usb-storage drivers (impliedly also usbcore), at which point I can do:


# mount -o rw,uid=1000,gid=1000 -t vfat /dev/sda /mnt/fob/

(where 1000 is my own login account's UID and GID).[1]

The bizarre thing is: That command (or anything like it) always returns:


mount: block device /dev/sda is write-protected, mounting read-only

[corncob Pipe] I believe that this error is more of a "fail-safe" than it is an annoyance...

[Swirl] I banged my head up against that problem for a couple of days, and "fdisk /dev/sda" kept insisting...

Um, my usb stick has a small switch on it's side to make it write-protected. You don't have by chance missed that one?

[Swirl] That was part of what I spent some part of a couple of days, looking for. No, there's no hardware switch. Moreover, doing a remount entirely via software _does succeed in making it writeable, following the initial mount. If the obstacle were a hardware switch, that would not have happened.


# fdisk /dev/sda
You will not be able to write the partition table.

...until it finally occurred to me to do...


# mount -o rw,remount /mnt/fob

[Pipe] Does it? To me, the mount command references "/mnt/fob" (unless of course, "/mnt/fob" points to "/dev/sda"???)

[Swirl] The point is that the _original mount command, which gets re-mounted here, uses /dev/sda, _not /dev/sda1. I'm trying to stress that at least _some memory sticks must be addressed as if they were SCSI-based floppy drives. No partition numbers, you see? This was non-obvious point #1.

[Pipe] What you have done here it to re-mount in in "rw" mode, yay!

[Swirl] Um, yes. That's exactly my point. The necessity to do so was non-obvious point #2.

[Swirl] Notice that the mount command references /dev/sda, instead of /dev/sda1. That's the other odd thing: I kept had no luck until I happened to try /dev/sda.

[Pipe] I think this is as a result of the way in which device files are referenced under Linux

[Swirl] Well, where were you when I was trying to mount the device, the first time?

I'm saying _the point is non-obvious_ because many other instructions one finds for mounting similar devices tell you otherwise, because intuition suggests that a memory stick ought to be addressed like a hard drive, such that it can have multiple partitions, and because it's the first time in a decade of addressing SCSI devices on Linux that I've not put a number after /dev/sda.

[Pipe] Oh, now I see what you meant. I mis-interpreted you the first time.

[Swirl] My guess is that the Easy Disk is being addressed as if it were a floppy disk instead of a hard drive partition, which is why it's sda instead of sda1.

[Pipe] Hmm, I don't quite follow you here. Why would a HD partition not utilise "sda1" if told to do so?

[Swirl] I'm not sure what's the nature of your confusion: Are you willing to simply take my word for it that addressing /dev/sda1 on the memory stick absolutely does not work, and that addressing /dev/sda does? That would make this conversation a great deal easier.

[Pipe] Oh, I'm not doubting you one bit, I just mis-read your sentence, I'm sorry, Rick.

[Swirl] And I believe you're missing my point: Because the Easy Disk is (it seems) classed as functionally the same as a floppy disk, there is no concept of partitioning, and thus no partition numbers.

[Pipe] And if Easy Disk were being addressed as a floppy then it ought to use "/dev/fdx" in this case, surely?

[Swirl] I beg your pardon, but no. That wouldn't be SCSI then, would it?

[Pipe] It would, it would.

[Scissors] I seem to recall that normal floppy disks used in an IDE-chain LS-120 bay are still accessed by the fdx mechanism. However the one around here lost a pin so I can't check.

[Swirl] I hope you're not under the impression that there aren't all Linux floppy disks must be addressed as /dev/fdx? SCSI ones are /dev/sdX, and parallel-port ones are /dev/pfN . Neither supports partitioning.

[Pipe] !

[Swirl] I have no idea why Linux always mounts it read-only, though.

[Pipe] Me neither.

I have no clue what it is recognised or attached like, but in my case I have to use /dev/sda1, else it will make trouble.

[Swirl] Yes, I've heard such reports. That is why I have concluded that some USB memory sticks are recognised as quasi-floppies, and some as quasi-hard disks.

I guess there are two (or more) controller producers and there comes this trouble.

[Swirl] If by "controller" you mean the USB chipset on the motherboard, I very much doubt that, because the difference seems specific to the add-on USB storage device, not the USB host.

No, I didn't mean the host, I meant the usb controller in the sticky. These need some controller too, don't they?

[Swirl] They need some sort of USB circuitry, and, to be sure, differences in such circuitry seem to have caused these differences in mode of operation between memory sticks. Whether it's common to call that circuitry a "USB controller", I am unsure, but such was not my impression.

Maybe there are even patents involved so that the other producer had to take a different approach (and maybe this also helped in case of "booting off floppy -> off usbstick). Anybody anything about the subject?

But there is a strange thing: sometimes there comes a error message saying that there is (can't rememeber what) wrong, then I try to mount /dev/sda, this will fail, and then I can mount /dev/sda1 - So much for deterministics.

[Pipe] Unless of course this is as a result of the Kernel's divine intervention??? Va savoir??

[Swirl] I'm ultimately less interested in kernel psychoanalysis than in making it do what I want, thanks.

Meet the Gang   1   2   3