...making Linux just a little more fun!
Dr. Parthasarathy S [drpartha at gmail.com]
I have come across this thing called FIFO named pipe. I can see how it works, but I find no place where I can put it to good use (in a shell script). Can someone give me a good application context where a shell script would need to use FIFO ? I need a good and convincing example, to be able to tell my students.
Thanks TAG,
partha
-- --------------------------------------------------------------------------------------------- Dr. S. Parthasarathy | mailto:drpartha at gmail.com Algologic Research & Solutions | 78 Sancharpuri Colony | Bowenpally P.O | Phone: + 91 - 40 - 2775 1650 Secunderabad 500 011 - INDIA | WWW-URL: http://algolog.tripod.com/nupartha.htm ---------------------------------------------------------------------------------------------
Kapil Hari Paranjape [kapil at imsc.res.in]
Hello,
On Tue, 22 Jun 2010, Dr. Parthasarathy S wrote:
> I have come across this thing called FIFO named pipe. I can see how it > works, but I find no place where I can put it to good use (in a shell > script). Can someone give me a good application context where a shell > script would need to use FIFO ? I need a good and convincing example, > to be able to tell my students.
This is not an example but a proto-example.
Suppose you need to provide some way for people/accounts to send you data AND you want to control (using unix permissions) who can send the data.
In that case you can use a fifo with appropriate unix permissions which people can write to. Your program then reads this data and processes it.
Here is an example. You want members of the (unix) group 'fifo101' to send you questions but they want to be sure that their submissions are anonymous and you want to check that these are indeed questions being submitted.
IF they understand the fifo AND they trust that there is no one (including system administrator) who is peering over their shoulder, then creating a fifo for them to dump their questions to is one easy way to do this. Your program reads from the fifo and saves the data to a file after checking that it is indeed a legitimate question (using some kind of regular expression!).
Regards,
Kapil.
Ben Okopnik [ben at linuxgazette.net]
Hi, Partha -
On Tue, Jun 22, 2010 at 09:09:46AM +0530, Dr. Parthasarathy S wrote:
> I have come across this thing called FIFO named pipe. I can see how it > works, but I find no place where I can put it to good use (in a shell > script). Can someone give me a good application context where a shell > script would need to use FIFO ? I need a good and convincing example, > to be able to tell my students.
FIFOs occupy an important but rarely-used niche in the *nix world. Most of the time, you don't need them... but when you do, there's nothing else that will work. You can think of them as the filesystem version of sockets: a method for communicating between a running process and a program, or between two running processes. The thing that makes FIFOs so useful is that they act as a buffer: the "source" process fills the FIFO by writing to it, while the "target" process or program empties it out by reading from it (and blocks if there's no content, or until there is content.) This allows you to do the data collection and processing asynchronously.
Like sockets, FIFOs can be a little tricky to explain in simple words (at least for me; I've met people who have managed to explain very complex concepts in incredibly simple, punchy ways. I hold that kind of skill - or is it a talent? - in awe.) I've rarely found any need to explain them - from, say, a shell-scripting perspective, they're a fairly advanced topic - but I believe that it's important to understand them, for your own knowledge and competence at least.
I can give you a few examples of where I've used FIFOs in my own scripting/programming; perhaps you can extract a minimal-length demo out of these for your students, or come up with your own example.
1) Imagine the following situation: you have a 10GB file called 'mysql_data.gz', and you need to load its contents into a database on a remote machine. You're pretty certain that your local machine does not have enough diskspace left to hold the uncompressed version of this file. What to do?
# Fire up the reader process - MUST do this first! mysql -h<hostname> -u<username> -p<password> <dbname> LOAD DATA INFILE '/tmp/loader.sql' INTO TABLE table; # Create the named pipe mknod -p /tmp/loader.sql # Fire up the writer process gunzip -c mysql_data.gz > /tmp/loader.sql
Assuming that you have a fast network, the reader process will be emptying out the FIFO as the writer process is filling it. Even if the writer is significantly faster, you're still going to gain a lot of advantage, speed-wise, by doing it this way.
2) If you are running some sort of a dual conversion on a very large file - say, you have a utility that converts SWF to MPG, and only allows writing to an output file, but what you actually want as the end product is an AVI - then you can avoid the huge intermediate files by using a named pipe as the temporary "receiver" (the MPG file.) Sure, it will "inflate" while the first process fills it up - but it will "deflate" down to nothing when the second conversion process, MPG->AVI, reads it. This is especially useful if you're using a script to convert multiple files.
3) I wanted to run a tiny xterm window that displayed a filtered version of the new content of /var/log/mail.info (this was my preferred version of "biff"; it let me know when new mail came in, showed me the subject, etc.) Initially, I tried implementing it as some version of "xterm -e tail -f /var/log/mail.info", but I discovered that this was too fragile: the xterm would "lose track" of the data from "tail -f" and lock up (at least it wouldn't display any new information.) So, I made the process much more robust by separating out the pieces and feeding the data through a FIFO.
Here's the script, with (hopefully) enough comments to explain everything. Note that you can also use 'mkfifo' to create these named pipes; I tend to stick with 'mknod' because it's a bit more portable (i.e., works with a variety of Unixen.)
#!/bin/bash
# Created by Ben Okopnik on Mon Mar 7 08:49:20 EST 2005
# Creates a tiny window showing mail retrieval
# Check to see if this process is already running by looking for the
# xterm title (MailTail) in 'ps' via 'pgrep'
[ -z "`pgrep -f MailTail`" ] && {
# Create the FIFO
mknod /tmp/.mailtail p > /dev/null 2>&1
# Make sure the FIFO will be destroyed on exit
trap 'rm /tmp/.mailtail' 0
# Copy data to FIFO
tail -n 1 -f /var/log/mail.info > /tmp/.mailtail 2>/dev/null &
# Fire up xterm; read .mailtail via Perl-based filter
xterm -fn 6x9 -bg black -fg yellow -geometry 40x2-0-0 -T MailTail -name MailTail -e perl -wlne\
'next unless /reading message\s+(.*\s*)
\d+ of \d+\s+\(\d+)\s+octets/; print "$1: $2)"' \
/tmp/.mailtail
}
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Dr. Parthasarathy S [drpartha at gmail.com]
Thanks Ben. It will take me a lot of time to really understand and then explain all the stuff you have given me. This time the writer (Ben) has fileld the FIFO. Now the reader (Partha) will read the FIFO.
Thanks again,
Will keep you informed (or even write up a case study for LG)
partha
-- --------------------------------------------------------------------------------------------- Dr. S. Parthasarathy | mailto:drpartha at gmail.com Algologic Research & Solutions | 78 Sancharpuri Colony | Bowenpally P.O | Phone: + 91 - 40 - 2775 1650 Secunderabad 500 011 - INDIA | WWW-URL: http://algolog.tripod.com/nupartha.htm ---------------------------------------------------------------------------------------------
Lew Pitcher [lpitcher at teksavvy.com]
On June 21, 2010 23:39:46 Dr. Parthasarathy S wrote:
> I have come across this thing called FIFO named pipe. I can see how it > works, but I find no place where I can put it to good use (in a shell > script). Can someone give me a good application context where a shell > script would need to use FIFO ? I need a good and convincing example, > to be able to tell my students.
I'm not certain that this is either a good or convincing example, but then again, named pipes (FIFOs) are an arcane part of Unix IPC and simple examples are usually neither good nor convincing.
As I said, named pipes are a part of Unix IPC, and that implies that there are two (or more) processes running. In the case of my example below, a single "server" process writes to the FIFO, while multiple "client" processes read from it. Here, the "server" generates passwords for users, while each client retrieves a single password; think of this as a simple "I need a new password" server and its clients.
The assumptions for this demo: - both server and client(s) are run in the same working directory - mkfifo is not a priviledged command, and can be run by non-root users - (at least) two shell sessions are running
First, for the server side: In a separate shell session, cwd to the target directory and mkfifo PassWords while true ; do dd if=/dev/random bs=6 count=1 2>/dev/null | \ mmencode 2>/dev/null | \ tr '+/=' 'AZ9' >PassWords done
Next, the client side: In a separate shell session, cwd to the target directory and cat PassWords
Repeated attempts to cat PassWords will print single, random passwords.
Neil Youngman [ny at youngman.org.uk]
On Tuesday 22 June 2010 17:37:34 Ben Okopnik wrote:
> The thing that makes FIFOs so > useful is that they act as a buffer: the "source" process fills the FIFO > by writing to it, while the "target" process or program empties it out > by reading from it (and blocks if there's no content, or until there is > content.)
The only time I've used FIFOs for real, I had syslog writing to a FIFO. Can you guess what comes next?
Neil
Mulyadi Santosa [mulyadi.santosa at gmail.com]
On Wed, Jun 23, 2010 at 21:42, Neil Youngman <ny at youngman.org.uk> wrote:
> The only time I've used FIFOs for real, I had syslog writing to a FIFO. Can > you guess what comes next?
put a "grep" that waits for this FIFO and took only the important string?
-- regards, Freelance Linux trainer and consultant blog: the-hydra.blogspot.com training: mulyaditraining.blogspot.com
Neil Youngman [ny at youngman.org.uk]
On Wednesday 23 June 2010 18:50:53 Mulyadi Santosa wrote:
> On Wed, Jun 23, 2010 at 21:42, Neil Youngman <ny at youngman.org.uk> wrote: > > The only time I've used FIFOs for real, I had syslog writing to a FIFO. > > Can you guess what comes next? > > put a "grep" that waits for this FIFO and took only the important string?
Logical, but I'm thinking along different lines.
Here's a hint. FIFOs block the writer when they're full. Back then IIRC 4kB was all it took to fill up a FIFO.
Neil
Ben Okopnik [ben at linuxgazette.net]
On Wed, Jun 23, 2010 at 08:48:36AM +0530, Dr. Parthasarathy S wrote:
> Thanks Ben. It will take me a lot of time to really understand and > then explain all the stuff you have given me. This time the writer > (Ben) has fileld the FIFO. Now the reader (Partha) will read the FIFO.
[laugh] It's too bad that you can't have multiple processes draining the buffer...
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Ben Okopnik [ben at okopnik.com]
On Wed, Jun 23, 2010 at 10:36:52AM -0400, Lew Pitcher wrote:
> On June 21, 2010 23:39:46 Dr. Parthasarathy S wrote: > > I have come across this thing called FIFO named pipe. I can see how it > > works, but I find no place where I can put it to good use (in a shell > > script). Can someone give me a good application context where a shell > > script would need to use FIFO ? I need a good and convincing example, > > to be able to tell my students. > > I'm not certain that this is either a good or convincing example, but then > again, named pipes (FIFOs) are an arcane part of Unix IPC and simple examples > are usually neither good nor convincing.
Boy, isn't that the truth? I've been teaching shell scripting for years, have read what seems like every single book available on the subject, and have never seen a good, simple example for a 'while' loop. Having given it some thought, I've come to the same conclusion you have: once you get to a certain level of complexity (and that might just mean a simple but niche-specific function), good examples aren't simple, and simple examples aren't good.
> mkfifo PassWords > while true ; > do > dd if=/dev/random bs=6 count=1 2>/dev/null | \ > mmencode 2>/dev/null | \ > tr '+/=' 'AZ9' >PassWords > done
Nice. For those who don't have emacs (and thus 'mmencode') installed, this would work:
while : do perl -MMIME::Base64 -wne'read(ARGV,$a,6); ($a=encode_base64($a,""))=~tr#+/=#AZ9#; print"$a\n"; exit' /dev/urandom>/tmp/PassWords done
Come to think of it, you could (as root) set up a "password device" in /dev by running the above in a system startup script - that would be a rather cool thing - although you'd probably want to use /dev/urandom as the entropy source.
--
OKOPNIK CONSULTING
Custom Computing Solutions For Your Business
Expert-led Training | Dynamic, vital websites | Custom programming
443-250-7895 http://okopnik.com
Neil Youngman [ny at youngman.org.uk]
On Wednesday 23 June 2010 20:28:15 Neil Youngman wrote:
> On Wednesday 23 June 2010 18:50:53 Mulyadi Santosa wrote: > > On Wed, Jun 23, 2010 at 21:42, Neil Youngman <ny at youngman.org.uk> wrote: > > > The only time I've used FIFOs for real, I had syslog writing to a FIFO. > > > Can you guess what comes next? > > > > put a "grep" that waits for this FIFO and took only the important string? > > Logical, but I'm thinking along different lines. > > Here's a hint. FIFOs block the writer when they're full. Back then IIRC 4kB > was all it took to fill up a FIFO.
The problem I hit was that after a reboot, the reader was not automatically restarted. The FIFO filled up rapidly, causing syslog to block, then processes writing to syslog blocked and after a short time the system seized up.
Neil
Ben Okopnik [ben at linuxgazette.net]
On Thu, Jun 24, 2010 at 11:20:51AM -0400, Ben Okopnik wrote:
> > Nice. For those who don't have emacs (and thus 'mmencode') installed, > this would work: > > `` > while : > do > perl -MMIME::Base64 -wne'read(ARGV,$a,6); > ($a=encode_base64($a,""))=~tr#+/=#AZ9#; > print"$a\n"; exit' /dev/urandom>/tmp/PassWords > done > ''
Come to think of it, you don't need all that if you're using Perl anyway:
while :
do
perl -wle'@a=("a".."z",A..Z,0..9);print map$a[rand at a],0..6'>/tmp/PassWords
done
FAQ: How do you convert a shell script into a Perl script? A: You don't convert; you rethink. Many of the problems solved by shell scripts either disappear or don't exist in the first place with Perl.
I've been answering that question for years now, and just got caught by
it myself. Oh well. 
There's a well-known statistic about lumberjacks
getting hurt on the job: the highest numbers occur in the first and in
the ninth year...
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *
Ben Okopnik [ben at linuxgazette.net]
On Thu, Jun 24, 2010 at 04:30:16PM +0100, Neil Youngman wrote:
> > The problem I hit was that after a reboot, the reader was not automatically > restarted. The FIFO filled up rapidly, causing syslog to block, then > processes writing to syslog blocked and after a short time the system seized > up.
Ouch. That's as good as the one I heard about someone having deleted
/dev/null.
-- * Ben Okopnik * Editor-in-Chief, Linux Gazette * http://LinuxGazette.NET *