ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Network Your Shell Scripts with Netpipes

by Robert Bernier
05/27/2004

There's no advantage to a sysadmin programming his own utility when he can do the same job (though perhaps less efficiently) in a quarter of the time with a quick and dirty shell script.

Unfortunately, bash scripts can't solve all problems. Take, for example, making socket connections to send and receive data streams. Conventional wisdom states that you'll have to use an existing service such as FTP, or create your own using a programming language such as Perl or C. I don't like these solutions: I don't like the idea of having a service that only one person uses. I don't like using full-blown programming languages on a server just to make a couple of simple socket connections. What I want is a command-line utility that anybody can use to set up TCP connections without worrying about root privileges. I want something that works with standard input and output, is easy to use in a script, and has a permissions system to restrict who can use it.

netpipes is a suite of utilities for shell-script writers that builds on the idea of conventional pipes to allow different processes to communicate and share data using both TCP and Unix domain-based sockets across the network! Not only does it duplicate the pipe's behavior, but it uses a novel technique called Session Control Protocol (SCP) that provides a simple mechanism for creating multiple, lightweight connections over a single TCP session connection. You can have many datastreams at the same time instead of just one.

This article will introduce you to a few of the netpipes command line utilities. Refer to the man pages for more information.

There are six command-line utilities in the netpipes suite:

  1. faucet: The server end of a BSD network pipe.
  2. hose: The client end of a BSD network pipe.
  3. encapsulate: Multiplexes several datastreams or channels over a single channel.
  4. timelimit: Caps how long a spawned subprocess can exist.
  5. sockdown: Selectively shuts down all or a portion of a socket connection.
  6. getpeername (or getsockname, in Debian): Returns information about the socket connection.

Installing netpipes from Debian's testing distribution is easy enough:

apt-get install netpipes

Netpipes is also available from the FreeBSD ports collection.

faucet

This utility runs the server end. This example invocation sets up a server connection on port 2000 and gives a simple "hello world."

~$ faucet 2000 --out echo hello world

The --out switch means that the server will spit out the message hello world as soon as a connection is made. You can also run this in the background as a daemon by adding the --daemon switch:

~$ faucet 2000 --out --daemon echo hello world

We can test this out using telnet:

~$ telnet localhost 2000
Trying 127.0.0.1...
Connected to wolf.
Escape character is '^]'.
hello world
Connection closed by foreign host.

The server can also carry out more than one instruction:

~$ faucet 2000 --out sh -c 
	"echo hello world; echo hi again; sockdown 1 1; ls / > /tmp/temp.txt"

This example echoes two statements to the client, hello world and hi again. It then closes the output connection using sockdown, another netpipe utility. The next instruction sends the output to the server's own file system in the file temp.txt.

Note: It's a good idea to enclose multiple commands within the bash shell command sh -c to avoid confusion.

hose

hose is the client utility. We can contact the first server example in the following manner:

~$ hose localhost 2000 --in cat
hello world
~$

The --in switch means that the client expects incoming information from the server. cat is of course the concatenate utility that takes input and puts it out as standard output.

The client can also send a datastream to the server. The server echoes the string hello it's my turn.

server: ~$ faucet 2000 --in sh -c "cat"
client: ~$ hose localhost 2000 --out sh -c "echo -e hello it\\047s my turn"

hose and faucet Combinations

This table shows several examples of simple piping -- one-way communications between the client and server.

Server (faucet)Client (hose)result
faucet 2000 --out echo hello world hose localhost 2000 --netslave hello world (comes out on the server's terminal console)
faucet 2000 --in echo I am hit, `date` hose localhost 2000 --netslave I am hit, Tue Mar 9 18:51:31 EST 2004 (comes out on the client's terminal console)
faucet 2000 --in sh -c "grep bernier - > temp1.txt" hose localhost 2000 --in --out sh -c "ps aux" Creates a file called temp1.txt on the server with information provided by the client.
faucet 2000 --out sh -c "ps aux" hose localhost 2000 --in --out sh -c "cat <&0>temp2.txt Creates a file called temp2.txt on the client with information provided by the server. Notice the use of redirection in cat <&0> temp2.txt.

Two-Way Server/Client Communications

Piping feeds the output of one process as input to another, but lacks the ability of running bidirectional communications. You can, however, create two-way data flow.

netpipes permits you to choose whether it's the client or server that sends, or receives, data upon connection by using the --in and --out switches in a specific order on both the client and server commands. For example, if the client sends data as soon as it connects to the server, then the first switch, reading from left to right, is --out. The server must complement this by using the --in switch.

A Single-Paired Transaction

The simplest interaction is a single-paired transaction. These two invocations are run separately in two difference consoles.

The first one to invoke is this one, which sets up a server waiting for a connection. The first one is:

faucet 2000 --in --out sh -c "cat <&0>from_client.txt;ps aux"

Two commands run on the server side. The first one, cat<&0>from_client.txt, takes the output from the contacting client connection as standard input and directs it into a text file called from_client.txt. The second command, ps aux, directs its output to the client stream. This output is sent only after the standard input that was sent by the client side has completed -- after the client has closed the socket connection.

This is the second process. It starts on the client side:

hose localhost 2000 --out --in sh -c "(ps aux;sockdown 1 1); \
	grep postgres > from_server.txt"

This command is similar to the server invocation in that there are two sets of commands. The first set is ps aux; sockdown 1 1. Since the --out switch was the first switch used, the ps aux sends its standard output over the TCP connection. The sockdown 1 1 closes the socket connection. You can run several commands with different output, as in cmd1; cmd2; cmd3...; sockdown 1 1, but the sockdown command must always come last.

The second client-side command, grep postgres > from_server.txt, takes the output that the server sent from its ps aux, filtering out all mentions of any process that has the word postgres in them, and saves those results to the file from_server.txt.

This example (see the screenshot) works because the client side has performed a shutdown system call on the client's standard output file descriptor, making the socket read-only. Adding another paired transaction of input and output to the commands will fail.

A Multi-Paired Transaction

The single-paired transaction of the last example succeeds in making one paired transaction by closing the write socket on one end; this unfortunately means that you can't have more than one pair of back-and-forth communications. netpipes has another function that permits multi-paired datastreams to create a running dialog between the two socket connections of client and server. This two-way communication uses the Session Control Protocol (SCP). SCP is a standard proposed by Sun Systems a few years back that enables several transactions to run concurrently. netpipes implements SCP by spawning a separate process for each transaction pair. It then uses a utility called encapsulate that attaches them to specific file descriptors.

This screenshot shows a simple question-and-answer session between the server and client:

A client/server Q and A session
Figure 1. A client/server Q&A session.

Here are the server and client scripts:

#!/bin/bash
# This is the server side script, server.sh
# define the question array
Q=("Hello, who are you?"\
"What is your last name?"\
"What city do you live in?")

# carry out the interrogation
for i in 0 1 2
do
encapsulate --fd 3 -so5i4 sh -c \
	"echo ${Q[i]}>&5;exec 5>&-;cat<&4"
done

#!/bin/bash
# This is the client side script, client.sh
#define the answer array
A=("My name is Robert"\
"My last name is Bernier"\
"I live in Ottawa")

# give the answers
for i in 0 1 2
do
encapsulate --fd 3 -si4o5 sh -c "cat<&4;echo ${A[i]}>&5"
done

The two scripts use three file descriptors; file descriptor 4 serves as input, file descriptor 5 is the output, and file descriptor 3 multiplexes the sub-process channels of file descriptors 4 and 5. The multiplexor, file descriptor 3, actually carries the other two file descriptors.

The command exec 5>&- closes file descriptor 5, thus making it possible for the socket to receive standard input from file descriptor 4. By the way, the client and server descriptors don't have to use the same numbers, but it is less confusing that way.

Notice how the switches -si4o5 and -so5i4 have a reversed order between client and server. You must follow this convention or the connection will fail!

Streaming Multimedia Files

Here's an interesting way to stream MP3s, MOVs, and AVIs. All you need to do is to set up the server as such:

faucet 2000 --out sh -c "cat War\ -\ Low\ Rider.mp3" -daemon

and run your client this way:

hose localhost 2000 --in sh -c "cat<&0 | mplayer - -cache 8192"

What's really cool is that you can run as many clients as you want at the same time -- each stream is independent! With a little scripting, you can make even make an interactive session using SCP where the client can choose the multimedia file to play.

Caveats

Running netpipes multiplexed connections is not the most responsive mechanism when it comes to socket connections.

You'll need to understand redirection, file descriptors, and how they work in bash (or your preferred shell) in order to take advantage of the full power of netpipes. bash may not have the most intuitive scripting language, but it certainly is among the most powerful.

Debugging a netpipes script can be an adventure into the unknown, because there are no error messages explaining why a connection fails. Take your time and start by writing small scripts.

Pick one way of doing things. Be consistent with file descriptor naming conventions between the client and server. Use strace and ps to see what the utilities are doing. Reading the netpipes man pages lends meaning to the saying that there's more than one way to skin a cat.

Of course, watch out when reading the man pages. They are confusing and could stand a bit of cleanup.

Want a little more security? The original netpipes package included a utility called ssl-auth. As far as I can tell, this utility no longer exists, but you can no doubt replicate it using something like stunnel.

You don't need to use bash. Perl or any other language that uses file descriptors will work fine, too.

Ideas

What can you do with this utility? One activity that comes to mind is to run a menu-driven utility: one side sends over a text file with choices, and the other side carries out actions based on the choices taken. Wrap this in stunnel to make for a very efficient and secure data exchange system -- perfect for those non-root accounts that need to set up socket connections on the fly and don't have the traditional services available for them. This is a great mechanism for lightweight utilities. You could even incorporate them into emergency boot distributions.

References

Robert Bernier is the PostgreSQL business intelligence analyst for SRA America, a subsidiary of Software Research America (SRA).


Return to ONLamp.com.

Copyright © 2009 O'Reilly Media, Inc.