An issue tangentially related to memory performance is the releasing of shared memory segments on multiprocessor computers. These resources are needed for jobs that execute in parallel across multiple component processors. They are not always cleared properly when a job teminates abnormally and must be removed by hand in this case ( if enough uncleared shared memory segments accumulate, eventually parallel jobs will begin to fail. Constant rebooting of the system for every user that uses these resources and has parallel codes crash may not be an attractive solution for large clusters. The technique makes use of the ipcs and ipcrm commands. Other issues like running out of swap space, or removing paging area will be considered later. Refer to Interprocess Communications for more information on the various of types of interprocess communications (IPC).
The ipcs command (a user command) will list the current system status for message queues, shared memory segments and semaphores:
$ ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
------ Semaphore Arrays --------
key semid owner perms nsems status
------ Message Queues --------
key msqid owner perms used-bytes messages
If you issue this command and there are no conficts this is what you will see.
Here is a script in csh (can also be written in bash - change to csh to run this script (type csh and press return)) that will clear shared memory segments and semaphores. When run by a normal user, it clears his own resources; when run by root, it clears all resources. Note that there is no danger in removing resources currently in use as they will not be deleted until they are released by the corresponding process. The ipcrm command is used to remove the resources.
#!/bin/csh
# clearipc - clear shared memory segments and semaphores
setenv PATH /bin:/usr/bin:/usr/local/bin:/home:/home/test
if ("`whoami`" == "root") then
set us="0x" # "0x" is the hex key value in every line
else
set us="$user"
endif
# The two awk commands are needed because there may or may not be a space
# separating the items in the first two columns if the ipcs output.
foreach thing (`ipcs -m | grep $us | awk -Fm '{print $2}' | awk '{print $1}'`)
ipcrm -m $thing
end
foreach thing (`ipcs -s | grep $us | awk -Fs '{print $2}' | awk '{print
$1}'`)
ipcrm -s $thing
end
The script could be easily extended to clear message queues as well, or to make the kind of item(s) to be cleared as a command line argument. Of course you could directly use the ipcs and ipcrm commands.
Large programming efforts often use separate processes to manage complexity and risks. Sometimes separate processes provide enhanced performance on multiprocessor systems. Client/server processes are separate by their very nature. However, once applications become separate processes, there exists a gulf between them when they need to share data. Thus depending on the processes several types of IPC's may occur. These include:
- Regular files with locking.
- FIFO's (named pipes).
- Anonymous pipes.
- Sockets.
- Signals.
Regular files when used with the appropiate lock techniques, can be used to communicate between processes. FIFO's and anonymous pipes can also be used to from pipelines between separate processes. Sockets allow communications with local and remote processes. Finally processes can notify each other using signals.
The other three forms of IPC are:
- Message Queues.
- Shared Memory.
- Semaphores.
These three forms of IPC establish a new group of facilities because you create and control them in a diifferent manner that the preceeding forms. Except for signals, all preceeding forms used file descriptors to access and to control them. However the above three use different handles.
The Message Queue:
The UNIX message queue implements a priority-based queue of messages. The message is simply a short block of memory holding an application-defined message. When a message is queued, it is stored within the kernel memory so that it can be later retrieved by another process. Figure 1 below shows how messages are queued, stored and retreived. The figure shows three processes queuing messages and one process receiving messages. Message queues in general, however, can be queued by many processes and received by many processes. Every queued message has a message priority called a message type. This is set by using the msgsnd() function and is again used by the msgrcv() function that receives the message. These are UNIX library calls. An example program using this is given in program listing 1 for a client/server. The client issues a request and the server receives the message and responds. This message type determines the priority of the message when it is queued. Figure 2 shows a series of messages from A to J being queued. The number preceeding each message letter indicates the priority of the message. For example, 3C indicates message C was queued at priority 3.
Figure 1: The Message Queue Store within the kernel.
Figure 2: Priority Messages placed in a Message Queue.
The UNIX kernel queues each message into a sub-queue that corresponds to the message priority. If no process is removing messages from the queue, Figure 3 shows how the nine messages would be sorted on the basis of their messge priority. The lowest numbers indicate the highest message priority in messge queues. When the receiving process retrieves a message, it has several choices. These are
- Receive a message of priority x or receive no messages if no messages exist with priority x.
- Receive the lowest numbered (highest priority) message that is less than or equal to priority x.
- Receive the first message on the queue in a first-in, first-out manner, without regard to priority.
While Figure 3 shows that all messages are queued by priority the UNIX kernel also maintains another linked list that allows it to fetch messages on a FIFO basis. In this manner, a process may choose to ignore the priority of messages and simply fetch the earliest message that was queued. Since messages can be retrieved for a specific message priority, it is possible to use the message priority (message type) to address a message to one of several receiving processes. The message priority is a 31-bit value (the sign-bit cannot be used) on a 32-bit platform and bigger for a 64-bit one. Thus 64-bit platforms can accomodate higher number of processes. Consequently some applications have used the message type for the process ID. Each receiving process simply fetches messages that correspond to its process ID. Figure 3 shows an illustration of this. Each process selects its own messages in Figure 3 by using its process ID as the message priority.
Figure 3: Processes reading messages by process ID.
Program Listing 1: Example of message queues using a Client/Server Program.
msq.h - The
Msq Class Definition File
// msq.h
#include
<sys/types.h>
#include
<sys/ipc.h>
#include
<sys/msg.h>
struct Msg
{
long msgtyp; // Message
type/priority
};
class Msq {
enum state {
ready, notReady };
key_t key; // IPC
Key
int msqid; // IPC
ID
int error; // Last
errno
protected:
void _verify(state s);
public:
Msq();
Msq &create(key_t key, int flags);
Msq &access(key_t
key);
Msq &dispose();
Msq &destroy();
msqid_ds &stat(msqid_ds
&stbuf);
Msq &change(msqid_ds
&stbuf);
int send(Msg &msg,size_t
size,int flags=0);
int recv(Msg &msg,size_t
&size,size_t maxsz,long msgtyp,int flags=0);
inline int getError() {
return error; }
inline key_t getKey() {
return key;}
};
// End msq.h
msqveri.cc -
The Implementation of Msq::_verify(),Msq::dispose(), and the Constructor
Msq::Msq()
//
msqveri.cc
#include <stdlib.h>
#include <errno.h>
#include "msq.h"
///////////////////////////////////////////////////////////////////////////////////////////////////////////
// (private) Msq::_verify
// Checks to see that the object is in a
ready or
// not ready state. If the state is not correct
// the error EINVAL is thrown.
///////////////////////////////////////////////////////////////////////////////////////////////////////////
void
Msq::_verify(state s)
{
if (
s == ready && msqid < 0 )
throw error = EINVAL; //
Object is not open
if (
s != ready && msqid >= 0 )
throw
error = EINVAL; // Object is
open!
}
///////////////////////////////////////////////////////////////////////////////////////////////////////////
// Msq::dispose
// Disposes of the current message queue
reference, if
// any. The object is
re-initialized to the not-ready
// state.
///////////////////////////////////////////////////////////////////////////////////////////////////////////
Msq &
Msq::dispose() {
key =
IPC_PRIVATE;
msqid = -1;
return
*this;
}
// Msq::Msq
// Constructor.
This
constructor calls upon the
// method Msq::dispose() to initialize
the object.
///////////////////////////////////////////////////////////////////////////////////////////////////////////
Msq::Msq() {
Msq::dispose(); // Initialize this object
}
// End msqveri.cc
msqcr.cc - The Implementation of the Msq::create() Method
// msqcr.cc
#include
<stdlib.h>
#include
<errno.h>
#include
"msq.h"
///////////////////////////////////////////////////////////////////////////////////////////////////////////
// Msq::create
// ARGUMENTS:
// key
IPC Key of the message queue or IPC-PRIVATE
// flags The permission bits, and
possibly IPC-EXCL
// This method creates a message
queue. Object must be
// in a not-ready state.
///////////////////////////////////////////////////////////////////////////////////////////////////////////
Msq &
Msq::create(key_t key,int flags) {
_verify(notReady);
// Object must not be open
flags |= IPC_CREAT;
// Force a create symantic
// Attempt to create
the message queue
msqid = msgget(this->key =
key,flags);
if ( msqid == -1 )
throw error = errno;
return *this;
}
//End msqcr.cc
msqac.cc - The implementation of the Msq::access()
Method
// msqac.cc
#include
<stdlib.h>
#include <errno.h>
#include "msq.h"
///////////////////////////////////////////////////////////////////////////////////////////////////////////
//Msq::access
// ARGUMENTS:
// key IPC Key of the message queue or
IPC-PRIVATE 12:
// This method accesses a message
queue. Object must be
// in a not-ready state.
///////////////////////////////////////////////////////////////////////////////////////////////////////////
Msq &
Msq::access(key_t key) {
_verify(notReady); // Object must not be open 21:
//Attempt to create the message queue
24:
msqid =
msgget(this->key = key,0);
if (
msqid == -1 )
throw error =
errno;
return *this;
}
// End msqac.cc
msqdest.cc-The Implementation of the Msq: :destroy() Method
// msqdest.cc
#include <stdlib.h>
#include <errno.h>
#include "msq.h"
/////////////////////////////////////////////////////////////
// Msq::destroy
// Destroys the message queue. The object must be in a
// ready state. The object is placed into a not-ready
// state upon successful completion.
/////////////////////////////////////////////////////////////
Msq &
Msq::destroy() {
_verify(ready); // Object must be open
if ( msgctl(msqid,IPC_RMID,0) == -1
throw error = errno;
Msq::dispose(); // Re-initialize this object
return *this; // Return in not-ready state
}
// End msqdest.cc
msqstat. cc-The Implementation of the Msq::stat() Method
// msqstat.cc
#include <stdlib.h>
#include <errno.h>
#include "msq.h"
/////////////////////////////////////////////////////////////
// Msq::stat
// ARGUMENTS
// stbuff The struct msqid-ds structure to populate
// with message queue information.
// This method fills the supplied buffer with status
// information about the current queue. The object must
// be in the ready state.
/////////////////////////////////////////////////////////////
msqid_ds &
Msq::stat(msqid_ds &stbuf) {
_verify(ready); // Object must be open
if ( msgctl(msqid,IPC_STAT,&stbuf) == -1)
throw error = errno;
return stbuf;
// End msqstat.cc
msqchg. c-The Implementation of the Msq: :change() Method
// msqchg.cc
#include <stdlib.h>
#include <errno.h>
#include "msq.h"
/////////////////////////////////////////////////////////////
// Msq::change
// ARGUMENTS
// stbuff The struct msqid_ds structure containing
// the changes to be made.
// Only the values msg_perm.uid, msg_perm.gid, msg_perm.mode
// and msg_qbytes values can be changed. The value
// msg_qybytes can only be increased by the superuser.
// Object must be in the ready state.
/////////////////////////////////////////////////////////////
Msq &
Msq::change(msqid_ds &stbuf) {
_verify(ready); // Object must be open 24:
if ( msgctl(msqid,IPC_SET,&stbuff) == -1 )
throw error = errno;
return *this;
}
// End msqchg.cc
msgsend. cc-The Implementation of the Msq: :send() Method
// msqsend.cc
#include <stdlib.h>
#include <errno.h>
#include "msq.h"
/////////////////////////////////////////////////////////////
// Msq::send
// ARGUMENTS
// msg The message to be sent
// size The total size of the message
// flags zero or IPC_NOWAIT (optional)
// RETURNS:
// 0 No message sent (with IPC_NOWAIT)
// 1 Message was sent
// Sends a message of size bytes to the message queue.
// The size must include the total size of the message,
// including the message type. The object must be in a
// ready state.
/////////////////////////////////////////////////////////////
int
Msq::send(Msg &msg,size_t size,int flags)
int z;
size_t msgsz = size - sizeof msg.msgtyp;
_verify(ready);
do {
z = msgsnd(msqid,&msg,msgsz,flags);
} while ( z == -1 && errno == EINTR);
if ( z ) {
if ( flags & IPC_NOWAIT && errno == EAGAIN )
return 0; // Not sent
// Other fatal error:
throw error = errno;
}
return 1; // Succeeded
}
// End msqsend.cc
msq recv. cc-The Implementation of the Msq::recv() Method
// msqrecv.c
#include <stdlib.h>
#include <errno.h>
#include "msq.h"
/////////////////////////////////////////////////////////////
// Msq::recv
// ARGUMENTS
// msg The receiving buffer for the message
// size The returned size of the message
// maxsz The maximum size of the returned message
// msgtyp The message type to use (priority)
// flags Flags IPC_NOWAIT, IPC_EXCEPT and
// IPC_NOERROR (optional)
//RETURNS
// 0 No message returned (with IPC_NOWAIT)
// 1 Message was returned
// This method receives a message from the message queue.
// Object must be in a ready state.
/////////////////////////////////////////////////////////////
int
Msq::recv(Msg &msg,size t &size,size t maxsz,long msgtyp,int flags)
int z;
size_t msgsz = maxsz - sizeof msg.msgtyp;
_verify(ready);
do {
z = msgrcv(msqid,&msg,msgsz,msgtyp,flags);
} while ( z==-1 && errno == EINTR );
if ( z==-1 ) {
if ( flags & IPC_NOWAIT && errno==EAGAIN )
return 0; // No message read
throw error = errno; // Error occurred
}
size = z + sizeof msg.msgtyp; // Return size
return 1; // Successful
}
// End msqrecv.cc
statmsg. h-The Declaration of the StatMsg Message Structure
// statmsg.h
struct StatMsg : Msg {
enum {
stat, // stat a pathname
lstat, // lstat a pathname
stop // stop the server
} request; // Request type
int error; // zero if successful
pid_t PID; // Requesting Process ID
union {
char path[256]; // Pathname to stat
struct stat stbuf; // stat(2) or lstat(2) info
} u; // union
}
// End statmsg.h
statsrv. cc - The statsrv Server Listing
// statsrv.cc
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include "msq.h"
#include "statmsg.h"
int
main(int argc,char **argv) {
int quit = 0; // True when stop received
Msq q; // Message queue object
StatMsg m; // Message buffer
size_t msz; // Message size
char pathname[256+1]; // Local copy of pathname
msqid_ds mstat; // Message queue info
(void) argc;
(void) argv;
// Create the server message queue
try {
q.reate(OxFEEDFOOD,0600);
} catch ( int e ) {
errno=e;
perror("Creating a queue");
}
// Obtain queue information
try {
q.stat(mstat);
} catch ( int e ) {
errno = e;
perror("q.stat()");
}
printf("Queue permissions were: %@4o\n",mstat.msg_perm.mode);
// For demonstration purposes,
// make the queue read & writable to all
mstat.msg_perm.mode = 0666;
try {
q.change(mstat);
} catch ( int e ) {
errno = e;
perror("q.change()");
}
printf("Queue permissions now : %04o\n",mstat.msg_perm.mode);
// Server message loop
do {
// Receive a message of type 1
try {
q.recv(m,msz,sizeof m,1,0);
} catch ( int e ) {
errno = e;
perror("Receiving from queue");
return 1;
}
// Process message
switch ( (int) m.request ) {
case StatMsg::stat : // stat(2) request
strncpy(pathname,m.u.path,sizeof pathname);
pathname[sizeof pathname-1] = 0;
m.error = stat(pathname,&m.u.stbuf) ? errno : 0;
break;
case StatMsg::lstat : // lstat(2) request
strncpy(pathname,m.u.path,sizeof pathname);
pathname[sizeof pathname-1] = 0;
m.error = lstat(pathname,&m.u.stbuf) ? errno : 0;
break;
case StatMsg::stop : // stop server
quit = 1; // Stop the server
m.error = 0; // Ack request
break;
default : // Unknown request
m.error = EINVAL;
}
// Reply to client
m.msgtyp = m.PID; // Reply to this process
try {
q.send(m,sizeof m);
} catch ( int e ) {
errno = e;
perror("q.send()");
return 1;
}
} while ( !quit );
// Destroy the message queue
q.destroy();
return 0;
// End statsrv.cc
statcln.cc-The Source Listing for the statcln Client Program
// statcln.cc
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include "msq.h"
#include "statmsg.h"
int
main(int argc,char **argv) {
int x;
Msq q; // Message queue object
StatMsg m; // Message buffer
size_t msz; // Message size
char *pathname; // Pathname to query
(void) argc;
(void) argv;
// Access the queue
try {
q.access(OxFEEDFOOD);
} catch ( int e ) {
errno = e;
perror("Accessing statsrv queue");
}
// Issue server requests for each command line
// argument. If the argument starts with then
// request a lstat(2) instead of stat(2)
for ( x=1; x<argc; ++x ) {
// Form the server request
if ( 1strcasecmp(argv[x],"STOP") )
// STOP SERVER REQUEST :
m.request = StatMsg::stop;
else {
// STAT(2) or LSTAT(2) REQUEST
if ( argv[x][0] == '$' ) {
m.request = StatMsg::lstat;
pathname = argv[x] + 1; // Skip
else {
m.request = StatMsg::stat;
pathname = argv[x]; // Pathname
}
strncpy(m.u.path,pathname,sizeof m.u.path);
}
// Initialize other message components
m.error = 0; // Clear
m.PID = getpid(); // Our process ID
m,msgtyp = 1; // Send to the server
// Send the request to the server
try {
q.send(m,sizeof m); // Send the message
} catch ( int e ) {
errno = e;
perror("s.send()");
return 1; // Bail out
}
// If the request is to stop, then exit loop
if ( m.request == StatMsg::stop )
break; // There will be no reply
// Wait for the response
try {
q.recv(m,msz,sizeof m,getpid(),0);
} catch ( int e ) {
errno = e;
perror("Receiving from queue");
return 1;
}
// Report response
printf("RESPONSE %14s : ",pathname);
if ( m.error != 0 )
printf(" ERROR: %s\n",strerror(m.error));
else
printf(" SIZE: %ld bytes\n",(long)m.u.stbuf.st_size);
}
// Exit client program
q.dispose(); // Reset object
return 0;
}
// End statcln.cc
To make the files do the following:
$ make
CC -c -Wall
-fhandle -exceptions msqveri.cc
CC -c -Wall -fhandle -exceptions msqcr.cc
CC -c -Wall -fhandle -exceptions msqac.cc
CC -c -Wall -fhandle -exceptions msqdest.cc
CC -c -Wall -fhandle -exceptions msqstat.cc
CC -c -Wall -fhandle -exceptions msqchg.cc
CC -c -Wall -fhandle -exceptions msqsend.cc
CC -c -Wall -fhandle -exceptions msqrecv.cc
ar r libmsq.a msqveri.o msqcr.o msqac.o msqdest.o msqstat.o msqchg.o msqsend.o
msqrecv.o
CC -c -Wall -fhandle -exceptions statserv.cc
CC -o statsrv statsrv.o -L. -lmsq -lstdc++
CC
-c -Wall -fhandle -exceptions statcln.cc
CC -o statcln statcln.o -L. -lmsq -lstdc++
$
Once the executibles are
prepared, start up the server program as follows:
$ ./statsrv &
$
Queue permissions were: 0600
Queue permissions now: 0666
The misplaced $ character is due to the shell
issuing a prompt to the user before the server program wrote its output to the
terminal. The server displays before (0600)
and after (0666) set of permission
bits.
With the server ready for requests, you can now
issue requests on the ./statcln command line:
$ ./statcln
& /etc/hosts STOP
RESPONSE
/etc/hosts :
SIZE: 279 bytes
[1] 12543 Exit 0
In this example, the first argument /etc/hosts
requested the system library call stat() of
the hosts file from the server. The response from the server showed that the
file's size was 279 bytes. This can be verified by the ls -l /etc/hosts command.
The STOP argument caused the program ./statcln to request the server shut down,
which it did.
Shared Memory
When multiple processes cooperate, they often
need to share tables of data. UNIX provides this in the form shared memory
facility. Figure 4 shows how one shared memory region can be shared by three
processes. Although the concept of sharing memory is a simple one, a number of
complications can occur. For example in Figure 4 the shared region may be
attached to each process' memory space at a different memory address. This means
that if memory addresses are used within the shared table, they will not be
usable in all processes. Memory offsets must be used instead. This is the reason
that shared libraries must be compiled to use position-independent code. Another
complication is the problem of synchronization between three processes. If
multiple processes are changing areas of shared memory region, how can a given
process know that a particular component of data is complete? Even the process
of replacing an integer value is not atomic on many CPU platforms. Thus for
synchronization it is better to use semaphores. Program listing 2 gives a
program that illustrates the use of shared memory. It is in a modular form and
is designed to allow shell programs like csh or bash to share global variables
using shared memory. This is the key in MPI languages. Here global
variables like MPI_INT or MPI_LONG etc are constantly needed. Note that this
differs from exported shell environment variables which cannot be altered by
child processes. In this case any process using this program can inquire or
alter the value of a global variable.
Figure 4: A memory region shared with three processes.
Program Listing 2: Source files for the glovar utility program.
globcr. c - The globvar Source Module That Calls shmget () to Create Shared Memory
/* globcr.c */
#include "globvar.h"
/* Create a new shared memory variable pool */
void
create_vars(int shm_size) {
int z; /* Status code */
int semid; /* Semaphore IPC ID
int offset; /* Byte offset */
union semun un; /* Union of semctl() args */
struct shmid_ds shminfo; /* Shared memory info */:
/* Create shared memory region */
z=shmget(IPC_PRIVATE,shm_size,IPC_CREAT|0600);
if ( z == -1 ) {
fprintf(stderr,"%s:
shmmget(,%d,IPC_CREAT)\n",strerror(errno),shm_size);
exit(l);
}
shmid = z;
/* IPC ID
*/
/* Create semaphore for this
region */
z=semget(IPC_PRIVATE,1,IPC_CREAT|0600);
if (z == -1) {
fprintf(stderr,"%s:
semget(,IPC_CREAT)\n",strerror(errno));
exit(l);
}
semid = z;
/* IPC ID */
/* Discover the actual size of the
region */
z=shmctl(shmid,IPC_STAT,&shminfo);
if (z == -1) {
fprintf(stderr,"%s: shmctl(%d,IPC_STAT)\n",strerror(errno),shmid);
exit(l);
}
shm_size = shminfo.shm_segsz; /* Actual size of the memory
region
*/
/* Initialize binary semaphore to
value of 1 */
un.val = 1;
z=semctl(semid,0,SETVAL,un.val);
if (z == -1) {
fprintf(stderr,"%s:
semctl(%d,0,SETVAL)\n",strerror(errno),semid);
exit(l);
}
/* Attach shared memory, and
initialize it */
attach_vars();
/* Attach shared memory */
globvars->semid = semid; /* Place semaphore ID into shared memory
*/
offset = (int)
( &globvars->vars[0] - (char *)globvars
);
globvars->size = shm_size -
offset;
globvars->vars[0] =
globvars->vars[1] = 0;
}
globat. c - The Source Module That Calls shmat () to Attach Shared Memory
/* globat.c */
#include "globvar.h"
/* Attach the shared variable pool */
void
attach_vars(void)
/* Attach shared memory region */
globvars = (GlobVars *)shmat(shmid,0,0);
if ( (void *)(globvars) == (void *)(-l)) {
fprintf(stderr,"%s: shmat(%d,0,0)\n",strerror(errno),shmid);
exit(l);
}
}
globdest. c - The Source Module That Calls shmdt () and Destroys the Shared Memory
/* globdest.c */
#include "globvar.h"
/* Destroy the shared memory variable pool */
void
destroy_vars(void) {
int z; /* Status code */
int semid; /* Semaphore IPC ID */
union semun un; /* Union of semctl() args */
/* Lock the shared memory region */:
glob_lock();
semid = globvars->semid; /* Semaphore IPC ID */
/* Destroy locking semaphore */
z=semctl(semid,0,IPG_RMID,un);
if (z == -1) {
fprintf(stderr,"%s: semctl(%d,O,IPC_RMID)\n",strerror(errno),semid);
exit(l);
}
/* Detach shared memory */
z=shmdt(globvars);
if (z == -1) {
fprintf(stderr,"%s: shmdt(2)\n",strerror(errno));
exit(l);
}
/* Destroy shared memory */
z=shmct1(shmid,IPC_RMID,NULL);
if (z == -1) {
fprintf(stderr,"%s: shmct1(%d,IPC_RMID)\n",strerror(errno),shmid);
exit(l);
}
}
globvar. h - The Global globvar Utility Definitions
/* globvar.h */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/sem.h>
#define GLOBVARENV "GLOBVAR" /* Environment variable*/
typedef struct {
int semid; /* Semaphore's IPC ID */
int size; /* Size of the vars[] array */
char vars[l]; /* Start of variable storage */
} GlobVars;
extern int shmid; /* IPC ID of shared memory region */
extern int shm_size; /* Size of shared memory region */
extern GlobVars *globvars; /* Shared memory region */
extern int semid; /* IPC ID of the locking semaphore set */
extern void create_vars(int shm_size);
extern void attach_vars(void);
extern char *get_var(const char *varname);
extern void set_var(const char *varname,const char *value);
extern void destroy_vars(void);
extern void glob_lock(void);
extern void glob-unlock(void);
extern void unset_var(const char *varname);
#ifndef HAVE_SEMUN /* Does sys/sem.h define this? */
union semun {
int val; /* Value */
struct semid_ds *buf; /* IPC_STAT info */
u_short *array; /* Array of values */
};
#endif
/* End globvar.h */
globlk. c - The Semaphore Locking Routines
/* globlk.c */
#include "globvar.h"
static struct sembuf wait { 0, -1, SEM_UNDO };
static struct sembuf notify { 0, +1, SEM_UNDO };
/* Perform a semaphore operation */
static void
do_semop(struct sembuf *op) {
int z; /* status code */
do {
z = semop(globvars->semid,op,l);
} while (z == -1 && errno == EINTR);
if ( z ) {
fprintf(stderr,"%s: semop(2)\n",strerror(errno));
exit(l);
}
}
/* Wait on semaphore to lock shared memory */
void
glob_lock(void) {
do_semop(&wait);
}
/* Notify semaphore to unlock shared memory */
void
glob_unlock(void) {
do_semop(¬ify);
}
globget. c - The Source Module That Looks Up a Global Variable in Shared Memory
/* globget.c */
#include "globvar.h"
/* Return the string pointer for a variable's value */
char *
get_var(const char *varname) {
char *cp; /* Scanning pointer */
int nlen = strlen(varname); /* Length of variable name */
for ( cp = &g1obvars->vars[0]; *cp; cp += strlen(cp) + 1 )
if ( !strncmp(varname,cp,nlen) && cp[nlen] == '=' )
return cp + n1en + 1; /* Pointer to it's value */
return NULL; /* Variable not found */
}
globvar.c - The Main Program for the globvar Utility
/* globvar.c */
#include "globvar.h"
int shmid = -1; /* IPC ID of shared memory */
GlobVars *globvars = NULL; /* Shared memory region */
/* Usage instructions */
static void
usage(void) {
puts("globvar [-i] [-s size] [-e] [-ul [-r] [-c]" " var ... var=value...");
puts("Options:");
puts(" -i Initialize new globvar pool");
puts(" -s size Size of this pool, in bytes");
puts(" -e Dump all values (after changes)");
puts(" -u Unset all named variables");
puts(" -r Remove this pool of values");
puts(" -c Clear all variables");
puts(" -h This info.");
/* Main program */
int
main(int arge,char **argv) {
int rc = 0; /* Return code */
int optch; /* Option character */
int cmdopt_i = 0; /* -i to create var pool */
int cmdopt_c = 0; /* -c to clear variables */
int cmdopt_r = 0; /* -r to remove pool */
int cmdopt_e = 0; /* -D to dump the variables */
int cmdopt_u = 0; /* -u to unset named variables */
int cmdopt_h = 0; /* -h usage option */
int cmdopt_s = 4096; /* Default for -s */
char *cp, *ep; /* Character pointers */
unsigned long ul; /* Converted ulong */
const char cmdopts[] "hirs:ecu";
/* Parse command line options */
while ( (optch = getopt(argc,argv,cmdopts)) != -1 )
switch ( optch ) {
case 'c' : /* -c to clear variables */
cmdopt_c = 1;
break;
case 'i' : /* -i initialize a new pool */
cmdopt_i = 1;
break;
case 'e' : /* -e to dump all variables like env */
cmdopt_e = 1;
break;
case 'r' : /* -r to remove the pool */
cmdopt_r = 1;
break;
case 's' : /* -s size; affects -i */
ul = strtoul(optarg,&ep,0);
if ( *ep ) {
fprintf(stderr,"Bad size: -s %s\n",optarg);
rc = 1;
} else {
cmdopt_s = (int) ul;
break;
case 'u' : /* -u to unset all listed variables */
cmdopt_u = 1;
break;
case 'h' : /* -h to request help */
cmdopt_h = 1;
break;
default :
rc = 1;
}
/* Give usage display if errors or -h : */
if ( cmdopt_h | | rc ) {
usage();
if ( rc )
return rc,
}
/* Create/Access global variable pool */
if ( cmdopt_i ) {
/* Create a new shared memory variable pool */
create_vars(cmdopt_s);
printf("%d\n",shmid);
} else if ( (cp = getenv(GLOBVARENV)) != NULL ) {
/* Extract IPC key from GLOBVAR environment variable */
ul = strtoul(cp,&ep,0);
if (*ep ) {
fprintf(stderr,"%s has bad IPC key\n",cp);
return 1;
}
shmid = (int)ul;
attach_vars();
}
/* Do we have enough information to find the pool? */
if ( !globvars ) {
fprintf(stderr,"You must use -i or define"
" environment variable %s.\n",GL0BVARENV);
return 1;
/* -c clears all variables */
if ( cmdopt_c ) {
glob_lock();
globvars->vars[0] = globvars-vars[1] = 0;
glob_unlock();
}
/* Now process variable requests */
for (; optind < argc; ++optind ) {
cp = strchr(argv[optind],'=');
glob_lock();
if ( !cp ) {
/* Just have a variable name, so return value or unset */
if ( !cmdopt_u ) {
if ( (cp = get_var(argv[optind])) != NULL ) {
puts(cp); /* Just emit value of variable */
} else {
fprintf(stderr,"Variable %s not found\n",argv[optind]);
rc = 1;
}
} else {
unset_var(argv[optind]);
} else {
/* Change the variable's value */
*cp = 0;
set_var(argv[optind],++cp);
}
glob_unlock();
}
/* Dump all variables (for debugging) */
if ( cmdopt_e ) {
glob lock();
for ( cp=&globvars->vars[O]; *cp; cp+=strlen(cp)+l )
puts(cp);
glob_unlock();
}
/* If -r option, destroy the global variable pool */
if ( cmdopt_r )
destroy_vars();
return rc;
}
globset.c - The Implementation of the globvar Variable Assignment Functions
/* globset.c */
#include "globvar.h"
/* Change the value of a global variable */
void
set_var(const char *varname,const char *value) {
int z; /* status code */
char *var = get_var(varname); /* Locate variable if it exists */
char *cp; /* utility char pointer */
int nlen = strlen(varname); /* Length of variable name */
int vlen = strlen(value); /* Length of variable's value */
int in_use = 0; /* Bytes in use */
int avail = globvars->size; /* Bytes available */
if ( var ) { /* Does variable exist? */
in_use = (int)( var - &globvars->vars[0] ) + 1;
avail -= in_use; /* Bytes available for new value */
z=strlen(var + nlen + 1); /* Length of current string */
if ( vlen > avail + z )
goto nospc; /* Insufficient space */
/* Now delete the variable */
var = var - nlen - 1; /* Point to start of entry */
for ( cp=var+strlen(var)+l; *cp; var += z, cp += z ) {
z = strlen(cp) + 1; /* Length of next value */
memmove(var,cp,z); /* Move it up */
}
}else {
/* Find end of global storage */
for ( var = &globvars->vars[01; *var; var += strlen(var) + 1 )
;
in_use = (int)( var - &globvars->vars[0] ) + 1;
avail -= in_use; /* Bytes available for new value */
if ( nlen + 1 + vlen > avail )
goto nospc;
}
/* Append VARIABLE=VALUE\0 to end of shared region */
strcpy(var,varname); /* Variable name */
var += nlen; /* Point past variable name */
*var++ = '='; /* The equal sign */
strcpy(var,value); /* The variable's value */
var[vlen+l] = 0; /* 2 null bytes mark the end */
return; /* Successful */
/* Insufficient space to store this variable */
nospc:
fprintf(stderr,"%s: %s='%s'\n",strerror(ENOSPC),varname,value);
exit(l);
}
globun.c - The Unset Feature of globvarls implemented by globun.c
/* globun.c */
#include "globvar.h"
/* Unset a variable */
void
unset_var(const char *varname) {
int z; /* status code */
char *var = get_var(varname); /* Locate variable if it exists */
char *cp; /* utility char pointer */
int nlen strlen(varname); /* Length of variable name */
if ( !var )
return; /* Variable is already unset */
/* Now delete the variable */
var = var - nlen - 1; /* Point to start of entry */
for ( cp=var+strlen(var)+l; *cp; var - z, cp - z ) {
z = strlen(cp) + 1; /* Length of next value */
memmove(var,cp,z); /* Move it up */
}
*var = 0; /* two nulls mark the end of vars */
return; /* Successful */
}
To make the files do the following:
$ make
CC -c -Wall
-DHAVE_SEMUN -g globat.c
CC -c -Wall -DHAVE_SEMUN -g globcr.c
CC -c -Wall -DHAVE_SEMUN -g globget.c
CC -c -Wall -DHAVE_SEMUN -g globlk.c
CC -c -Wall -DHAVE_SEMUN -g globset.c
CC -c -Wall -DHAVE_SEMUN -g globvar.c
CC -c -Wall -DHAVE_SEMUN -g globdest.c
CC -c -Wall -DHAVE_SEMUN -g globun.c
CC -o globvar globat.o globcr.o globget.o globlk.o globset.o globvar.o globdest.o globun.o
$
The usage jinformation is available with the -h option.
$ ./globvar -h
To use the globvar utility program create the global memory pool. This is done using the -i option ( the option -s can be used to change the default memory segment size of 4KB). The use of the -i option causes a private shared memory region to be created and displays the IPC ID on standard output. The ipcs command can now be used to confirm that a shared memory region and a semaphore were created. Private sharted memory regions eliminate any possibility of IPC key clashes. It will require, however, that you pass the IPC ID of you global variable pool to those other shell programs that need access to it. Notice also that when the ipcs command is issued the output indicates that the permissions established are such that only the owner of the shared memory has access to it. This keeps the values of your global pool safe from other users.
A global variable pool can be destroyed using the -r option. The GLOBVAR environment variable names the IPC ID of the global memory pool that you are working with. So if from the ipcs command you see that you have an ID of 12345 say then issue the following to destroy the pool.
$ GLOBVAR=12345 ./globvar -r
Normally the initialization of a global memory pool is performed so that the IPC ID is recorded in the exported shell environment variable GLOBVAR as follows:
$ GLOBVAR=`./globar -i`
$ export GLOBVAR
Establishing the IPC ID in the exported shell variable GLOBVAR allows all future globvar command accesses to contact the correct instance of the global memory pool, which was just created. Changes to the GLOBVAR environment variable will permit you to work with different collections of global variables if you need to.
Once the shell variable GLOBVAR is initialized global variables can be added to the the global memory pool as follows:
$ ./globvar VAR1=XYZ VAR2=123
$
The values contained in the global memory pool can be individually fetched or dumped in bulk with the -e option.
$ ./globvar VAR1
XYZ
$ ./globvar VAR2
123
$ ./globvar -e
VAR1=XYZ
VAR2=123
$
To copy global variables to shell variables you can use the usual shell syntax for this. Global variables can be naturally altered by the utility:
$ ./globvar VAR2="A different value"
$ ./globvar -e
$ VAR1=XYZ
$ VAR2=A different value
$
The -u option "unsets" each global variable named similar to the shell built-in command unset. The -c option clears the global memory pool so that no variables remain.
Another feature in UNIX kernels allows
unrelated process to share information by permitting memory to be mapped to a
regular file or a character device. All executible files in the UNIX
kernels are mapped to virtual memory pages. These pages of memory are marked as
being executable only within the process memory (on many platforms, this often
implies that they are readable as well). In this manner only those memory
pages needed are actually paged into memory upon demand. For large programs,
this is more efficient than loading the entire program into memory at startup.
Memory mapping simply extends this idea to application data files. Figure 5
shows how a memory-mapped file might be accessed from within a process's memory
space. This shows that the mapping may be larger that the actual file itself.
This is often true because the virtual memory management performed by the UNIX
kernel must use a fixed page size. Thus there is an extra region above the files
mapping. These extra bytes will be zeroed when the mapping is established. When
you application examines memory within the mapped region, pages of data are
retrieved from the file as necessary to make the memory cells available.
Likewise if memory cells are modified, the changes are written back out to the
file (depending upon options selected) at a time determined by the kernel. There
are methods to control this behaviour and its timing.
Figure 5: A file is mapped to process memory.
Memory functions performed by the UNIX kernel
are restricted to operating in multiples of the virtual memory (VM) page size.
The library function getpagesize() returns
this information. A memory mapping is established with the help of a system
function mmap(). When mmap() is successful the starting address for
the mapping of at least len bytres is
returned. Otherwise the value MAP_FAILED is returned instead, with the error
code deposited into erro. A memory mapped
region often requires its attributes to be queried or changed in some fashion.
Four system function calls are designed for this purpose.
- mprotect() Change the access of the indicated memory pages.
- madvise() Advise the UNIX kernel how you intend to use your memory
again.
- mincore() Determine if pages of mapped memory are currently in
memory.
- msync() Where modifications exist, indicate what regions of
memory should be updated to the mapped files.
The memory mapped regions are usually
automatically unmapped by the kernel when the system library call execve() is issued or when the process terminates.
It may occur in an application, however, that the memory-mapped file is needed
only temporarily. The system library call munmap() can be used to unmap it.
Semaphores
A UNIX semaphore keeps track of a count and
notifies the interested process when the count changes. The simplest semaphore
is the binary semaphore, which can only hold the count 0 or 1. A mutex is a
simple form of a binary semaphore, which is used when programming with threads.
Other semaphores allow you to track instances of a particular resource. For
example, if you have three transaction servers available to serve client
processes, you would initialize the semaphore count to 3. As clients
attach to and reserve a transaction server you would decrement this
count. When the count reaches zero, the semaphore indicates that no remaining
resources exist at this time. Later, when a client finishes with a transaction
server it increments the semaphore count. When all clients complete the count
increments to the intial count 3. In this manner the semaphore or data structure
keeps track of the number of available resources. The act of decrementing a
semaphore count is known as waiting on the semaphore as the requestor
must wait for the resource when the count reaches zero. The
act of incrementing a semaphore is known as notifying the semaphore. Individual
semaphores work well for controlling individual resources. However obtaining
several resources at once is often reqluired. Imagine a small bowling alley that
has 50 pairs of bowling shoes, 30 bowling balls and 6 bowling alleys available.
To bowl, a person needs one pair of shoes, one bowling ball, and an alley.
However, a patron cannot bowl if any of the resources (shoes, ball, alley) are
busy. A semaphore set allows the caller to request all of the resources at once.
In this way there is no potential for deadlocks, since the request either
completely succeeds of it fails (waits). Figure 6 illustrates a semaphore set
just discussed. Within the set semaphore 0 controls the resource -bowling ball,
semaphore 1 controls the resource -bowling shoes, and semaphore 2 controls the
resource -bowling alleys. It is not necessary to request all the resources in a
semaphore set. A patron may choose to bring his own shoes or bowling
ball. A group of patrons usually shares a bowling alley, and so the total
number of resources would be requested. The benefit of grouping these resources
into one set is that the caller can obtain all resources needed in one system
call without worrying about deadlock situations. If any of the requested
resources are not available, the caller simply waits until all resources become
available.
Figure 6: A semaphore set.
The UNIX kernel provides IPC ID values for
processes to refer to specific instances of message queues, shared memory and
semaphore sets. The IPC ID is an integer value that is determined by the kernel,
and is not known by the calling process until it has been returned in a create
call. The IPC ID is similar to a file descriptor for a specific IPC resource. It
can be positive or zero but never negative. Although the IPC ID value is a
convenient handle for resources once they are created, they are not well suited
for a prearranged rendezvous. If three different processes must attach to a
shared memory region, how do the two processes that did not create the shared
region find out what the IPC ID of that resource is? To solve this difficulty,
the UNIX kernel also provides facilities for working with IPC key values.
The IPC key value is defined by the C data type
key_t. This permits a system-wide 32-bit
key value to be specified. Although files use a hierarchical file system IPC key
values are not hierarchical. The 32-bit key applies to the host system on a
system-side basis. The IPC key is like a filename, whereas the IPC ID is like an
open file descriptor. To avoid conflicts use the ipcs command to get a list
of the keys that are in use. Once you have chosen an IPC key value it is
possible for a process to gain access to a message queue, shared memory region,
or semaphore set by specifying it. As long as all your processes agree on this
key in advance, they will locate the common IPC resource. Once the access is
granted, the kernel returns the IPC ID value that is used for that resource. The
IPC key is only required for the initial rendevouz.
You create IPC resources with system calls
named after teh type of resource. The library function msgget() creates a message queue, while the
library function shmget() and semget() create shared memory regions and
semaphores respectively. The msgget()
function is the simplest of these and further details can be got from the UNIX
documentation. Similary processes that not create the shared memory resource
must look it up to discover the IPC ID. This can be performed using the same
system call that is used for creation. Once the IPC ID value is known the IPC
key is not longer required for the access to the resource. The resource can now
be accessed directly. Unlike files that need to opened the resource can be
accesses immediately. The one exception to this rule is that shared memory must
be attached to your process memory space before it can be referenced using the
system call shmat().
IPC resources can outlive your process. When a
process terminates for any reason all files are closed and its shared memory is
detached, but its IPC resources will continue to exist. If IPC resources are no
longer required, they must be explicitly destroyed. There a system calls to
perform this function:
- msgctl() a library function for message queues
- semctl() a library function for semaphores
- shmctl() a library function for shared memory
When a message queue or a semaphore is
destroyed, they are destroyed immediately. Since IPC resources are not opened
like files they do not stay open until closed. When a message queue or semaphore
set is destroyed the UNIX kernel immediately discards them. If a message queue
or semaphore operation is subsequently attempted on the destroyed IPC ID, the
error EIDRM is returned.
Shared memory is handled differently. When
shared memory is used, it must be attached to the current process at a specific
address. When the shared region is no longer required, or the process
terminates, the shared region is detached from the current process. Due to this
behaviour when a process destroys shared memory the shared memory region exists
until the last process detaches it.