ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Controlling Resources with Solaris Projects

by Gian-Paolo D. Musumeci, author of System Performance Tuning, 2nd Edition

A Case Study in Performance Analysis

Alighieri Financial Services, a company that helps expatriates manage their finances, is in something of a mess. Its complex financial-modeling workloads have to share resources with internal applications such as payroll and account management, and these conflicts are starting to cause severe performance problems--nothing is getting done on time, and their storage keeps filling up. The company needs some way to effectively control resource availability.

Fortunately, Alighieri just upgraded its server, a Sun Ultra Enterprise 4500, to Solaris 9. This upgrade makes our task somewhat straightforward because a powerful new way to manage processes has been introduced--projects. A project is a grouping of processes that are subject to a set of constraints. For example, we could limit a project to how many processes it could have, or how large of a file it could create.

Before we get into fixing Alighieri's problems, let's briefly cover the background. The key file to keep track of here is the project database, /etc/project.

The project file contains a series of one-line entries, one for each configured project. (If you don't want to edit the /etc/project file by hand, you can use the projadd, projdel, and projmod commands to add, remove, or modify projects, respectively.)

Each line takes the following form:


     project-name:project-id:comment:user-list:group-list:attributes
task.max-cpu-time The maximum amount of CPU time available to all the processes in this project (in milliseconds).
task.max-lwps The maximum number of lightweight processes available to all the processes in this project.
process.max-cpu-time The maximum amount of CPU time available per process for each process in this project (in milliseconds).
process.max-file-descriptor The maximum number of file descriptors permitted to each process in this project.
process.max-file-size The maximum file size available to processes in this project (in bytes).
process.max-core-size The maximum size for a core file created by a process in this project (in bytes).
process.max-data-size The maximum heap size for a process in this project (in bytes).
process.max-stack-size The maximum stack size for a process in this project (in bytes).
process.max-address-space The maximum amount of address space (over all segments) available for a process in this project (in bytes).

(There are other resource controls available--these are just the ones that are most immediately interesting.)

Each attribute is specified by setting it equal to a triplet, like this:


     attribute=(privlevel,value,action)

There are three available values for privlevel. The first, "basic", means that the threshold can be modified by the process, without any special privileges; the second, "privileged", means that the threshold can be changed only by the superuser; the third, "system", means that the threshold is fixed for the duration of the operating system instance (for example, until a reboot).

Multiple attributes can be specified for a single project; separate them with semicolons.

Now that we've walked through the theory, let's create a project.

We can put the following line in /etc/project:


     test:10000:test
project:jqpublic::process.max-file-size=(privileged,16777216,deny)

Alternatively, we could run projadd to create this project:


     purgatorio# projadd -p 10000 -c 'test project' -U jqpublic test
     purgatorio# grep ^test: /etc/project
     test:10000:test project:jqpublic::

(Note that projadd and projmod won't let you change the resource-control field.)

This project is named "test" and has a project ID of "10000". Membership is restricted to the user "jqpublic," and any project running in the test project will not be allowed to create a file greater than 16MB (16MB = 16,777,216 bytes).

When jqpublic next logs in, they can run `projects` to get a list of all the projects they are allowed to participate in:


     jqpublic@purgatorio% projects
     default group.staff test

To find out which project is associated with the currently running shell, try id -p:


     jqpublic@purgatorio% id -p
     uid=127(jqpublic) gid=10(staff) projid=10(group.staff)

(A user's default project can be changed by editing /etc/user_attr. The syntax to use for this is "username::::project=projectname". For example:


     gdm::::project=test

For more information, consult the man page for user_attr(4).).

In this case, jqpublic's project is 'default.' This project has no resource constraints assigned to it. Let's confirm that the resource limitation works on the maximum file size we defined for the test project:


     jqpublic@purgatorio% id -p
     uid=127(gdm) gid=10(staff) projid=10(group.staff)
     jqpublic@purgatorio% mkfile -v 32M testfile.project=default
     testfile.project=default 33554432 bytes
     jqpublic@purgatorio% newtask -p test csh
     jqpublic@purgatorio% id -p
     uid=127(gdm) gid=10(staff) projid=10000(test)
     jqpublic@purgatorio% mkfile -v 32M testfile.project=test
     testfile.project=test 33554432 bytes
     Could not set length of testfile.project=test: File too large

The resource constraint worked!

You can also set up the resource-management framework so that it writes a notification to the syslog when a resource control is tripped. This can be done via rctladm:


     # rctladm -e syslog process.max-file-descriptor
     # rctladm |grep process.max-file-size
     process.max-file-size       syslog=notice  [ lowerable deny
file-size ]

An attempt by a user in the 'test' project to generate a 32MB file generates the following warning in /var/adm/messages:


     Jun 24 23:33:09 british-museum.internal genunix: [ID 883052
kern.notice] privileged rctl process.max-file-size (value 16777216)
exceeded by process 2428

One other interesting feature about projects is that the prstat command can summarize running processes by project, if you give it the -J switch. This can be quite useful if, for example, you have a "Webserver" project to which all your Web server daemons belong. (If you aren't familiar with prstat, I encourage you to play around with it: it's a useful application. It gives output that is quite similar to that of top, but it is more lightweight. It became available in Solaris 8.)

jqpublic@purgatorio% prstat -cJ
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
2315 gdm 257M 257M cpu15 60 0 0:00:05 2.2% rmop/1
2316 gdm 5704K 4800K cpu19 59 0 0:00:00 0.2% prstat/1
198 root 2888K 2232K sleep 59 0 0:00:00 0.0% nscd/20
179 root 3464K 1912K sleep 59 0 0:00:00 0.0% syslogd/12
296 root 2696K 1776K sleep 59 0 0:00:00 0.0% devfsadm/5
201 root 1464K 1128K sleep 59 0 0:00:00 0.0% powerd/2
313 root 1784K 1144K sleep 59 0 0:00:00 0.0% ttymon/1
166 daemon 2472K 1736K sleep 9 0 0:00:00 0.0% statd/1
167 root 3672K 1984K sleep 59 0 0:00:00 0.0% automountd/2
165 root 184K 1448K sleep 59 0 0:00:00 0.0% lockd/2
139 root 2432K 1816K sleep 59 0 0:00:00 0.0% inetd/1
116 root 2144K 1136K sleep 59 0 0:00:00 0.0% rpcbind/1
60 root 3160K 2368K sleep 59 0 0:00:00 0.0% picld/4
55 root 2288K 1568K sleep 59 0 0:00:00 0.0% syseventd/14
99 root 1664K 712K sleep 59 0 0:00:00 0.0% in.routed/1

PROJID NPROC SIZE RSS MEMORY TIME CPU PROJECT
10000 3 264M 263M 4.5% 0:00:05 2.4% test
0 32 78M 47M 0.7% 0:00:44 0.0% system
3 3 5624K 4232K 0.1% 0:00:00 0.0% default
10 2 2928K 2448K 0.0% 0:00:00 0.0% group.staff

Total: 40 processes, 102 lwps, load averages: 0.17, 0.04, 0.02

Using the resource-management framework contained in projects, we can create a set of projects for each major task being used by Alighieri Financial Services. And we can easily control just how much processor time is permissible for each group of processes, and we can control a host of other resources that could possibly be competed against. This illustrates a side of performance management that isn't very glamorous--but it's extremely useful. If we can control resource competition, we can ensure that the applications that are particularly performance-sensitive receive the support from the system that they need in order to be fast.

System Performance Tuning

Related Reading

System Performance Tuning
By Gian-Paolo D. Musumeci, Mike Loukides

Gian-Paolo D. Musumeci is a research engineer in the Performance and Availability Engineering group at Sun Microsystems, where he focuses on network performance.


Return to ONLamp.com

Copyright © 2009 O'Reilly Media, Inc.