Controlling Resources with Solaris Projectsby Gian-Paolo D. Musumeci, author of System Performance Tuning, 2nd Edition
A Case Study in Performance Analysis
Alighieri Financial Services, a company that helps expatriates manage their finances, is in something of a mess. Its complex financial-modeling workloads have to share resources with internal applications such as payroll and account management, and these conflicts are starting to cause severe performance problems--nothing is getting done on time, and their storage keeps filling up. The company needs some way to effectively control resource availability.
Fortunately, Alighieri just upgraded its server, a Sun Ultra Enterprise 4500, to Solaris 9. This upgrade makes our task somewhat straightforward because a powerful new way to manage processes has been introduced--projects. A project is a grouping of processes that are subject to a set of constraints. For example, we could limit a project to how many processes it could have, or how large of a file it could create.
Before we get into fixing Alighieri's problems, let's briefly cover the background. The key file to keep track of here is the project database,
The project file contains a series of one-line entries, one for each configured project. (If you don't want to edit the
/etc/project file by hand, you can use the
projmod commands to add, remove, or modify projects, respectively.)
Each line takes the following form:
The project-name field is the name of the project, and must be specified as an alphanumeric string--periods (.) and colons (:) are not allowed.
The project-id field is a unique numerical identifier for the project. The maximum project ID is 2,147,483,647; project IDs between 0 and 99 are reserved for the system.
The comment field is a description of the project.
The user-list is a list of all the users who are allowed to associate with the project. Wildcards are allowed: an asterisk (*) permits all users to join the project, and an exclamation point (!) followed by a username excludes that user. An exclamation point followed by an asterisk (!*) excludes all users.
The group-list field is analogous to the user-list field. Wildcard constructs are valid here, just as they are in the user-list field.
The attributes field defines the resource-control attributes associated with the project. Here are a few of the most interesting options:
|task.max-cpu-time||The maximum amount of CPU time available to all the processes in this project (in milliseconds).|
|task.max-lwps||The maximum number of lightweight processes available to all the processes in this project.|
|process.max-cpu-time||The maximum amount of CPU time available per process for each process in this project (in milliseconds).|
|process.max-file-descriptor||The maximum number of file descriptors permitted to each process in this project.|
|process.max-file-size||The maximum file size available to processes in this project (in bytes).|
|process.max-core-size||The maximum size for a core file created by a process in this project (in bytes).|
|process.max-data-size||The maximum heap size for a process in this project (in bytes).|
|process.max-stack-size||The maximum stack size for a process in this project (in bytes).|
|process.max-address-space||The maximum amount of address space (over all segments) available for a process in this project (in bytes).|
(There are other resource controls available--these are just the ones that are most immediately interesting.)
Each attribute is specified by setting it equal to a triplet, like this:
There are three available values for
privlevel. The first, "basic", means that the threshold can be modified by the process, without any special
privileges; the second, "privileged", means that the threshold can be changed only by
the superuser; the third, "system", means that the threshold is fixed for the
duration of the operating system instance (for example, until a reboot).
Multiple attributes can be specified for a single project; separate them with semicolons.
Now that we've walked through the theory, let's create a project.
We can put the following line in
Alternatively, we could run
projadd to create this project:
purgatorio# projadd -p 10000 -c 'test project' -U jqpublic test purgatorio# grep ^test: /etc/project test:10000:test project:jqpublic::
projmod won't let you change the resource-control field.)
This project is named "test" and has a project ID of "10000". Membership is restricted to the user "jqpublic," and any project running in the test project will not be allowed to create a file greater than 16MB (16MB = 16,777,216 bytes).
When jqpublic next logs in, they can run `projects` to get a list of all the projects they are allowed to participate in:
jqpublic@purgatorio% projects default group.staff test
To find out which project is associated with the currently running shell,
jqpublic@purgatorio% id -p uid=127(jqpublic) gid=10(staff) projid=10(group.staff)
(A user's default project can be changed by editing
/etc/user_attr. The syntax to use for this is "username::::project=projectname". For example:
For more information, consult the man page for
In this case, jqpublic's project is 'default.' This project has no resource constraints assigned to it. Let's confirm that the resource limitation works on the maximum file size we defined for the test project:
jqpublic@purgatorio% id -p uid=127(gdm) gid=10(staff) projid=10(group.staff) jqpublic@purgatorio% mkfile -v 32M testfile.project=default testfile.project=default 33554432 bytes jqpublic@purgatorio% newtask -p test csh jqpublic@purgatorio% id -p uid=127(gdm) gid=10(staff) projid=10000(test) jqpublic@purgatorio% mkfile -v 32M testfile.project=test testfile.project=test 33554432 bytes Could not set length of testfile.project=test: File too large
The resource constraint worked!
You can also set up the resource-management framework so that it writes a notification to the syslog when a resource control is
tripped. This can be done via
# rctladm -e syslog process.max-file-descriptor # rctladm |grep process.max-file-size process.max-file-size syslog=notice [ lowerable deny file-size ]
An attempt by a user in the 'test' project to generate a 32MB file
generates the following warning in
Jun 24 23:33:09 british-museum.internal genunix: [ID 883052 kern.notice] privileged rctl process.max-file-size (value 16777216) exceeded by process 2428
One other interesting feature about projects is that the prstat
command can summarize running processes by project, if you give it the
-J switch. This can be quite useful if, for example, you have a
"Webserver" project to which all your Web server daemons belong. (If you
aren't familiar with
prstat, I encourage you to play
around with it: it's a useful application. It gives output that is
quite similar to that of
top, but it is more lightweight. It
became available in Solaris 8.)
jqpublic@purgatorio% prstat -cJ
Total: 40 processes, 102 lwps, load averages: 0.17, 0.04, 0.02
Using the resource-management framework contained in projects, we can create a set of projects for each major task being used by Alighieri Financial Services. And we can easily control just how much processor time is permissible for each group of processes, and we can control a host of other resources that could possibly be competed against. This illustrates a side of performance management that isn't very glamorous--but it's extremely useful. If we can control resource competition, we can ensure that the applications that are particularly performance-sensitive receive the support from the system that they need in order to be fast.
Gian-Paolo D. Musumeci is a research engineer in the Performance and Availability Engineering group at Sun Microsystems, where he focuses on network performance.
Return to ONLamp.com