SystemTap usage is based on SystemTap scripts (*.stp). They tell SystemTap which type of information to collect, and what to do once that information is collected. The scripts are written in the SystemTap scripting language that is similar to AWK and C. For the language definition, see http://sourceware.org/systemtap/langref/.

The essential idea behind a SystemTap script is to name events, and to give them handlers. When SystemTap runs the script, it monitors for certain events. When an event occurs, the Linux kernel runs the handler as a sub-routine, then resumes. Thus, events serve as the triggers for handlers to run. Handlers can record specified data and print it in a certain manner.

The SystemTap language only uses a few data types (integers, strings, and associative arrays of these), and full control structures (blocks, conditionals, loops, functions). It has a lightweight punctuation (semicolons are optional) and does not need detailed declarations (types are inferred and checked automatically).

For more information about SystemTap scripts and their syntax, refer to Section 5.3, “Script Syntax” and to the stapprobes and stapfuncs man pages, that are available with the systemtap-doc package.

Tapsets are a library of pre-written probes and functions that can be used in SystemTap scripts. When a user runs a SystemTap script, SystemTap checks the script's probe events and handlers against the tapset library. SystemTap then loads the corresponding probes and functions before translating the script to C. Like SystemTap scripts themselves, tapsets use the filename extension *.stp.

However, unlike SystemTap scripts, tapsets are not meant for direct execution—they constitute the library from which other scripts can pull definitions. Thus, the tapset library is an abstraction layer designed to make it easier for users to define events and functions. Tapsets provide useful aliases for functions that users may want to specify as an event (knowing the proper alias is mostly easier than remembering specific kernel functions that might vary between kernel versions).

The main commands associated with SystemTap are stap and staprun. To execute them, you either need root privileges or must be a member of the stapdev or stapusr group.

For a list of options for each command, use --help. For details, refer to the stap and the staprun man pages.

Apart from the commands above which are used in a setup where you build the kernel modules on the same machine that you want to probe, there is also a specific set of commands for a client-server setup: systemtap-client and systemtap-server, the latter containing a number of subcommands. This set of commands allows you to compile a SystemTap module on a machine other than the one on which it will be run. For more information about this specific setup and the commands involved, refer to Section 5.2, “Installation and Setup” and to the stap-server and stap-client man pages.

To avoid giving root access to users just for running SystemTap, you can make use of the following SystemTap groups. They are not available by default on SUSE Linux Enterprise, but you can create the groups and modify the access rights accordingly.

The following list gives an overview of the SystemTap main files and directories.

If you use the SystemTap client-server setup, the following directories are also important:

For this setup, install the following packages (using either YaST or zypper).

To check if all packages are correctly installed on the machine and if SystemTap is ready to use, execute the following command as root.

stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'

It probes the currently used kernel by running a script and returning an output. If the output is similar to the following, SystemTap is successfully deployed and ready to use:

Pass : parsed user script and 59 library script(s) in 80usr/0sys/214real ms.
Pass : analyzed script: 1 probe(s), 11 function(s), 2 embed(s), 1 global(s) in 
 140usr/20sys/412real ms. 
Pass : translated to C into
 "/tmp/stapDwEk76/stap_1856e21ea1c246da85ad8c66b4338349_4970.c" in 160usr/0sys/408real ms. 
Pass : compiled C into "stap_1856e21ea1c246da85ad8c66b4338349_4970.ko" in 
 2030usr/360sys/10182real ms. 
Pass : starting run. 
 read performed
Pass : run completed in 10usr/20sys/257real ms. 

In case any error messages appear during the test, check the output for hints about any missing packages and make sure they are installed correctly. Rebooting and loading the appropriate kernel may also be needed.

A SystemTap compile server listens for connections from SystemTap clients on a secure SSL network port and accepts requests to run the SystemTap front-end. The server advertises its presence and configuration on the local network using avahi (a free Zeroconf implementation that allows programs to publish and discover services and hosts in a local network without any specific configuration). The compile server broadcasts its IP address, port, and details about the Linux kernel it runs. Thus, the SystemTap client can automatically detect a compile server on the network that is compatible to the client's kernel version.

As SystemTap exposes kernel internal data structures and potentially private user information, it provides several layers of security:

5.2.2.2. Setting Up the Server¶

You have two choices for setting up the SystemTap compile server: you can run it as root or as non-root user. This has implications on the certificate management on server- and client-side and on the process of establishing a given compile server as trusted by a given client. For the SSL connection between the compile server and the SystemTap client, you need to create a certificate for authentication. Depending on how the SystemTap compile server is set up (as root or as non-root), the location of the server certificate differs. When set up as root user, the certificate is stored in a database at /etc/systemtap/ssl/server. However, when the compile server is set up as non-root (usually by the user stap-server), the server certificate is stored in a database in the systemtap-server user's home directory: ~/.systemtap/ssl/server.

Procedure 5.1. Running the Compile Server as Non-root User¶

For this setup, it is advisable to create a dedicated system group and user for the compile server.

1. Log in as root.

2. Create a home directory for the compile server user, for example:

   mkdir /var/lib/stapserver

3. Add a system group for the operation of the compile server. In the following example, the group is named stap-server and the group ID (GID) is 155, but you can also specify a different group name or GID:

   groupadd -g 155 -r stap-server

4. Add a user belonging to the group you created before and specify the user's home directory:

   useradd -c "SystemTap Compile Server" -u 155 -g stap-server -d \ /var/lib/stapserver -m -r -s /sbin/nologin stap-server

   The command above will create a user named stap-server with the user ID 155. The user's finger information is specified with -c and the options -g and -d specify the user's main group (stap-server) and his home directory you created in Step 2, respectively. The user account will be a system account (specified with -r) and the user will not be able to log in, as his login shell is set to /sbin/nologin with the -s option.

5. Change the owner and the group for the home directory to use:

   chown -R stap-server.stap-server /var/lib/stapserver/

6. Run a shell as user stap-server and pass the stap-gen-cert command to generate a SystemTap certificate:

   su -s /bin/sh - stap-server -c  /usr/bin/stap-gen-cert

   You are prompted to set a password for the SystemTap server certificate and key database.

7. Enter a password for the SystemTap server certificate and confirm it.

   This generates a certificate (stap.cert) that is stored in the systemtap-server user's home directory—in this case: /var/lib/stapserver/.systemtap/ssl/server.

8. Start the compile server with:

   su -s /bin/sh - stap-server -c /usr/bin/stap-start-server

   Upon first start of the compile server, this creates a client-side certificate database in the systemtap-server user's home directory (~/.systemtap/ssl/client) to which the server's certificate has now automatically been added. Thus, a server started by the user stap-server is automatically trusted by clients run by that user.

Procedure 5.2. Running the Compile Server as root User¶

Compared to Procedure 5.1, “Running the Compile Server as Non-root User”, this setup is much simpler but it has security implications.

	Security Risk
In the following setup, the compile server certificate is stored in /etc/systemtap/ssl/server, together with the client-side database located at /etc/systemtap/ssl/client. As these files are accessible for anyone, anyone can run the stap-client command, thus potentially exposing kernel internal data structures and private user information.

1. Log in as root.

2. Create a SystemTap certificate by executing the following command:

   /usr/bin/stap-gen-cert

   You are prompted to set a password for the SystemTap server certificate and key database.

3. Enter a password and confirm it.

   The certificate (stap.cert) is generated. In contrast to the setup as non-root, it is stored in a database located at /etc/systemtap/ssl/server.

4. Start a SystemTap server on the local host by using the following command:

   /usr/bin/stap-start-server

   At the same time, a client-side certificate database is created at /etc/systemtap/ssl/client. The server certificate is automatically added to the client-side certificate database.

The client-side certificate database created for root is also the global client-side database for all clients on the host. Thus, a server started by root is automatically trusted by clients run by any user on that host: Any user can now compile kernel modules on the compile server using the stap-client command. For more information about the security implications, see the Safety and Security section of the stap-server man page.

5.2.2.4. Using the Client¶

After you have set up the SystemTap compile server and client as described in the previous sections, you can make use of the stap-client program. It is analogous to the stap front-end, except that it tries to find a compatible SystemTap compile server on the local network. It then uses this server for compiling the SystemTap script into a module, loading the kernel module and enabling the probes (passes 1-4 of a SystemTap session). If requested, pass 5 actions are performed on the localhost using staprun. For more information about a SystemTap session and the individual passes, see Section 5.2.1, “Classic Setup and Initial Test”.

	Executing stap-client
You can run stap-client either as root or as non-root. If run as non-root, the underlying staprun command needs to be suid and the user executing stap-client must be a member of the stapdev group. For more information, refer to Section 5.1.3, “Commands and Privileges”.

Usually, a running SystemTap compile server on the local network advertises its presence using avahi and is automatically detected by the SystemTap client. The following procedure illustrates how to make use of the SystemTap client-server setup and covers the most common commands and options needed for that.

1. To make sure that a compatible SystemTap server is running on your local network, execute the following command on the SystemTap client:

   stap-find-servers

   This invokes avahi-browse to find servers. The details of any servers found are echoed to standard output. If this command does not return anything, no compatible SystemTap server can be found on your network.

2. In this case, log in to the compile server and run

   stap-start-server

   This starts avahi-publish-service in the background. The server listens for connections on a random port and advertises its presence on the local network using the avahi daemon. If the server is started successfully, the process ID of the server is echoed to standard output.

   Note that stap-start-server does not work for the initial setup as described in Procedure 5.2, “Running the Compile Server as root User”, where /usr/bin/stap-serverd is used instead. stap-start-server puts the server in the background—thus, you would not see the prompt asking for the server certificate password.

   ps -ef | grep avahi

   should now return an output similar to the following:

   
   avahi 3300     1  0 15:14 ?      00:00:00 avahi-daemon: running [linux-48zp.local]
   root  4687  4655  0 18:03 ttyS0  00:00:00 avahi-publish-service Systemtap Compile Se
   root  4700  4160  0 18:05 ttyS0  00:00:00 grep avahi

3. To run a simple test, execute the following command on the SystemTap client:

   stap-client -e 'probe begin { printf("Hello"); exit(); }'

   This compiles and executes a simple example on any compatible SystemTap server found on the local network. If the test is successful, it prints “Hello” to standard output.

Instead of using any compatible server found on the network, you can also determine which SystemTap server to contact and use. To do so, run the stap-client command with the --server option. It lets you specify the hostname or IP address of the SystemTap server, optionally also a port (which is useful fore connecting to non-local servers). For more information and details about the other available commands and options, refer to the stap-server and stap-client man pages.

A SystemTap script can have multiple probes. They must be written in the following format:

probe event {statements}

Each probe has a corresponding statement block. This statement block must be enclosed in { } and contains the statements to be executed per event.

probe begin
{
   printf ("hello world\n")
   exit ()
}

If your statement block holds several statements, SystemTap executes these statements in sequence—you do not need to insert special separators or terminators between multiple statements. A statement block can also be nested within another statement blocks. Generally, statement blocks in SystemTap scripts use the same syntax and semantics as in the C programming language.

SystemTap supports a number of built-in events.

The general event syntax is a dotted-symbol sequence. This allows a breakdown of the event namespace into parts. Each component identifier may be parametrized by a string or number literal, with a syntax like a function call. A component may include a * character, to expand to other matching probe points. A probe point may be followed by a ? character, to indicate that it is optional, and that no error should result if it fails to expand. Alternately, a probe point may be followed by a ! character to indicate that it is both optional and sufficient.

SystemTap supports multiple events per probe—they need to be separated by a comma (,). If multiple events are specified in a single probe, SystemTap will execute the handler when any of the specified events occur.

In general, events can be classified into the following categories:

probe timer.s(4)
{
   printf("hello world\n")
}

For detailed information about supported events, refer to the stapprobes man page. The See Also section of the man page also contains links to other man pages that discuss supported events for specific subsystems and components.

Each SystemTap event is accompanied by a corresponding handler defined for that event, consisting of a statement block.

function function_name(arguments) {statements}
probe event {function_name(arguments)}

printf ("%s(%d) open\n", execname(), pid())

[...]
vmware-guestd(2206) open
hald(2360) open
[...]
   

foo = gettimeofday( )

global count_jiffies, count_ms
probe timer.jiffies(100) { count_jiffies ++ }
probe timer.ms(100) { count_ms ++ }
probe timer.ms(12345)
{
  hz=(1000*count_jiffies) / count_ms
  printf ("jiffies:ms ratio %d:%d => CONFIG_HZ=%d\n",
    count_jiffies, count_ms, hz)
  exit ()
  }