Linux Operating systems provide the ability to track system calls with strace utility. As simple explanation strace intercepts and prints system calls made by the related process. We know that Linux is actually an operating system kernel. Operating system kernel is responsible for low-level operations like device and hardware management, memory management, processes management, providing an interface for user-level processes and applications. Strace is especially used for diagnostic, instructional and debugging operations.

Syntax

The syntax of strace is like below.

strace  [-CdffhikqrtttTvVxxy]  [-In]  [-bexecve]  [-eexpr]...  [-acolumn] [-ofile] [-sstrsize] [-Ppath]... -ppid... / [-D] 
       [-Evar[=val]]... [-uusername] command [args] 
 
       strace -c[df] [-In] [-bexecve] [-eexpr]...  [-Ooverhead] [-Ssortby] -ppid... / [-D] [-Evar[=val]]... [-uusername]  command 
       [args]

Help

Simple and fast help about the tool can get with the following command.

$ strace -h

Trace Execution

Linux processes are generally started by calling them from bash or running background commands. In this example, we will start a process by calling the command. But by calling command we will send the command to the strace as a parameter like below. Keep in mind that this is not an emulation mode the command will be run and complete its job. While doing its job the system calls and related information will be printed to the console. While tracing information like system call name, address, library, file stat will be provided.

$ strace ls

Trace Specific System Calls

In the previous example, we have traced a command completely. This has created a lot of information and related output. This may be too much for us because we are looking for only some of them. To make things more clear to track we can provide the system call we want to trace. In the following example, we have only wanted to list open system call which is used to open files.

$ strace -e open ls

Save Trace Result To File

Up to now, we have print the trace result to the terminal. Our example process was simple to run and created smaller outputs. But what will happen if we want to run a long and complex process? Or we want to analyze the trace output later. In these situations we can save the trace output to a file. In this example, we have saved the trace output to the file named ls-trace.txt by using -o option. Then we can read the file with cat command.

$ strace -o ls-trace.txt ls

Trace Already Running Process

We have created new processes to trace in previous examples. But there are some situations we want to trace all ready fired and running processes. For example we have web servers running Apache and we want to trace Apache. How can we trace all ready running processes? First, we will find our process PID.

$ ps -C snapd

Trace With Process ID

We have learned  that our snap daemon named snapd have PID 1193. Then we will provide this PID to the strace to trace with -p option like below.

$ strace -p 1193

Write Output To A File

Another useful usage is saving the trace to a file with -o option

$ strace -o snapd-trace.txt -p 12793

And using tail command reading the trace output in real time.

$ tail -f snapd-trace.txt

Print Time Stamp

Time stamp information is very important especially metering the performance. In normal usage of strace time information is not provided. Timestamp information can be printed with -t option. In the example, we simply print timestamp which is consists of the current hour, minute and second information.

$ strace -t ls

Print Relative Timestamp

In the previous example, we have printed time stamp information in normal day format. But we may want to use a relative timestamp. The relative timestamp is calculated according to the process start time and process start time is set as 0 and all other system calls time is expressed according to the process start time.

$ strace -r ls

Print Time Stamp More Precise

In the previous example, we have printed the time stamp. This timestamp have normal precision where hour, minute and second information is provided. In some situations, we may need to get a more precise metric from the trace. The -tt option will create more precise metrics in nanoseconds about the trace.

$ strace -tt ls

Generate Statistics Report of System Calls

While tracing a process there is a lot of system calls used. Metrics and statistics about these calls can be printed in a table. This table will include metrics like time, seconds, call count, errors and related system call.

$ strace -c ls

Follow Forked Threads

Operating systems provide threads to multiple processes for more performance. Some daemon or server processes like Apache, Nginx, etc. uses mainly threads. Normally strace do not trace these threads. The -f and -ff options made the strace to find threads with the related process and trace them too.

$ strace -f ls