Did you know that you aren’t limited to working on one thing at a time while on a Linux command line? You can actually “minimize” a program that you are in, get back to the command line, and then return to the program whenever you’d like.

When you run a program or script on the Linux command line (from now on referred to as the shell), you are creating a new job. For those that are used to GUI environments, each of these jobs is somewhat like a window on the desktop. Just as you can have multiple windows and switch between them, the shell is capable of managing multiple jobs and allows you to switch between them.

There is a lot to cover, so let’s start simple by describing what states a shell job can be in.

Jobs States

Foreground Jobs

Up to now, you may have only worked with foreground jobs in the shell. Foreground jobs are ones that don’t return you to the command prompt until you exit out of the job or the job finishes.

If you run “ping yahoo.com” from the shell, the ping process will become a foreground job and will not return you to the command prompt until you stop it.

Just in case you didn’t know, you can stop the ping process by pressing Ctrl+c. Unlike in many applications that you may be used to, Ctrl+c does not mean “copy” in a Linux shell. Instead of copying text, this signals the foreground process to terminate by sending it the SIGINT (Signal Interrupt) signal.

An example of a foreground job that won’t return you to the command prompt until it is finished is the zip command. Imagine that you have a large amount of files inside a folder called “stuff”. If I run “zip -r stuff.zip stuff” and don’t know about job management, I won’t have access to the shell until zip finishes its task and returns me to the prompt.

Background Jobs

Putting a job in the background allows the job to run exactly as if it were in the foreground except that it does not receive user input. If a command produces output and is put in the background, it will still produce the output.

Why would putting a job in the background be helpful then you may wonder. There are a number of reasons, and they will become clear in the examples below.

Stopped Jobs

There is technically a third state that a job can be in. Jobs that aren’t in the foreground yet are also not in the background have been stopped.

Like background jobs, stopped jobs do not receive user input. Unlike background jobs, stopped jobs do not produce output. As a matter of fact, they don’t do anything. They have essentially been put on hold and won’t do any processing until they are resumed or killed.

Multitasking 101: Stopping and Resuming Processes

It may seem like stopped jobs are the odd man out, but they are actually the key to getting started with multitasking on the Linux shell. I use stopped jobs every day, and I couldn’t imagine using the shell without them.

Imagine that you are in your favorite shell text editor working on a program or modifying config files, and you can’t remember what the IP address you need is but you could easily look it up with dig. You could easily open up another shell connection and run the command, but it would be quicker to simply stop the editor, run the command quickly, copy the text, and bring your editor to the foreground.

The following is a sequence that illustrates how this example would look on your shell.

[chris@local ~]$ sudo vi /etc/httpd/conf/httpd.conf
[sudo] password for chris:
Ctrl+z
[1]+  Stopped          sudo vi /etc/httpd/conf/httpd.conf
[chris@local ~]$ dig +short chrisjean.com
96.125.165.12
[chris@local ~]$ fg %1

After running these commands, you will be back in your editor.

The key to stopping a running job is the Ctrl+z key combination. Again, some of you may be used to Ctrl+z as the shortcut to undo, but in the Linux shell, Ctrl+z sends the SIGTSTP (Signal Tty SToP) signal to the foreground job. When you press this key combination, the running program will be stopped and you will be returned to the command prompt.

You may notice the “%1” after fg. The “1” is the job number that is to be brought to the foreground. The job number is listed in brackets when the job was stopped. These numbers are crucial when you have multiple jobs in stopped states. At any time, you can run “jobs” to get a list of the current jobs.

I stop jobs every day, and it greatly reduces the number of active shell sessions I have running. Sometimes I have stopped jobs that I set up ahead of time when I first access the shell. For example, I may open up a MySQL session, connect it to the database of the site I’m working on, and then stop it. Any time I need to run a query, I can load up the job by running “fg 1“, run the query, and then stop the job again.

Multitasking 202: Running Jobs in the Background

Stopping jobs is great, but what if we want to be returned to the command prompt while still allowing the job to run? Sending a job to the background will do exactly that.

As I talked about in my post about finding files, the locate command is extremely helpful. Sometimes you need to manually run the updatedb command in order for the locate database to update. The problem is that the updatedb command can take a very long time to run, sometimes it can take more than an hour.

I’ll use the example of the updatedb command to show how you can take a foreground process and put it in the background so that you can free up the shell.

[chris@local ~]$ sudo updatedb
[sudo] password for chris:
Ctrl+z
[1]+  Stopped            sudo updatedb
[chris@local ~]$ bg %1
[1]+ sudo updatedb &
[chris@local ~]$

There are a few items of note here.

  1. I’m back at the command prompt and can do anything I’d like while the job works merrily away in the background.
  2. Notice how I used “%1” with bg just as I did with fg. Both commands accept a job number to tell it which job is to be sent to the foreground/background.
  3. After running bg, a line similar to when I pressed Ctrl+z was printed showing the command and job number. However, there is also the addition of the “&” at the end. We’ll get into the reason for that in the next section.

When a process that is running in background finishes, you will receive a message the next time your command prompt refreshes. The prompt refreshes when you run another command or simply hit Enter to get a new prompt.

Continuing from the previous example, I receive output similar to the following when the updatedb command finishes:

[chris@local ~]$ 
[1]+  Done              sudo updatedb
[chris@local ~]$

As with all the other job-related output, you will be notified of the job number and the command.

Multitasking 303: Starting Jobs in the Background

Now that you see the value of having jobs run in the background, you might wonder if you can just start the command in the background to begin with. In fact, you can!

Remember that odd ampersand “&” after the command in the output of the bg command? The ampersand means that the job is running in the background. You can run a command with the ampersand at the end to automatically put the job in the background.

Going back to the updatedb example again, we could have simply run the following to have the job in the background and get our command prompt back immediately:

[chris@local ~]$ sudo -v
[sudo] password for chris:
[chris@local ~]$ sudo updatedb &
[1] 15231
[chris@local ~]$

You might wonder what that completely unnecessary “sudo -v” command is all about. Remember that when you send a job to the background, it cannot receive input from the user. Since sudo prompts for the password when it hasn’t been run recently, I ran that command before running the background sudo command to ensure that the password wouldn’t be prompted for. If you do send a sudo command to the background and sudo prompts for a password, you need to bring it to the foreground first to enter the password and then send it to the background again.

The output when sending the job directly to the background is different than running bg. Instead of telling you the command, it tells you the actual process ID. Every program that runs in Linux is a process and has a process ID. You can use these process IDs to kill a specific process, change the process’ execution priority, and isolate the resource utilization by that process.

Even though you’ve put a task in the background, you can still bring it to the foreground using the fg command. In the previous example, I can pull the background process to the foreground by running “fg %1“.

Multitasking Graduate School: Killing and Prioritizing Jobs

Now that you know the fundamentals of managing jobs, it’s time to get more advanced and tell you how to terminate jobs and modify their execution priority.

Terminating Jobs

Sometimes a process goes rogue and either won’t respond to input, is consuming massive amounts of resources, or both. At times like this, it’s very important to know how to kill a job which will cease the program’s execution and free up its used resources.

The command used to kill processes is appropriately called kill. The kill command can accept either process IDs or job numbers (the job number must be preceded by a percent sign, “%”).

If my updatedb job from before fails to respond, the job has a job number of “1”, and I want to tell the job to stop, I could run the following command:

[chris@local ~]$ sudo kill %1
[sudo] password for chris:
[chris@local ~]$

Notice that there isn’t any confirmation that the kill command was successful. You can check to ensure that the job has been terminated by running the “jobs” command and looking for that job’s listing.

By default, the kill command sends a SIGTERM (Signal Terminate) signal to the process; however, sometimes this is not enough to stop the execution of an out of control process. To shut down these processes, we need something more powerful.

[chris@local ~]$ sudo kill -9 %1
[sudo] password for chris:
[chris@local ~]$

This may not look much different, but the addition of the “-9” option tells the command that things are serious now. This option causes a SIGKILL (Signal Kill) signal to be sent to the process. A kill signal cannot be intercepted by a process and causes the process to immediately terminate without allowing it to clean up after itself. It’s for this reason that you should only use SIGKILL after first sending the process a SIGTERM.

As mentioned before, you can also run a kill command on a process ID rather than a job number. If we wished to kill a process with a process ID of 12267, we can run the following command:

[chris@local ~]$ sudo kill 12267
[chris@local ~]$

Note that you don’t have to always run the kill command with sudo, you just have to ensure that whatever privilege level you are running the kill command at has the ability to control that process. In other words, if you are trying to kill a process run with sudo, you’ll need to use sudo to kill it as well.

Prioritizing Jobs

Linux does a great job of prioritizing tasks automatically; however, what Linux thinks is best isn’t always what you want. Basically, Linux always wants to make a process finish as soon as possible, which means that you get a nice bit of lag when you try to do your work as a CPU or disk hungry process churns away. Fortunately, there are tools that will let you tell Linux what you want.

First, let’s deal with managing a process’ use of CPU time. If you are going to run a command that will eat up large amounts of CPU cycles and you want to make sure that this process doesn’t keep you from doing your work, you can run the command with nice to give the command a lower-than-default priority.

[chris@local ~]$ sudo -v
[sudo] password for chris:
[chris@local ~]$ sudo nice updatedb &
[1] 16312
[chris@local ~]$

As before, a junk sudo command was run first to take care of the password prompt.

Running the command in nice lowers the processes CPU priority and will result in smoother CPU performance for your other applications.

What if you already ran a command whose process is now greedily chewing away at your CPU resources? That’s where the renice command comes into play.

With renice, you can change the CPU priority of an existing process.

[chris@local ~]$ sudo -v
[sudo] password for chris:
[chris@local ~]$ sudo updatedb &
[1] 16324
[chris@local ~]$ sudo renice 10 16324
16324: old priority 0, new priority 10
[chris@local ~]$

The renice command accepts a priority as the first argument and the process ID as the second argument. The priority argument has a range from -20 to 19 where -20 is the highest priority and 19 is the lowest. In other words, a higher numeric value priority results in lower CPU priority.

That takes care of the CPU, but admittedly, the updatedb command does more damage to the IO or disk traffic than it does to the processor. How can we reign in a process’ use of the disk so that performance for the rest of the system improves? For that, we use the ionice command.

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

[chris@local ~]$ sudo -v
[sudo] password for chris:
[chris@local ~]$ sudo updatedb &
[1] 16324
[chris@local ~]$ sudo ionice -c3 -p16324
[chris@local ~]$

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

[chris@local ~]$ sudo -v
[sudo] password for chris:
[chris@local ~]$ sudo updatedb &
[1] 16324
[chris@local ~]$ sudo ionice -c2 -n7 -p16324
[chris@local ~]$

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automated script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

[chris@local ~]$ ps -C updatedb
PID TTY          TIME CMD
4234 ?        00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

Another useful command is “ps -u user” (replace “user” with your username) which lists all processes your user has access to. Basically, these are processes that your user or some process using your user’s permissions started.

Another really helpful use of ps is “ps aux“. This command will list all the system’s processes, which user owns the process, and even detail the resource usage of the process. This is very helpful for finding processes that you don’t know enough about to find using other methods.

Mutitasking Doctorate: Using Job Management in Scripts

Keep in mind that many scripts execute from inside shell environments. This means that you can manage the script’s job using the tools I’ve given you here.

If you look in the /etc/cron.daily folder, you’ll find many scripts that execute once a day. These scripts can create a huge load on a system while they execute. If around the clock performance is crucial, you can prioritize the shells that the scripts run in which causes all the commands run inside that shell to follow the same prioritization rules.

This all sounds very complex, but it’s actually very simple.

Let’s open up the /etc/cron.daily/mlocate file (pick another file if you don’t have this one). At the top, you should see a line like the following:

#!/bin/bash

This shebang line tells Linux what interpreter is used to execute the script. My example script uses the bash shell as the interpreter. Other scripts may use the sh shell instead of the bash shell, but that won’t make any difference.

Below the first line, put the following:

renice 10 $$
ionice -c3 -p$$

The $$ variable is a special variable that contains the shell’s own process ID. The two commands above cause the CPU priority to lower to 10 and changes the IO priority to class 3 (Idle).

Add this type of customization to your resource hungry scripts, and your system will perform more smoothly around the clock. Keep in mind that this also can result in the scripts taking much longer to complete.

You can use those same commands above using the $$ variable to change the CPU and IO priority of shells that you open in order to lower the impact you have while working on the system. This can be a smart thing to do if you need to compile software or run other intensive processes on production machines.

Important Commands

Here’s a list of all the commands, key combos, and other important bits that I went over with a brief description for each. It’s like the word list at the end of children’s books.

  • fg – Brings a stopped or background job to the foreground
  • bg – Sends a stopped job to the background
  • jobs – Lists the current jobs and their corresponding job number
  • Ctrl+c – Key combination that sends a SIGINT signal to the foreground process. Use this to close processes that don’t stop (such as ping or top) or terminate a process that no longer responds or is no longer needed.
  • Ctrl+z – Key combination that sends the SIGTSTP signal to the foreground process. Use this to send a foreground process to a stopped state.
  • & – Use an ampersand at the end of a command to send the resulting job to the background immediately
  • kill – Sends signals to processes based on process ID or job number. Useful to send SIGTERM or SIGKILL signals to forcibly stop processes.
  • nice – Runs the given command with a lower-than-default CPU priority
  • renice – Changes an existing process’ CPU priority
  • ionice – Changes an existing process’ IO priority

Closing Thoughts

Frankly, I didn’t expect to make such a long post. If this post has helped you, please leave a comment and let me know what you thought.

I’d greatly appreciate it if you’d share the post by using one of the icons below. Or you can click here to Digg this post.

Did I help you?