I recently tutored a friend in C coding. Since I hadn’t worked with C in at least 8 years, I really needed to have some references to rely on for syntax and other specifics. Fortunately, there are some easy man pages that can be installed in Ubuntu that offer helpful information that I was able to use to help refresh my memory.

These man pages are easily installed by installing the manpages-dev package. You can install this package via Synaptic or directly on the command line. I like the command line method personally, so I ran sudo apt-get install manpages-dev from the command line to quickly install the package.

After installing the package, I’m able to access man pages for functions such as printf, opendir, and putc. For each function, it shows the valid syntax as well as what library is required to make use of the function.

The information isn’t limited to functions as you can also access information on the libraries, such as stdio or string.

To access any of this information, simply run man [function or library name] such as man stdio.

This package isn’t limited to C functions/libraries. It is a general use Linux development suite of man pages. For a full list of what is installed, check out the file list.

All of the WordPress themes that I work on for iThemes are managed as Git repositories. Recently, we moved past the 100 repositories mark. That’s a lot of repositories to manage, and unfortunately, too many of those repositories contain duplicated information.

Later on, I might delve into how we use Git to manage our theme repos. For today, however, I’d like to focus on how I quickly and easily pushed up changes to more than a dozen repos in a single, albeit long, Bash command.

I had finished making updates to 16 Flexx repos, and I needed to push all of those changes up. Since I had multiple working repos in that folder, I was lucky that each of these repos began with the text “Flexx”. Also, since they are all part of the same series and need to keep the same version number, that simplified the tagging as all could be tagged as 2.5.0.

Given this information, I simply ran the following command from the directory that contained all the repository directories:

There’s a lot going on here, so I’ll break it up and explain what I’m doing.

This is basically a compound command. Bash’s for command is being used to step through the results of ls|grep Flexx. Each result will be stored to the variable i which can be referred to with $i.

In other words, I’m going to loop through each directory that contains the text “Flexx” (technically, it isn’t limited to just directories, but I don’t have any files containing “Flexx”, so I’m safe). For each iteration of the loop, the variable $i will contain the name of the folder I’m interested in.

The most important bit here is do. do simply begins the functional part of the loop and could be followed by any command that I wanted to start the loop with.

The echo command isn’t technically necessary. I simply have it there so that I can better determine what output belongs to which repository.

This is a very important command. It changes the directory to the repository I wish to run commands on.

This is the command that actually does what I want to do in each repo directory. This adds all the modified files, commits them with a message of “2.5.0″, pushes the changes to the remote repository, creates a new tag of 2.5.0, and pushes up the new tag.

I have all of these commands chained together with && so that if one command fails, the other ones won’t run. This prevents tagging in the event that the repository couldn’t be committed or pushed.

It’s a small command but necessary. This switches back to the main directory that holds all the repos, thus returning us back to a state that the next loop iteration can work with.

As with the other echo command, this simply allows me to keep track of what is going on more easily.

The done command finishes out the loop iteration.

I hope that by sharing this you can gain a bit of insight into how I work with repositories while also learning more about how you can do things with Bash.

I’ve had a lot of fun recently posting about how to do stuff on the command line in Linux. My focus is specifically for Ubuntu users, but the information and techniques can be used for any Linux distro.

Since I’m probably going to end up with a lot of content under this topic, I’ve decided to create a dedicated tag: Mastering the Command Line. I’ve gone through my older posts on this topic and tagged them as well. So, make sure to check out Mastering the Command Line if you want to know how to become a command line power user.

Back to today’s topic. You’re starting to learn how to use the command line, but it’s annoying to always have to type in similar commands over and over. If only there were a way to pull up commands that you’ve already run to run again as is or to quickly modify. Today, I’m going to teach you how to do exactly this.

Introduction to BASH Command Line History

BASH (Bourne Again SHell) is the default command line shell that is used in Ubuntu, and most other distros for that matter. The instructions that I give here are specific to BASH and may or may not work in other shells. Since most people don’t modify which shell they access through Terminal, I only point this out to inform those who have changed their shell that their experiences may differ depending on the shell used.

When you close a Terminal session, BASH writes all the commands you’ve run for that session to your ~/.bash_history file. When you load a new session, this file is read and those commands along with the new commands you execute for that session become your new command history.

It’s important to keep in mind that commands executed for a session are only written when you exit. If you have two sessions running concurrently, they cannot cross-reference the newly-executed commands run in the other session. Furthermore, if you force a Terminal session to quit, the command history will not be updated.

Simple Command History Navigation

The easy way to get started with your command history is to simply navigate with the up and down arrow keys. Using the up key, you can scroll through previous commands one command at a time. To scroll back down the list, you can use the down arrow key.

Once you’ve found the desired command, you can treat it as if you’ve typed that command in the command line. You can simply press Enter to execute the command again, or you can modify the command as needed before executing it.

This method is very good for quickly running commands that you have to execute repeatedly without many other commands between.

Listing and Searching Command History

BASH provides the history command that when executed with no options will list the entire command history. Each command will be preceeded with a number that represents that command’s command number. I’ll tell you how to use these numbers below.

You can also have the history command produce just the last certain number of commands by supplying a number. For example, if you want to see the previous ten commands, you can run the following:

Notice that the last command is the same as the command you ran to produce it. So, “history 1″ will just show you “[command number] history 1″.

Searching History

If you remember that you ran a really cool command and can’t quite remember how you did it, combining history with grep provides an easy way to find that command with just a piece of the command.

For example, let’s say that I ran a command that listed all the files that end in “.php” inside the ~/wordpress folder it then filters for just matches that contain the text “link” and then reverses the output. I was quite proud of that command, but I can’t remember how I did it. So, I search through my history.

Since I don’t think that I have the word “link” in many of my commands, I pick it as the target to search for and run the following:

That’s what I was looking for. Now I can bask in the glory of my ingenious command and use it as the inspiration for other inspired words of command line magic.

Running Previous Commands Again

Searching through the history is great and all, but I’d rather not have to copy and paste found commands. I’d like to just run them immediately. BASH to the rescue again.

If you’d like to run the previous command again quickly, you can simply run “!!”. The first exclamation point tells BASH that you are running a previous commands. The second exclamation point tells BASH that it’s the last command that you’d like to run again. It’s a double exclamation point simply to make it quick and easy to type in.

I’ve already shown you how to look through your history and find a specific command in your history complete with the command number. Now is the time to make use of that number. If I wanted to execute command number 510 (which bash) again using the history, I can simply run the following:

Notice how running the command again first lists the actual command and then the output.

BASH substitutes the !number command with the actual command from the history. This allows you to supply additional parameters to the command. The following is a silly example, but it gets the point across.

Notice how the command now has the additional parameter added to the original command. As expected, the full command is listed followed by the command output.

You can also do the standard manipulations such as piping the output, “!412 | grep wp-admin”, or redirecting the output, “!510 > shell_locations.txt”.

Searching and Running Previous Commands

The “!” functionality of BASH also allows you to search through the history and execute the first match.

The basic form of this simply searches for matches at the beginning of the command. “!find” executes the most recent command that begins with “find”.

You can also search the entire command for a match. “!?link?” executes the most recent command that contains “link”.

Advanced History with Regular Expressions

As with many tools in the Linux world, you can attain great power by combining the tool with regular expressions. The history in BASH allows you to run previous commands with a search and replace in Regex fashion.

Let’s take my searching through the ~/wordpress example, command number 412, and run it again with some modifications. This time, instead of searching ~/wordpress, I’d like to search through /home/site/public_hmtl/wordpress. I’d do that with the following:

Note that the search and replace is done by adding “:s|find|replace|” to the end of the history command. I used the pipes, “|”, to deliminate the find and replace portions to make it easier to add the forward slashes, “/”, to the terms. You can also use “:s/find/replace/”.

Closing Thoughts

I hope that you enjoy my Mastering the Command Line series. If you have any command line-specific requests, please leave a comment, and I will see about creating a tutorial on that topic.

Remember, those who forget history, are doomed to repeat it by manual entry.

Have you ever seen a bunch of ^M characters in a text file? This odd character at the end of a line can also be represented as a Ctrl+M or <CTRL>M. You don’t know what it is, and you want it to go away.

Today, I’ll help you understand what that odd ^M character is, why it is in some of your documents, and how to get rid of them.

All About the Newline

In text documents, lines are separated by what is called a newline (also known as a line break or end of line). Different operating systems have different character codes that represent this newline.

For example:

• DOS-based systems, including Windows, as well as a number of other, older non-Unix OSes use a carriage return (CR) character followed by a line feed (LF) character.
• Commodore and the old Apple OSes before OS X used a CR character. Since OS X+ is based partly on BSD, which is in turn based on Unix, the new Apple OSes use the Unix newline method described below.
• Unix and its derivatives (Linux, BSD, and others) all use the single LF character to represent a newline.

So what does all of this mean to you? It means that text documents that come from a Windows system won’t always play nice in Linux. The converse is true however. If you create a text file in Linux, many programs will fail to recognize the single LF as a newline and will render the document without any line breaks.

dos2unix and unix2dos to the Rescue

Fortunately, there are a couple of very easy to use programs that make dealing with this file format mess much easier. They are dos2unix and unix2dos.

These programs basically do exactly what their name implies: dos2unix takes a file and converts all DOS-style newlines to Unix-style newlines. unix2dos takes a file and converts all Unix-style newlines to DOS-stlye newlines.

I put Ubuntu and CentOS in the title because I’m going to give instructions for installing these programs on each of these distros. Why just these two? They are the two that I work with most often and are representative of the lion’s share of what people are using these days. If you need help with a different distro, please let me know in a comment.

Installing dos2unix and unix2dos in Ubuntu

There aren’t any dos2unix or unix2dos packages that can be found in Synaptic; howver, there is a packages that will install them for you. Simply open up Synaptic and install the tofrodos package. If you are like me and prefer to do this from the command line, you can run the following command:

That’s all there is to it. Not only will dos2unix and unix2dos install, but alias programs fromdos and todos will be installed as well. These additional programs work in the same manner, so it’s purely a matter of preference which ones you use.

Installing dos2unix and unix2dos in CentOS

I really thought that I had to install these in CentOS, but amazingly, the programs are already installed by default. I tested this in a Virtual Machine fresh install, and the programs were there on the first boot.

So, CentOS users, you’re already good to go.

Using dos2unix

Fortunately, using these programs couldn’t be easier.

Let’s say that you are in a Terminal (Applications > Accessories > Terminal) viewing text files. Maybe you just downloaded a new WordPress plugin and you are reading the readme.txt file. It doesn’t really matter. However, there is a problem. The readme.txt file has a bunch of ^M characters at the end of each line, and it’s really distracting.

Simply exit out of the editor you are currently in, since the file will be modified, and run the following command:

If there are multiple files, you can specify each one with a space separating each. For example:

The program doesn’t produce any output. Simply reopen your text file and look at all the beautiful non-existant ^M’s.

Using unix2dos

The unix2dos program has the exact same syntax as dos2unix. However, I thought it might be helpful to describe a situation in which you might need to use it.

You’re working on a project. You’ve just sent out your batch of files, and another member on the project complains that you are being a jerk and remove all the newlines in your text files. This other member is most likely using a Windows application that doesn’t understand the Unix newline format. Rather than getting into a format war, it’s typically better and quicker to simply convert your text files to a DOS/Windows format.

If you have a folder full of files that all need to be converted, simply run:

Now you can send these new files to your project group and hopefully avoid any more unproductive drama.

File Formats in Vi

If you happen to use Vi, you can change Vi back and forth between DOS and Unix modes for newlines with a simple command. “:set ff=dos” sets the editor to use DOS newline encoding and will save the file in a DOS-encoded format. “:set ff=unix” sets the editor use Unix newlines and will save the file in a Unix format.

Note that changing the format to dos from unix will always work as expected. This is because any file that contains just LF characters for new lines will be converted to CRLF while lines that already end in CRLF will be left as is.

If your Vi config defaults to a unix format and you open a DOS file, you will see the ^M characters. You can either use the dos2unix conversion utility first or change Vi first to the dos format and then to unix.

If you’d like Vi to default to DOS or Unix formating each time you start a new Vi session, you can add the setting to your ~/.vimrc file. In that file, either add “set ff=dos” or “set ff=unix” depending on your needs. Note the lack of the colon, :, in the .vimrc entries.

For more information on the ff or fileformat setting in Vi, check out the official documentation.

Closing Thoughts

Maybe one day we won’t have to worry about these types of things. For now however, it’s good to know the tools that make these problems easily manageable.

BTW, Happy Daylight Savings Day everyone. I hope you enjoyed the loss of an hour of sleep.

Did you know that you aren’t limited to working on one thing at a time while on a Linux command line? You can actually “minimize” a program that you are in, get back to the command line, and then return to the program whenever you’d like.

When you run a program or script on the Linux command line (from now on referred to as the shell), you are creating a new job. For those that are used to GUI environments, each of these jobs is somewhat like a window on the desktop. Just as you can have multiple windows and switch between them, the shell is capable of managing multiple jobs and allows you to switch between them.

There is a lot to cover, so let’s start simple by describing what states a shell job can be in.

Jobs States

Foreground Jobs

Up to now, you may have only worked with foreground jobs in the shell. Foreground jobs are ones that don’t return you to the command prompt until you exit out of the job or the job finishes.

If you run “ping yahoo.com” from the shell, the ping process will become a foreground job and will not return you to the command prompt until you stop it.

Just in case you didn’t know, you can stop the ping process by pressing Ctrl+c. Unlike in many applications that you may be used to, Ctrl+c does not mean “copy” in a Linux shell. Instead of copying text, this signals the foreground process to terminate by sending it the SIGINT (Signal Interrupt) signal.

An example of a foreground job that won’t return you to the command prompt until it is finished is the zip command. Imagine that you have a large amount of files inside a folder called “stuff”. If I run “zip -r stuff.zip stuff” and don’t know about job management, I won’t have access to the shell until zip finishes its task and returns me to the prompt.

Background Jobs

Putting a job in the background allows the job to run exactly as if it were in the foreground except that it does not receive user input. If a command produces output and is put in the background, it will still produce the output.

Why would putting a job in the background be helpful then you may wonder. There are a number of reasons, and they will become clear in the examples below.

Stopped Jobs

There is technically a third state that a job can be in. Jobs that aren’t in the foreground yet are also not in the background have been stopped.

Like background jobs, stopped jobs do not receive user input. Unlike background jobs, stopped jobs do not produce output. As a matter of fact, they don’t do anything. They have essentially been put on hold and won’t do any processing until they are resumed or killed.

Multitasking 101: Stopping and Resuming Processes

It may seem like stopped jobs are the odd man out, but they are actually the key to getting started with multitasking on the Linux shell. I use stopped jobs every day, and I couldn’t imagine using the shell without them.

Imagine that you are in your favorite shell text editor working on a program or modifying config files, and you can’t remember what the IP address you need is but you could easily look it up with dig. You could easily open up another shell connection and run the command, but it would be quicker to simply stop the editor, run the command quickly, copy the text, and bring your editor to the foreground.

The following is a sequence that illustrates how this example would look on your shell.

[chris@local ~]$ sudo vi /etc/httpd/conf/httpd.conf
[sudo] password for chris:
Ctrl+z
[1]+  Stopped          sudo vi /etc/httpd/conf/httpd.conf
[chris@local ~]$ dig +short chrisjean.com
96.125.165.12
[chris@local ~]$ fg %1

After running these commands, you will be back in your editor.

The key to stopping a running job is the Ctrl+z key combination. Again, some of you may be used to Ctrl+z as the shortcut to undo, but in the Linux shell, Ctrl+z sends the SIGTSTP (Signal Tty SToP) signal to the foreground job. When you press this key combination, the running program will be stopped and you will be returned to the command prompt.

You may notice the “%1” after fg. The “1″ is the job number that is to be brought to the foreground. The job number is listed in brackets when the job was stopped. These numbers are crucial when you have multiple jobs in stopped states. At any time, you can run “jobs” to get a list of the current jobs.

I stop jobs every day, and it greatly reduces the number of active shell sessions I have running. Sometimes I have stopped jobs that I set up ahead of time when I first access the shell. For example, I may open up a MySQL session, connect it to the database of the site I’m working on, and then stop it. Any time I need to run a query, I can load up the job by running “fg 1“, run the query, and then stop the job again.

Multitasking 202: Running Jobs in the Background

Stopping jobs is great, but what if we want to be returned to the command prompt while still allowing the job to run? Sending a job to the background will do exactly that.

As I talked about in my post about finding files, the locate command is extremely helpful. Sometimes you need to manually run the updatedb command in order for the locate database to update. The problem is that the updatedb command can take a very long time to run, sometimes it can take more than an hour.

I’ll use the example of the updatedb command to show how you can take a foreground process and put it in the background so that you can free up the shell.

[chris@local ~]$ sudo updatedb
[sudo] password for chris:
Ctrl+z
[1]+  Stopped            sudo updatedb
[chris@local ~]$ bg %1
[1]+ sudo updatedb &
[chris@local ~]$ 

There are a few items of note here.

1. I’m back at the command prompt and can do anything I’d like while the job works merrily away in the background.
2. Notice how I used “%1″ with bg just as I did with fg. Both commands accept a job number to tell it which job is to be sent to the foreground/background.
3. After running bg, a line similar to when I pressed Ctrl+z was printed showing the command and job number. However, there is also the addition of the “&” at the end. We’ll get into the reason for that in the next section.

When a process that is running in background finishes, you will receive a message the next time your command prompt refreshes. The prompt refreshes when you run another command or simply hit Enter to get a new prompt.

Continuing from the previous example, I receive output similar to the following when the updatedb command finishes:

[chris@local ~]$ 
[1]+  Done              sudo updatedb
[chris@local ~]$ 

As with all the other job-related output, you will be notified of the job number and the command.

Multitasking 303: Starting Jobs in the Background

Now that you see the value of having jobs run in the background, you might wonder if you can just start the command in the background to begin with. In fact, you can!

Remember that odd ampersand “&” after the command in the output of the bg command? The ampersand means that the job is running in the background. You can run a command with the ampersand at the end to automatically put the job in the background.

Going back to the updatedb example again, we could have simply run the following to have the job in the background and get our command prompt back immediately:

[chris@local ~]$ sudo date
...
[chris@local ~]$ sudo updatedb &
[sudo] password for chris:
[1] 15231
[chris@local ~]$ 

You might wonder what that completely unnecessary “sudo date” command is all about. Remember that when you send a job to the background, it cannot receive input from the user. Since sudo prompts for the password when it hasn’t been run recently, I ran a throwaway command before running the background sudo command to ensure that the password wouldn’t be prompted for. If you do send a sudo command to the background and sudo prompts for a password, you need to bring it to the foreground first to enter the password and then send it to the background again.

The output when sending the job directly to the background is different than running bg. Instead of telling you the command, it tells you the actual process ID. Every program that runs in Linux is a process and has a process ID. You can use these process IDs to kill a specific process, change the process’ execution priority, and isolate the resource utilization by that process.

Even though you’ve put a task in the background, you can still bring it to the foreground using the fg command. In the previous example, I can pull the background process to the foreground by running “fg %1“.

Multitasking Graduate School: Killing and Prioritizing Jobs

Now that you know the fundamentals of managing jobs, it’s time to get more advanced and tell you how to terminate jobs and modify their execution priority.

Terminating Jobs

Sometimes a process goes rogue and either won’t respond to input, is consuming massive amounts of resources, or both. At times like this, it’s very important to know how to kill a job which will cease the program’s execution and free up its used resources.

The command used to kill processes is appropriately called kill. The kill command can accept either process IDs or job numbers (the job number must be preceded by a percent sign, “%”).

If my updatedb job from before fails to respond, the job has a job number of “1″, and I want to tell the job to stop, I could run the following command:

[chris@local ~]$ sudo kill %1
[sudo] password for chris:
[chris@local ~]$ 

Notice that there isn’t any confirmation that the kill command was successful. You can check to ensure that the job has been terminated by running the “jobs” command and looking for that job’s listing.

By default, the kill command sends a SIGTERM (Signal Terminate) signal to the process; however, sometimes this is not enough to stop the execution of an out of control process. To shut down these processes, we need something more powerful.

[chris@local ~]$ sudo kill -9 %1
[sudo] password for chris:
[chris@local ~]$ 

This may not look much different, but the addition of the “-9″ option tells the command that things are serious now. This option causes a SIGKILL (Signal Kill) signal to be sent to the process. A kill signal cannot be intercepted by a process and causes the process to immediately terminate without allowing it to clean up after itself. It’s for this reason that you should only use SIGKILL after first sending the process a SIGTERM.

As mentioned before, you can also run a kill command on a process ID rather than a job number. If we wished to kill a process with a process ID of 12267, we can run the following command:

[chris@local ~]$ sudo kill 12267
[chris@local ~]$ 

Note that you don’t have to always run the kill command with sudo, you just have to ensure that whatever privilege level you are running the kill command at has the ability to control that process. In other words, if you are trying to kill a process run with sudo, you’ll need to use sudo to kill it as well.

Prioritizing Jobs

Linux does a great job of prioritizing tasks automatically; however, what Linux thinks is best isn’t always what you want. Basically, Linux always wants to make a process finish as soon as possible, which means that you get a nice bit of lag when you try to do your work as a CPU or disk hungry process churns away. Fortunately, there are tools that will let you tell Linux what you want.

First, let’s deal with managing a process’ use of CPU time. If you are going to run a command that will eat up large amounts of CPU cycles and you want to make sure that this process doesn’t keep you from doing your work, you can run the command with nice to give the command a lower-than-default priority.

[chris@local ~]$ sudo date
[sudo] password for chris:
Fri Mar  6 10:55:11 CST 2009
[chris@local ~]$ sudo nice updatedb &
[1] 16312
[chris@local ~]$ 

As before, a junk sudo command was run first to take care of the password prompt.

Running the command in nice lowers the processes CPU priority and will result in smoother CPU performance for your other applications.

What if you already ran a command whose process is now greedily chewing away at your CPU resources? That’s where the renice command comes into play.

With renice, you can change the CPU priority of an existing process.

[chris@local ~]$ sudo date
[sudo] password for chris:
Fri Mar  6 10:56:25 CST 2009
[chris@local ~]$ sudo updatedb &
[1] 16324
[chris@local ~]$ sudo renice 10 16324
16324: old priority 0, new priority 10
[chris@local ~]$ 

The renice command accepts a priority as the first argument and the process ID as the second argument. The priority argument has a range from -20 to 19 where -20 is the highest priority and 19 is the lowest. In other words, a higher numeric value priority results in lower CPU priority.

That takes care of the CPU, but admittedly, the updatedb command does more damage to the IO or disk traffic than it does to the processor. How can we reign in a process’ use of the disk so that performance for the rest of the system improves? For that, we use the ionice command.

With the ionice command, you can set the IO priority for a process to one of three classes: Idle (3), Best Effort (2), and Real Time (1). The Idle class means that the process will only be able to read and write to the disk when all other processes are not using the disk. The Best Effort class is the default and has eight different priority levels from 0 (top priority) to 7 (lowest priority). The Real Time class results in the process having first access to the disk irregardless of other process and should never be used unless you know what you are doing.

If we wish to run the updatedb process in the background with an Idle IO class priority, we can run the following:

[chris@local ~]$ sudo date
[sudo] password for chris:
Fri Mar  6 11:02:43 CST 2009
[chris@local ~]$ sudo updatedb &
[1] 16324
[chris@local ~]$ sudo ionice -c3 -p16324
[chris@local ~]$ 

If we’d rather just lower the Best Effort class priority (defaults to 4) for the command so the process isn’t limited to idle IO periods, we can run the following:

[chris@local ~]$ sudo date
[sudo] password for chris:
Fri Mar  6 11:07:12 CST 2009
[chris@local ~]$ sudo updatedb &
[1] 16324
[chris@local ~]$ sudo ionice -c2 -n7 -p16324
[chris@local ~]$ 

Again, the Real Time class should not be used as it can prevent you from being able to interact with your system.

You may wonder where you can get the process ID if you don’t know it, can’t remember it, or didn’t start the process (an automated script may have launched it). You can find process IDs with the ps command.

For example, if I had an updatedb program running in the background, and I wanted to find its process ID, I can run the following:

[chris@local ~]$ ps -C updatedb
PID TTY          TIME CMD
4234 ?        00:00:42 updatedb

This tells me that the process’ process ID (PID) is 4234.

Another useful command is “ps -u user” (replace “user” with your username) which lists all processes your user has access to. Basically, these are processes that your user or some process using your user’s permissions started.

Another really helpful use of ps is “ps aux“. This command will list all the system’s processes, which user owns the process, and even detail the resource usage of the process. This is very helpful for finding processes that you don’t know enough about to find using other methods.

Mutitasking Doctorate: Using Job Management in Scripts

Keep in mind that many scripts execute from inside shell environments. This means that you can manage the script’s job using the tools I’ve given you here.

If you look in the /etc/cron.daily folder, you’ll find many scripts that execute once a day. These scripts can create a huge load on a system while they execute. If around the clock performance is crucial, you can prioritize the shells that the scripts run in which causes all the commands run inside that shell to follow the same prioritization rules.

This all sounds very complex, but it’s actually very simple.

Let’s open up the /etc/cron.daily/mlocate file (pick another file if you don’t have this one). At the top, you should see a line like the following:

This shebang line tells Linux what interpreter is used to execute the script. My example script uses the bash shell as the interpreter. Other scripts may use the sh shell instead of the bash shell, but that won’t make any difference.

Below the first line, put the following:

The $$ variable is a special variable that contains the shell’s own process ID. The two commands above cause the CPU priority to lower to 10 and changes the IO priority to class 3 (Idle).

Add this type of customization to your resource hungry scripts, and your system will perform more smoothly around the clock. Keep in mind that this also can result in the scripts taking much longer to complete.

You can use those same commands above using the $$ variable to change the CPU and IO priority of shells that you open in order to lower the impact you have while working on the system. This can be a smart thing to do if you need to compile software or run other intensive processes on production machines.

Important Commands

Here’s a list of all the commands, key combos, and other important bits that I went over with a brief description for each. It’s like the word list at the end of children’s books.

• fg – Brings a stopped or background job to the foreground
• bg – Sends a stopped job to the background
• jobs – Lists the current jobs and their corresponding job number
• Ctrl+c – Key combination that sends a SIGINT signal to the foreground process. Use this to close processes that don’t stop (such as ping or top) or terminate a process that no longer responds or is no longer needed.
• Ctrl+z – Key combination that sends the SIGTSTP signal to the foreground process. Use this to send a foreground process to a stopped state.
• & – Use an ampersand at the end of a command to send the resulting job to the background immediately
• kill – Sends signals to processes based on process ID or job number. Useful to send SIGTERM or SIGKILL signals to forcibly stop processes.
• nice – Runs the given command with a lower-than-default CPU priority
• renice – Changes an existing process’ CPU priority
• ionice – Changes an existing process’ IO priority

Closing Thoughts

Frankly, I didn’t expect to make such a long post. If this post has helped you, please leave a comment and let me know what you thought.

I’d greatly appreciate it if you’d share the post by using one of the icons below. Or you can click here to Digg this post.

Many of you fellow Ubuntu users will be familiar with the “Search for Files” tool that allows you to look for files. As is true with most things in Linux, there are great desktop tools, but more power can be found in Terminal than any streamlined desktop tool can match.

Today, I’d like to introduce you to a few tools that can turn a chore of finding files into an easy process.

locate

The first tool that you should become comfortable with is also one of the simplest. The locate program works in a similar fashion to most graphical search tools.

Using

Let’s say that you’ve read up on how to modify X to provide enhanced video performance. The document that you are reading says to modify the xorg.conf file, but the document doesn’t say where that is and you don’t know how to find it. You can use locate to find it for you quickly:

Most likely, your system will return a number of results. For example, on one of my systems, I receive the following output:

It’s clear that /etc/X11/xorg.conf is the file we’re looking for, but what if dozens or even hundreds of results were returned? How could you manage all of that? If you know a little bit about regular expressions (every Linux user really should learn the basics), you can easily use regular expressions to search for directories or files. For example, we can search for an exact match for xorg.conf with the following command:

locate -r '/xorg.conf$'

If you want to look for files without case sensitivity, you can use the ‘-i’ option. For example:

locate -i 'monthly report'

Something that may confuse you when you first start using locate is that it not only returns matches, but if the match is a directory, it also lists all of the directories and files under that directory. To stop this behavior, use the ‘-b’ option. For example:

This will return a much shorter list of results than if you did not include the ‘-b’ option and all the matches will contain the search term as the final portion of the path.

Important Note

The locate command relies on a slocate database to function. This database caches the file system directory and file names and allows the locate command to operate in moments rather than taking long periods of time to search the entire file system.

This ability doesn’t come without cost though. By default, your system will run a command to update this database once a day at 6:25am. This command will update the database with any changes made to the file system since the last update.

If you’ve made changes recently, such as added or removed large amounts of files and directories, and you want to use the locate command to search through these new directories and files, you can manually update the slocate database at any time with the following command:

Depending on the volume of changes made since the slocate database was last updated, this could take a few seconds to a few minutes to update. Once it’s done, you’re ready to search again with locate.

which

locate is great for searching the entire file system. However, sometimes you just want to find where a command lives. That’s where which comes into play.

which only does one thing, but it does it well. Given the name of a command, which will tell you where the file for that command resides in the file system.

Let’s imagine that we want to find out where there firefox command resides. We can simply run the following to find out:

On my system, and most likely on your’s as well, which will tell us that it is located at /usr/bin/firefox.

Why would you ever want to search specifically for a command? Imagine that you have some code that needs to replace an old command. You really need to remove the the old one (I usually rename it to the same name followed by “.bak” to keep it around in case I need it) first before putting the new one in its place. which will help you find the file quickly and easily so you don’t have to search everywhere to find it.

whereis

The whereis command is similar to which but goes one step further. Not only does it tell you where the executable file is located at, but it will also locate the source, man page, and other associated directories as well.

Keeping with the which example, let’s ask whereis about the firefox command:

On my system, I get the following output:

This helped me find /usr/lib/firefox/plugins earlier when working on the solution for Flash. That location is where the global add-ons for Firefox are stored.

find

find is the ultimate search tool, the Swiss Army knife of Linux search if you will. find can do everything from search for files based on last modified times, owners and groups, permissions, type (file, directory, symbolic link, etc), whether the file is readable/writable/executable by the current user, file size, whether the file or directory is empty, and much more. You can also search for matches by regular expression.

When you use this tool in combination with the xargs command (which I’ll have to cover some other time), you can do truly amazing things on the command line. Imagine being able to recursively search through the current directory for files ending with “.bak” that were last modified more than 100 days ago and then deleting them with one command:

find . -name '*.bak' -type f -mtime +100 -print0 | xargs -0 /bin/rm -f

It will look very complex at first, but when you master the art of using find, it will become second nature. Soon, you’ll wonder how you ever searched for things without it.

To be frank, there is much more to find than I am willing to cover and there are many, many great introductions to the tool that do a better job than I would. So, I’ll simply point you to three great references that can be used to get going with find.

• Example uses of the Linux Command find
• (I apologize for the horrible colors in advance) Unix/Linux “find” Command Tutorial
• Using the find command

You just got your new CentOS dedicated server, and you notice that times in your logs aren’t quite right. You check the time from the command line (run “date”), and find that the timezone is set to US Eastern or some other timezone. How do you get this changed?

Unfortunately, this is not an easy thing to figure out. Fortunately though, it’s not hard to do with the right directions.

Please note that you have to have root access to be able to make the changes detailed below.

There are a series of time zone files located at /usr/share/zoneinfo. Select the appropriate named timezone for your location. For my location, Oklahoma, USA, I actually have two that I can select from: America/Chicago or US/Central. Make note of the appropriate folder and file for your timezone.

The active timezone used on your system is in the /etc/localtime file. The default will vary depending on your server host but often seems to be EST or EDT (depending on the time of year you are checking). We simply need to replace this file with the file we selected in the previous step.

Now, I say replace, but I actually recommend creating a link to the pertinent file rather than actually making a copy. Let me explain the reasoning for this quickly.

A crucial part of the timezone shift calculations is the daylight savings calculations. Many people don’t realize this, but the daylight savings days were changed in 2007. When this change happened, all servers needed to be updated with new zoneinfo files.

If your server has an automated process to update these files when daylight savings calculations change, your /usr/share/zoneinfo files will be updated but the /etc/localtime file will not be. So, if you simply made a copy of the file, you’ll have to know when these updates come out and manually copy the file over. If you create a link, everything will take care of itself.

Enough of the banter. On with the show.

First, make a backup of the existing localtime file. It’s always good practice to make backups of original config files.

Next, create the link:

Make sure to replace “America/Chicago” with the directory (if your zone has one) and filename of the timezone you wish to use.

Now you just need to test your change. Run “date” from the command line, and ensure that the appropriate time, date, and timezone are reported.

In many of my Tips ‘n Tricks or tutorial posts, I have commands that involve editing/creating files in Vi. For those that have never used Vi, it can be a daunting experience trying Vi for the first time.

There are many holy wars fought over the best editor to use from the *nix command line. I won’t claim that Vi is the best; rather, Vi is simply what I’m used to, it’s powerful, and it is always available on servers that I work on.

In order to best help those that don’t know how to use Vi but are interested in learning, this will serve as a very brief tutorial on how to get started.

Vi and Vim

The original Vi first saw development in 1976. Thus, it’s an editor that is more than thirty years old. It is older than Linux, and it is older than me. The fact that it is still so prevalent, is testament to the power and smart design that can be found in Vi. That said, I don’t technically use Vi; rather, I use Vim: Vi IMproved.

Vim has many features that extend the original functionality of Vi. By far, my two favorite features are unlimited undo (Vi only supports undo last command) and syntax highlighting. When you see images on my site that have syntax highlighted code, those are screenshots of Vim running in Terminal.

Many recent distros have Vim installed by default. In addition, most distros that I’ve worked with also have Vim in place of Vi. You can check this by running “vi” from the command line. If Vim is running in place of Vi, you’ll see a greeting screen that says “VIM – Vi IMproved” at the top.

Installing Vim

If you run “vim” from the command line and receive an error such as “-bash: vim: command not found”, then you don’t have Vim. Since features that Vim has are greatly beneficial, I recommend installing it if you don’t have it. In addition to the advanced features, Vim has enhanced arrow key support which most new users will appreciate as they get accustomed to the unintuitive nature of Vim.

Vim can be installed quickly using your distro’s package manger. If you are using Ubuntu or another APT distro, you can simply run “sudo apt-get install vim”. On CentOS and other RPM-based distros, gain access to a root shell and run “yum install vim”.

Once you have Vim installed, you will probably want to use Vim whenever you type “vi file”. You can do this easily by setting up an alias. Modify the /etc/bashrc file with whatever editor you are currently comfortable with and add the following line:

Save the file and exit the editor. Once you log into your shell again, this alias will be in place and all commands to vi will be routed to vim.

Getting Started With Vi

As I’m sure you’ve seen many times before, you can open a file for editing in Vi by running “vi filename”. This will indeed load the file for editing, but how do we actually modify the file?

Modifying Text

Vi is a modal editor. This means that the mode you are in when you first open the file is not an editing mode; rather, it’s more of a command mode where you can issue commands to Vi. In order to actually edit the file, you have to enter an editing mode.

Most of the time, the insert edit mode will be used. To enter the insert edit mode, simply press the ‘i’ key. You’ll notice that the last line of the interface will now display “– INSERT –” to indicate that we’ve changed into the insert mode.

Now that you are in the insert mode, you can simply delete and insert text as if you were in a normal editor. Once you are done making changes, hit the ESC key to exit out of the insert mode and return to the command mode, commonly referred to as “normal mode”.

Saving the File

Now that the file has been modified, we need to save our changes.

In order to save the file, you first need to be in normal mode (press ESC). You can then save the file by using the ‘:w’ command. This means that you need to press the ‘:’ key, the ‘w’ key, and Enter.

This command simply tells Vi to write the contents to the file.

If you wish to save the contents as a different file, give the write file command a file name. For example: “:w filename”.

Exiting Vi

Okay, we’ve opened a file, modified it, and saved it. Now we just need to figure out how we get out of Vi.

Similar to the write command, we just need to issue the quit command. This is done with ‘:q’.

Since Vi supports chaining commands, you can save the file and exit Vi with the ‘:wq’ command. Notice that ‘:qw’ will result in “E492: not an editor command: qw”. So, make sure that you put commands together in the correct order. Another command, ‘:x’, does the same thing as ‘:wq’.

More than Basic Editing

Frankly, this is just scratching the surface of what Vi is and how to use it. It will be enough to get most people started however.

I will revist the topic of Vi to give more details of how to do advanced uses and how to configure options to get the most out of it. For now, I have some links if you’d like to delve into the topic deeper.

• Vim Documentation
• Introduction to Advanced Vi
• Vi Cheatsheet
• Vi Lovers’ Homepage
• Efficient Editing with Vim
• Search and Replace in Vi

Here’s a quick tip that will help you work with multiple zip files on the command line.

If you are in a folder and have three zip files in it (a.zip, b.zip, c.zip) that you want to unzip, “no problem,” you think, “I can take care of that with one command.” So, you quickly run the following:

However, rather than having the unzip program nicely unzip each file one after another, you receive the following:

I’m sure that this is not what you were expecting. I know I certainly wasn’t expecting this when I first tried it. However, this problem can help us understand more of how the command line works.

The Problem

Whenever you use a wildcard (*), the shell itself will expand that and pass the results to the program rather than the program handling the expansion itself. That means that our previous command was actually expanded to the following before being executed:

Again, this may not look odd since other programs (such as mkdir, chmod, etc) can take one or more arguments and repeat the process for each. However, unzip is different and actually has use for the additional arguments. If you specify additional arguments after the zip file name, unzip will try to only extract those specific files from the archive rather than all the files.

The previous command told unzip that it should extract b.zip and c.zip from inside the a.zip archive. Since those files don’t exist inside a.zip, we get the nasty output seen earlier.

You might think that you will have to manually unzip each archive one at a time, but you’d be wrong.

The Solution

Just because the shell expands out wildcard characters automatically doesn’t mean that programs can’t as well. The simple solution to this problem is to quote the argument to prevent the shell from interpreting it:

That’s all there is to it. I’ve seen solutions from coding loops in bash script to running find combined with xargs, but none of that is necessary. Simply let unzip itself take care of things for you.

I touched on this topic in my First Day with Ubuntu at the Office post under Accessing Remove File Systems; however, I thought that this deserved its own topic.

For those that don’t know SSH stands for Secure SHell. In very non-technical terms, SSH allows you to access a system running a SSH server over a network. This other system can be another computer in your home or a remote system on the other side of the planet. SSH will allow you to connect to that other system and communicate with it securely. All the data passed back and forth is encrypted, so you don’t have to worry about people sniffing your passwords or valuable data.

You may be asking what all of that means to you. What it means is that you can connect to and control a computer that is somewhere else with the computer that is sitting right in front of you. If you have a remote webserver running Linux, SSH will let you install software, edit files, change the server configuration, access the database, and more. Did you forget a file on your computer at home? No problem, just SSH into your home computer and send the file to your email account or copy it directly to your office computer.

Hopefully those quick examples of what you can do is enough to whet your appetite.

Making SSH Work for You

When working on a Linux system, connecting to other Linux systems via SSH becomes so easy. Everything you need is bundled directly into the OS. All you need to do is load up the Terminal and run something that looks like the following:

After supplying the password, you’re connected.

So, if this is so easy, you might wonder what could make this any simpler. While the process I just described is definitely easy, there is always some server out there that requires a ssh command that is just difficult enough to be annoying. For example, imagine having to type the following each time you wanted to connect:

While it’s possible to remember this, who wants to?

Host-Based Config Options

Fortunately, ssh has a way to store options for each host.

This will load up the config file for editing in Gedit.

The format for setting options is as follows:

Using the config file, you can lose all those additional options and quickly connect by just using the following command:

Doesn’t that look a lot better?

Take the previous nasty example I provided. I can add the following to my config file for that specific host:

Host locutus.borg.domain.com
    User really_long_username
    Port 2222
    Protocol 2
    Cipher blowfish-cbc,aes256-cbc

After adding this to my ~/.ssh/config file, I can now connect with:

Easy cheesy.

There are a large variety of options available for the host-based config. For a full list of these options, run “man ssh_config” from Terminal.

Creating Hostname Aliases

Given the previous example, the hostname is locutus.borg.domain.com. That’s quite a bit to type, and it isn’t easy to remember how to spell locutus properly. Fortunately, the config file comes to the rescue again.

We can update our previous host definition to the following:

Host locutus.borg.domain.com locutus borg loc domain.com 192.168.1.105
    Hostname locutus.borg.domain.com
    User really_long_username
    Port 2222
    Protocol 2
    Cipher blowfish-cbc,aes256-cbc

Now I can easily connect to the system using any of the following commands:

Simply set up all the alias names you’d like to have after “Host” while seperating each one with a space. I always keep the original hostname in the list and add an additional alias to is easy to remember and type.

Keep the Connection Alive

Due to issues either caused by certain routers’ NAT firewalls or due to odd server configurations, I have found that my SSH connections will die if I leave then idle for too long. This is very annoying as it leaves me with a frozen session that I can’t do anything with. Fortunately, config comes to the rescue again. On each and every one of my host options in my config files, I have a ServerAliveInterval option that tells ssh to send a small keep-alive packet to the server at an interval that I specify.

For example:

Host domain.com
    User chris
    ServerAliveInterval 240

This tells ssh to send a keep-alive packet every 240 seconds, or four minutes. Now the only thing that terminates my connection is an actual loss of internet connection.

If you find that your connection still drops when idle, lower the number until it stabilizes. Even though the packet is small, you don’t want to send it too much. So, only lower the value to below a minute if you must.

Using Keypairs to Increase Security

SSH has the ability to use security keypairs to authenticate your session with the server. A security key can replace the need to supply your password when connecting to your server.

If you have access to modify your server’s SSH daemon settings, you can use your authentication key to disable password authentication. This means that an attacker would have to break your key’s encryption in order to connect rather than just having to guess your password.

Another great feature of keys is that you can use the same key for multiple server accounts. Since the private part of your key only resides on your local system, using the same key for multiple systems doesn’t decrease security in the same way that using the same password on multiple servers can. For example, if someone broke into one of your server accounts, they could get your public key, but this key will only give them the ability to give your key the ability to authenticate on other servers rather than giving them the ability to authenticate on the rest of your’s.

What This Means for Shared Servers

If your server is a shared host where you don’t have access to root, you typically can’t disable password authentication; however, keys can still greatly increase your security. Since you won’t need to supply your user’s password each time you connect since the key is doing the authentication, you can change your user’s password to something extremely strong and complex. This will make it very difficult for anyone to break into your account using brute-force methods.

Generating a SSH Authentication Key

Generating a key is extremely simple. Ubuntu, and most major distributions, have all the software you need already installed. Simply run the following command in Terminal:

Accept the defaults for all the options except the password.

Supplying the password is optional. If you simply press enter without supplying a password, your key won’t require a password to use. Let me share some information about what having a password on your key means to you.

• Unlike some password protection methods where the password is easily cracked, your key’s password is an integral part of the key. Without your password, the key is useless.
• If your key does not have a password, anyone who manages to gain access to your private key will have the ability to connect to every server your key is authenticated for without any password or other authentication required.
• On many systems, you will have to supply the password each and every time the key is used. Fortunately, for us Ubuntu users, we can simply opt to have the system remember the password for us as long as we are logged into our desktop. I’m sure that this feature is present on other systems, such as Linux Mint, but I haven’t tested which ones have this feature yet.

As you can see, there are great benefits to using a password on your key. There really aren’t any good reasons to not put a password on the key since Ubuntu will remember it for you.

By now, you should have generated your key. If you accepted the default location, you should have two new files in your ~/.ssh directory: id_dsa and id_dsa.pub.

The ~/.ssh/id_dsa file is your private key. This file should remain safe and secure on your system.

The ~/.ssh/id_dsa.pub file is your public key. The contents of this file will be used to authenticate your key with servers.

The permissions on the ~/.ssh/id_dsa file should be set up correctly by ssh-keygen. I like to make sure however as the wrong permission settings can cause the key to be ignored when authenticating. You can set the correct permissions by running:

Now it’s time to set up the server.

Setting up Your Server

Now you need to log into the server you want to use your key to authenticate with.

You need to create a ~/.ssh folder if one doesn’t already exist:

The -p will prevent an error from being thrown if the folder already exists.

Exit out of your server connection. We’re going to do a slick command from our Ubuntu Terminal to add our public key to the server.

From our desktop’s Terminal (not the server’s command line), run the following:

Replace hostname with the hostname that you use to connect to your server with. I’m assuming that you followed my instructions earlier and set up your ~/.ssh/config file to not require anything more than a hostname to set all the options needed to connect. If you haven’t done this, make sure you specify the port number (if other than 22), username, and any other necessary options to make the connection.

Testing

Now we just need to connect to the server again and see if it asks for the password.

If you are asked for a password, your authentication key was either not recognized, had improper permissions, or was not able to match with the server’s public key setting. A good way of checking on a potential problem is to run:

This will cause a large amount of verbose debug data to be printed during the connection phase. If the key is used properly, you should see the lines listed:

If you don’t see that the server can see the publickey authentication method or if that method isn’t the first one, your server isn’t set up to do key authentication. Contact your hosting provider and ask them if they can add this as it will increase server security.

If you can’t figure out the problem, please let me know. You can send me the output of your “ssh -v hostname” command via the Contact page, and I’ll do my best to help you out.

Conclusion

I hope that I helped you use ssh more efficiently and make your servers more secure at the same time. However, I’ve just scratched the surface on this topic. There are dozens of ways to use SSH in ways that I haven’t touched on.

Here are just a few additional things that you can do with SSH:

• Log into a visual desktop remotely by tunneling X
• Secure FTP alternative: use scp to easily transfer files between systems
• Mount remote file systems as local mount points
• Browse securely on insecure networks or bypass network restrictions to browse freely
• Easily run a command on remote servers directly. This is great for scripting status programs.
• And many more possibilities

I’m sure that I’ll touch on some of these subjects later on. If you’d like me to get to specific ones ASAP, please leave a coment and let me know what you’d like to see.