I recently did some work on an Ubuntu server with a compromised PHP site that was sending out massive amounts of spam email messages. After removing the offending scripts, I needed to clear out the mail queue to ensure that the server was not attempting to send backlogged spam messages.

The server in question is running Postfix as the MTA (Mail Transfer Agent) rather than Sendmail. Fortunately, clearing the entire queue was as easy as running the following:

[user@server ~]$ sudo postsuper -d ALL
postsuper: Deleted: 69967 messages
[user@server ~]$ 

If your server runs Sendmail, you will have to manually clear the mail queue by removing all the files at /var/spool/mqueue/. For example:

[user@server ~]$ sudo rm -f /var/spool/mqueue/*
[user@server ~]$ 

Ubuntu’s time and date indicator in Unity is a bit blah for my tastes. With the default settings on my system, it currently says “7:39 AM”. That’s accurate, but it could be much more helpful.

You can load up the Time & Date application from the dash, click the Clock tab, and customize the appearance a bit. However, the options offered in the application are very limited compared to what the indicator is capable of supporting. Sure, you can change it to the following with a few checkboxes “Thu Mar 26 2015 7:39:11 AM” or toggle it to a 24-hour time, but it really doesn’t look that nice to me.

Fortunately, the indicator can be set to a custom mode which supports all the options offered by the strftime function.

To set up your custom indicator, first make sure that you have the clock indicator enabled:

[chris@work ~]$ gsettings set com.canonical.indicator.datetime show-clock true

Next, set the indicator to use the custom format:

[chris@work ~]$ gsettings set com.canonical.indicator.datetime time-format custom

Finally, set your indicator format:

[chris@work ~]$ gsettings set com.canonical.indicator.datetime custom-time-format "%A, %B %e  %l:%M %P"

This changes my indicator to say “Thursday, March 26  7:40 am”.

The different options that can be used in the format string can be found on the strftime function’s man page.

I’ve been tinkering around with LXC containers on my Ubuntu systems. While setting up my user for unprivileged usage, I ran into the following problem:

[chris@server ~]$ lxc-attach -n u1
lxc_container: cgmanager.c: cgm_attach: 1291 Failed to get cgroup for controller all
lxc_container: attach.c: lxc_attach: 847 error communicating with child process
[chris@server ~]$ 

After a bit of tinkering around, I found that there is a bug that is causing this issue. Adding the LXC daily build PPA and updating fixed the issue for me. I did this as follows:

[chris@server ~]$ sudo apt-add-repository ppa:ubuntu-lxc/daily
 Those are daily builds of the master branch of git://github.com/lxc/lxc for all supported versions of Ubuntu.

Required or recommended dependencies are also backported from current Ubuntu and included in this PPA to provide an identical LXC experience on all supported Ubuntu releases.
 More info: https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/daily
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpovs0swu3/secring.gpg' created
gpg: keyring `/tmp/tmpovs0swu3/pubring.gpg' created
gpg: requesting key 7635B973 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpovs0swu3/trustdb.gpg: trustdb created
gpg: key 7635B973: public key "Launchpad PPA for Ubuntu LXC team" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
[chris@server ~]$ sudo aptitude update && sudo aptitude full-upgrade -y
...
The following NEW packages will be installed:
  lxcfs{a}
The following packages will be upgraded:
  liblxc1 lxc lxc-templates python3-lxc
4 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 794 kB of archives. After unpacking 12.3 kB will be used.
Writing extended state information...
...
Building tag database...
[chris@server ~]$ lxc-attach -n u1
root@u1:/# exit

Hopefully this fix will make it out to the default repos soon, as I don’t typically like to use daily repos if I can avoid them.

For the past few weeks I’ve had problems logging into servers using my SSH keys from Ubuntu desktops. The following would happen each time:

[chris@work ~]$ ssh server
Agent admitted failure to sign using the key.
Permission denied (publickey).
[chris@work ~]$ 

Oddly, this only happened from my Ubuntu desktop systems. My Ubuntu servers had no issue connecting.

After some digging, I found out that issues with the gnome-keyring were at fault. gnome-keyring doesn’t always handle specific formats of SSH keys correctly. Unfortunately, gnome-keyring was trying to handle all SSH key usage, preventing the keys from working.

If you are having this issue, you can confirm that gnome-keyring is at fault on your system by added SSH_AUTH_SOCK=0 in front of the ssh command as follows:

[chris@work ~]$ SSH_AUTH_SOCK=0 ssh server
Welcome to Ubuntu 14.10 (GNU/Linux 3.13.0-37-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

0 packages can be updated.
0 updates are security updates.

Last login: Wed Mar 11 11:49:50 2015 from host
[chris@work ~]$ 

If the connection works, then you likely have the same problem that I had. If the connection does not work, please read the Fixing other sources of agent failure section below for further ideas to fix your problem.

Fixing gnome-keyring interference

In order to fix the gnome-keyring interference, the “SSH Key Agent” will have to be disabled from the startup applications. In Unity’s dash, search for “startup” and select “Startup Applications”. This lists the programs that automatically run when your user logs in.

If you see “SSH Key Agent”, uncheck the box and reboot. After you log back in, your ssh connections should work normally.

If you do not see “SSH Key Agent”, you are reading this before the update has been released for your version of Ubuntu. You will first have to update your system to install pre-release updates. To do this, open up Unity’s dash, search for “software”, and select “Software & Updates”. Select the “Updates” tab. Enable the “Pre-released updates” option, provide your user password if requested, and click the “Close” button. If a dialog pops up saying “the information about available software is out-of-date”, click the “Reload” button and wait for the update to finish. Open up Unity’s dash, search for “software” again, and select “Software Updater”. Confirm any dialogs and provide your password if requested. This will install the updates that you just made available. After this is done, reboot your system.

After the reboot, open Unity’s dash, search for “startup” and select “Startup Applications”. Uncheck the option for “SSH Key Agent” and reboot. After you log back in, your ssh connections should work normally.

Note: Details about this bug and the update allowing for disabling the “SSH Key Agent” can be found in Bug #1387303 in Ubuntu’s bug tracker.

Fixing other sources of agent failure

If using SSH_AUTH_SOCK=0 in front of the ssh command does not fix your issue, your system could either not have an SSH agent available that provides the keys to the ssh process or the agent is not loaded with your keys.

The most simple thing to check is if running ssh-add can fix the issue. For example:

[chris@work ~]$ ssh-add
Identity added: /home/chris/.ssh/id_rsa (/home/chris/.ssh/id_rsa)
[chris@work ~]$ 

This means that your SSH agent was running, but it did not have your keys loaded. After this, see if you are now able to login successfully.

Another possibility is the following:

[chris@work ~]$ ssh-add
Could not open a connection to your authentication agent.
[chris@work ~]$ 

This means that you do not have an SSH agent running. To fix this, do the following:

[chris@work ~]$ eval "$(ssh-agent -s)"
Agent pid 9267
[chris@work ~]$ ssh-add
Identity added: /home/chris/.ssh/id_rsa (/home/chris/.ssh/id_rsa)
[chris@work ~]$ 

After this, you should be able to ssh into servers that you have valid keys for.

If you see the following:

[chris@work ~]$ ssh server
Permission denied (publickey).
[chris@work ~]$ 

This means that your keys have loaded, but your keys are not authorized to access the server.

I recently set up an IPv6 interface on a remote Ubuntu server. In order to finish this change, I needed to restart the interface. A while back, I would have used sudo service networking restart, but that doesn’t work very well any more:

[chris@server ~]$ sudo service networking restart
stop: Job failed while stopping
start: Job is already running: networking
[chris@server ~]$ 

After doing some more reading, I found that restarting the networking service can have some bad side effects in Ubuntu, so they more or less prevented this from working. The correct way is to manually bring the interface that needs to be restarted down and back up.

The interface that I’m working with is named eth0. Your interface could be named eth1, em1, or many other names. So, when following my examples, make sure to replace eth0 with the name of your interface.

If the /etc/network/interfaces file was updated when making your network changes, you can run the following to run a basic syntax check on the file. It won’t tell you if you made a mistake on the IP address or other configuration details specific to your network setup, but it can catch simple syntax errors.

[chris@server ~]$ sudo ifup -n eth0
ifup: interface eth0 already configured
[chris@server ~]$ 

The -n tells ifup to not actually take any actions, thus allowing for a quick syntax check of the /etc/network/interfaces file. The “interface eth0 already configured” message indicates that no configuration issues were found. The following shows what happens when a configuration error was found.

[chris@server ~]$ sudo ifup -n eth0
/etc/network/interfaces:20: unknown address type
ifup: couldn't read interfaces file "/etc/network/interfaces"
[chris@server ~]$ 

This indicates that a problem was found on line 20 of the configuration file. In this specific instance, I accidentally put “int6” instead of “inet6”.

Now that we know that the config file doesn’t have any syntax issues, the following one-liner takes the eth0 interface down and then back up:

[chris@server ~]$ sudo ifdown eth0 && sudo ifup eth0
Waiting for DAD... Done
[chris@server ~]$ 

And with that, the interface now has the new config.

The Heartbleed bug (more properly known as CVE-2014-0160) is very troubling. In one day, much faith in internet security was shaken. While there is nothing that can be done to fix the last two years that the bug has been in the wild, patching servers ASAP and getting newly generated SSL certs can ensure that the sites we control no longer add to the problem.

You can verify that your site is compromised by using this tool. Once following the instructions here, you can use the tool to verify that the fix is successful.

Ubuntu’s official security notice to heartbleed can be found here. The notice provides the version number of the patched openssl version. To ensure that Ubuntu is patched against the issue, the libssl1.0.0 package is the OpenSSL library that should be installed. The patched version depends on the version of Ubuntu:

• 13.10 — 1.0.1e-3ubuntu1.2
• 12.10 — 1.0.1c-3ubuntu2.7
• 12.04 — 1.0.1-4ubuntu5.12

Note that only supported versions received the update. 10.04 isn’t listed because it’s version of OpenSSL predates the bug. The absence of 13.04 in the list seems a bit odd, but it reached end of life support on January 27, 2014. You can see a list of Ubuntu’s versions and their support terms here.

If your version isn’t listed above, you need to upgrade to one of the supported versions ASAP. I don’t recommend upgrading to 12.10, as it reached end of life at the end of this month (given that some of my servers are running this version, I know what I’m going to be working on). You should either run a current LTS version (such as 12.04) or stay on the current version (currently 13.10).

Patching this bug in Ubuntu is quite easy:

[gaarai@linode ~]$ sudo apt-get update
Ign http://us.archive.ubuntu.com quantal InRelease
Get:1 http://ftp.osuosl.org precise InRelease [2,506 B]
...
Ign http://us.archive.ubuntu.com quantal-backports/restricted Translation-en_US
Ign http://us.archive.ubuntu.com quantal-backports/universe Translation-en_US
Fetched 2,758 kB in 10s (264 kB/s)
Reading package lists... Done
[gaarai@linode ~]$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  libssl-dev libssl-doc libssl1.0.0 openssl
...
Current status: 0 updates [-1].
[gaarai@linode ~]$ dpkg-query -l libssl1.0.0
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-==============================
ii  libssl1.0.0:amd64         1.0.1c-3ubuntu2.7 amd64             SSL shared libraries
[gaarai@linode ~]$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.10
Release:	12.10
Codename:	quantal
[gaarai@linode ~]$ 

As can be seen from the output, the system is running version 1.0.1c-ubuntu2.7. Since the server is running Ubuntu 12.10, this is the patched version of the library.

One of my Ubuntu systems would pop up the following message multiple times after logging in:

System program problem detected

Do you want to report the problem now?

It was really annoying as simply closing the messages would cause them to return after a reboot.

This was happening due to a number of files in the /var/crash/ directory. After removing these, the problem went away. I did so from the command line as follows:

[chris@home ~]$ sudo rm /var/crash/*
[chris@home ~]$ 

While the idea of a crash report helping fix problems is nice, I really question whether they are that valuable. Personally, I’d rather never have to deal with them.

To ensure that I never have to deal with them again, I disabled the apport service that is responsible for generating these crash reports.

To disable the service, first edit the /etc/default/apport file:

[chris@home ~]$ gksu gedit /etc/default/apport

# set this to 0 to disable apport, or to 1 to enable it
# you can temporarily override this with
# sudo service apport start force_start=1
enabled=1

Then change enabled=1 to enabled=0:

# set this to 0 to disable apport, or to 1 to enable it
# you can temporarily override this with
# sudo service apport start force_start=1
enabled=0

Save and close the file.

Finally, stop the still-running apport service:

[chris@home ~]$ sudo stop apport
apport stop/waiting
[chris@home ~]$ 

Sometimes I find that some Ubuntu installations don’t have the dig command available:

[chris@work ~]$ dig +short example.com
-bash: dig: command not found
[chris@work ~]$ 

The solution is to install the dnsutils package. For example:

[chris@work ~]$ sudo apt-get install dnsutils
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  rblcheck
The following NEW packages will be installed:
  dnsutils
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/147 kB of archives.
After this operation, 378 kB of additional disk space will be used.
Selecting previously unselected package dnsutils.
(Reading database ... 182371 files and directories currently installed.)
Unpacking dnsutils (from .../dnsutils_1%3a9.9.3.dfsg.P2-4ubuntu1.1_amd64.deb) ...
Processing triggers for man-db ...
Setting up dnsutils (1:9.9.3.dfsg.P2-4ubuntu1.1) ...
[chris@work ~]$ dig +short example.com
93.184.216.119
[chris@work ~]$