Observations, Musings & Gadgets

...things I find interesting

Articles in admin category

Individual keyfiles for apt repositories

As we use salt stack to automatically provision software as part of a preseeded auto install, one of the things we need to be able to do, is to add an apt repository.

There is a way to natively add the keys for a repository, but if you do that, the keys all end up in /etc/apt/trusted.gpg. We would much prefer that keys for third party repositories end up in a separate file in /etc/apt/trusted.gpg.d, as we typically add third party repositories to their own files in /etc/apt/sources.list.d

This is following the preference for leaving package installed config files as stock as possible and making local configuration changes in a .d directory.

Given a repositories gpg key, we cannot use the apt-key tool to add the key, as this would add it to the default keyfile, /etc/apt/trusted.gpg. So the answer is to use gpg.

gpg --no-default-keyring --keyring ./somerepo.gpg --import somerepo.key
cp somerepo.gpg /etc/apt/trusted.gpg.d
chmod 644 /etc/apt/trusted.gpg.d/somerepo.gpg

EDIT: This hasn't always worked, but I've found a better way.

gpg --no-default-keyring --keyring gnupg-ring:./somerepo.gpg --keyserver hkp://somekey.server --recv-key A5BE2D9C67A18DE6
cp somerepo.gpg /etc/apt/trusted.gpg.d
chmod 644 /etc/apt/trusted.gpg.d/somerepo.gpg
tagged as ubuntu, salt, deb

Creating your own deb files

Our automated Ubuntu installer needs to be able to handle what we call "unpackaged" software. By this we mean, that obscure piece of software that only exists as a tarfile of compiled binaries.

In the past, our first incarnation of salt controlled installs used salt's cmd.run to perform an rsync from a file server to the client machine. This was far from ideal, as even if the package was installed, the rsync would run to compare the two filesystems, which was both slow and uneccesary.

Then we discovered salt's native archive module. So now we use tar.gz files to push out these unpackaged bits of software.

It goes something like this. Touch a file on the target machine, install the software then use find / -newer marker.file >> filelist.txt. Then you weed out all the obviously unneeded files from your list of files (ie /sys /var/cache etc) and use that list to build a tar file. The we add other actions to our salt state to set additional paths (by adding files to /etc/profile.d) and other actions (a desktop icon, or symlink to /usr/bin perhaps).

We put the resulting tar file on our file server and use salt's archive module to extract, which has an unless clause to prevent unarchiving if it's already in place. You do have to create a hash file though.

You with me so far?

Last week, an install failed - java wasn't installed, which broke a few other package installs. Turns out the third party PPA we use to install Oracle Java hadn't updated in a while and the tar file it referenced on Oracle's servers had been removed. So I set about "repairing" the deb file. Having never worked with deb files, I was pleasantly surprised at how easy it was.

dpkg-deb -R original.deb somedir

This command will unpack the deb file into the directory specified.

somedir-
       |-DEBIAN
       |-usr
       |-bin

Inside the DEBIAN directory are the control files. The most important one (and infact the only required one) is the control file. We will look at this later.

For this particular deb file, the relevant bits were in the postinst script. This was actually the part of the process that grabbed the tar file from Oracle's site and installed it. I changed the script to pull a local copy of the tar file.

To rebuild the deb package is just as easy.

dpkg-deb -b somedir new.deb

Obviously, if you are hacking an existing deb file like this, you should probably change the name of the package to prevent possible conflicts. In this case, we kept the same version, with the thinking that, when the PPA updated, the newer version would overwrite the existing (hacked) version.

That was relatively painless. So what about creating a package from scratch? It's actually quite simple.

Armed with the information in the deb-control man page, I set about trying to create one.

Here's an unpackaged piece of software. Clojure is a general purpose programming language, that is supplied in a zip file. We typically install unpackaged software into /opt. So how do we go about that?

Firstly, we'll create the structure we need. Remember we want the software to be installed in /opt/clojure.

cd /tmp
mkdir clojure
mkdir clojure/{DEBIAN,opt}

By the way, if you haven't seen the curly braces used this way before, it's a sort of short hand. It means repeat this command, substituting each item in the list. So mkdir clojure/{DEBIAN,opt} is the same as typing..

mkdir clojure/DEBIAN
mkdir clojure/opt

If I wanted to be really lazy, I could have also typed mkdir -p clojure/{DEBIAN,opt} which would have made all three directories.

Now, where was I? That's right, creating the structure for the deb file. If we ignore the DEBIAN directory for the moment, all the files inside the deb will be copied into place on the target computer, relative to /. By placing our files in clojure/opt they will appear in /opt after we install our deb file.

The zipfile for clojure, when unzipped, creates a folder called clojure-1.8.0. We'll unpack that in the right place and rename it.

cd clojure/opt
unzip /some/path/to/clojure-1.8.0.zip
mv clojure-1.8.0 clojure
cd ../..

If you wanted to include other files, to be placed in other parts of the filesystem, just create the neccessary folders inside the clojure directory.

The only other file that is required is the control file, clojure/DEBIAN/control

This file specifies information about the package. There are four required items; Package, Version, Maintainer and Description.

There are also some optional items. The dpkg-deb program complains if you don't set the Architecture, so we will set that as well.

Package: clojure
Version: 1.8.0
Architecture: all
Maintainer: thats me <me@somewhere.com>
Description: Clojure is a robust, practical, and fast programming language with a set of useful features that together form a simple, coherent, and powerful tool.

That should be sufficient to make the new deb file.

dpkg-deb -b clojure clojure-1.8.0.deb

That's it. Deb package successfully built.

For anything more complicated than that, there are also scripts that fire before and after the install of the files, called preinst and postinst. If you do use these scripts, you should seriously consider implementing the prerm and postrm scripts to undo the actions of those scripts in case the package is uninstalled.

These scripts give you a lot of scope. If you want to find out how mainstream packages work, it's really easy to unpack existing deb files to see how their maintainers have made use of these scripts.

tagged as ubuntu, salt, deb

Lansweeper triggering an email deluge from sudo

Sometimes, systems designed to make life easier, only make life harder.

Since switching to Ubuntu for our desktop OS, we have been sporadically receiving emails that look like this.....

*** SECURITY information for hostname ***
hostname : Jul 11 15:30:57 : username : problem with defaults entries ; TTY=pts/20 ; PWD=/home/username ; 

At first, it was annoying, but it doesn't happen frequently enough that it's a problem, and it acts like a flag. We'll often take a look at a user's .bash_history to see if they are trying to do something they shouldn't be doing. In most cases it's naivety. They've been trying to do something and they find a web page that says, "Oh, you need to install packageX, it's easy just do sudo apt-get install packageX....." and they blindly follow those instructions.

Anyway, I digress. Recently we've been looking at Lansweeper as a tool to manage our machines. To gather information about our hosts, it needs credentials to access them. By default, it also rescans those hosts it knows about once every day. Something in that process was triggering multiple sudo error messages for each host and we ended up with 880 error messages. Every day.

So, time to bite the bullet and fix the underlying issue.

SSSD, when it installs, makes changes to nsswitch.conf. Specifically, it adds sss the the line controlling the source for sudo lookups.

sudoers file sss

Which isn't unreasonable. We use LDAP for identity and auth (and autofs). But we don't use it for sudo rules. I'm not sure I've ever heard of anyone putting their sudo rules into LDAP, but then again, you can have your DNS entirely in LDAP if you want to (why would you want to?), so having sudo rules in LDAP probably isn't that leftfield. Who am I to judge, right?

The problem is that if sudo cannot find a matching rule in /etc/sudoers or somewhere in /etc/sudoers.d, it then asks sssd to take a look, which generates a hard error every time.

The solution. Well, there are two. First, we can just remove the sss from the sudoers line in nsswitch.conf. I'm wary of doing that. It's not at all an outside chance that an update to sssd might just decide to reinstate that.

We manage our sssd.conf file. It's a jinja templated file, pushed out by Salt stack, with customisations for different use cases (mainly just the ldap_access_filter to control who can log into a machine). So it makes sense to apply changes to that file to prevent this issue.

How do we do that? It's actually quite simple. We accept that we can handle sudo requests and then say we have no way to resolve them.

In the [sssd] section, we add sudo to the list of services we handle.

[sssd]
services = nss, pam, autofs, sudo

Then in our main domain section we set our sudo_provider to none

[domain/LDAP]
id_provider = ldap
auth_provider = ldap
sudo_provider = none

"Hey, wouldya lookat dat!" No more email bomb every day.

tagged as ubuntu, sudo, lansweeper

Determining disk device in Ubuntu preseed

When we switched to using Ubuntu as our standard distribution for Linux desktops, one of the hardest things was getting to grips with preseeding.

Especially the disk partitioning.

Don't get me wrong. There are things about preseeding that are really good, like being able to preset options with debconf, but disk recipies are a bit of a black art.

So imagine the wailing and gnashing of teeth when, with the rise of SSD drives in desktop machines, /dev/sda is no longer your primary drive, and your carefully crafted pressed files fail to partition the disk.

One thing we most definitelty do NOT want, are hardware specific preseed files. The way we think it should work, is that you walk up to a machine, boot it from the network and choose a boot configuration based upon the machine's role, not its hardware. We have a PXE menu with about six or seven various configurations to choose from. Those different configurations (which are basically passing parameters to the preseed file - which is actually a python wsgi script that pulls together jinja templates to make a preseed file) specify whether or not the machine is a lab, staff or graduate machine, which software set to install, which printqueues to configure, etc. We should not need to know if the machine has an SSD (or what graphics card it has) to choose a correct preseed file. We want our installs to be as fire-and-forget as possible.

Here is where we can use some of the power inherent in preseeding.

We are going to use an early_command directive to determine the device name for the primary disk. Early commands are run before a particular stage of the install. In our case, we use an early command for the partitioning.

d-i partman/early_command string

This specifies a command to execute before the partitioning stage of the install. But what can we do to find out the device name for the primary disk? How about this.

~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   477G  0 disk
+-nvme0n1p1 259:1    0   237M  0 part /boot
+-nvme0n1p2 259:2    0  93.1G  0 part /
+-nvme0n1p3 259:3    0 381.2G  0 part /Scratch
+-nvme0n1p4 259:4    0   2.4G  0 part [SWAP]

The lsblk command reads the sysfs system to show information about the block devices attached to the system. Reading the man page for lsblk reveals some useful flags that can help us out. We can specify no header, list format, and sizes reported in bytes.

~# lsblk -lbn
nvme0n1   259:0    0 512110190592  0 disk
nvme0n1p1 259:1    0    248512512  0 part /boot
nvme0n1p2 259:2    0  99999547392  0 part /
nvme0n1p3 259:3    0 409300107264  0 part /Scratch
nvme0n1p4 259:4    0   2560622592  0 part [SWAP]

There are several things we could do here. From the man page we see that we can also control the output and specify the output columns and their order. So let's just return the size and device name so we can sort on size.

~# lsblk -lbn --output SIZE,NAME
512110190592 nvme0n1
   248512512 nvme0n1p1
 99999547392 nvme0n1p2
409300107264 nvme0n1p3
  2560622592 nvme0n1p4

Almost there. If we sort by size and then grab the biggest, we can cut the last field to give us our device name.

~# lsblk -lbn --output SIZE,NAME | sort -n | tail -n 1 | cut -d" " -f2
nvme0n1

Now we can use the output from the lsblk command to set the diskname.

d-i partman/early_command string \
  PRIMARYDISK=$(lsblk -lbn --output SIZE,NAME | sort -n | tail -n 1 | cut -d" " -f2);\
  debconf-set partman-auto/disk "$PRIMARYDISK";

Of course, this assumes that you want the largest disk to be your primary disk. For us, this is fine, as all our desktops only have single disks.

Your mileage may vary.

tagged as ubuntu, preseed

Showing DD progress

Have you ever been frustrated by dd's lack of feedback?

In newer versions of dd (8.24 or newer) you can use the status=progress parameter.

However, if you're using an older (ie Ubuntu 12.04) USB rescue image, you don't have the luxury of that.

So here's a tried and true method of getting dd to report on it's progress.

Basically, you send it a signal, using kill. Because of its name, it is easy to forget that kill can send many different signals to a process, not just -HUP or -KILL. Just to give you an idea, here's the list of signals that kill can send.

~# kill -l
 1) SIGHUP  2) SIGINT  3) SIGQUIT  4) SIGILL  5) SIGTRAP
 6) SIGABRT  7) SIGBUS  8) SIGFPE  9) SIGKILL 10) SIGUSR1
11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR
31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3
38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8
43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7
58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2

Here's a catch for new players. Bash has its own builtin kill command. The output above is from that. If you man kill, you will see there is a -L flag. If you try kill -L in a bash shell, it probably won't work. Instead, try the following to invoke the kill executable.

~# /bin/kill -L
 1 HUP      2 INT      3 QUIT     4 ILL      5 TRAP     6 ABRT     7 BUS
 8 FPE      9 KILL    10 USR1    11 SEGV    12 USR2    13 PIPE    14 ALRM
15 TERM    16 STKFLT  17 CHLD    18 CONT    19 STOP    20 TSTP    21 TTIN
22 TTOU    23 URG     24 XCPU    25 XFSZ    26 VTALRM  27 PROF    28 WINCH
29 POLL    30 PWR     31 SYS     

Anyway, the point of all that, is that you can send the -USR1 signal to the dd process to get it to report its progress.

Open a separate terminal (or change to another virtual terminal) and find the process id of dd.

~# pgrep -l '^dd$'
8385 dd 

Now you can send the USR1 signal to the process.

~# kill -USR1 8385

As soon as the USR1 signal is detected, dd will print the current statistics to STDERR.

~# dd if=/dev/random of=/dev/null bs=1K count=100
0+14 records in
0+14 records out
204 bytes (204 B) copied, 24.92 seconds, 0.0 kB/s

After reporting the status, dd will resume copying. You can repeat the kill command at intervals to see the current stats. Or why not use watch to send the USR signals at intervals.

~# watch -n 60 kill -USR1 8385

The output will appear in the terminal where the dd command is running, not the terminal you execute the kill command from.

tagged as shell