Pierre Carrier's blog

All my subtitle ideas sounded cheesy.

Page Cache My Assets

Many services rely on a dataset stored on disk.

If accesses are frequent and non-linear, performance remains reasonable as long as they are cached in memory. Then suddenly, some background job is triggered, a backup for example, and the data gets evicted from the page cache. Performance drops.

In many cases this is acceptable. The service throughput drops. In a request-reply model, requests gets queued. But if latency is not critical, remains below the client timeout, and if the machine is dimensioned properly, data will be cached again, the service will catch up. The temporary slowdown was acceptable.

In other cases, this is catastrophic. Think high-frequency trading.

Here comes pcmad, the “Page Cache My Assets” daemon. It simply locks and unlocks files in memory as requested. When a file is locked, it won’t be evicted, whatever happens on the server. If the system runs out of memory, the oom-killer would be likely to kill this daemon first, so the upstart job and systemd unit indicate that killing pcmad should be avoided at all costs.

A simple and documented protocol, based on ØMQ and MessagePack, makes pcma easy to integrate with your existing services. For scripts, we also ship a client.

The project reached 0.2.0. It isn’t stabilized yet, so the protocol might not remain backward-compatible. However the code is simple and passed reviews, so feel free to give it a go!

Tweak the MOTD under Ubuntu Precise

  • The MOTD is generated by scripts in /etc/update-motd.d. You can change those to affect the generated MOTD.

  • To stop displaying the MOTD on every login (“Welcome to Ubuntu[…]”), edit /etc/pam.d/login like so:

1
2
3
4
5
--- /etc/pam.d/login.before
+++ /etc/pam.d/login
@@ -85 +85 @@
-session optional pam_motd.so
+# session optional pam_motd.so

A similar change is required for all PAM services showing the MOTD, often including sshd.

  • To disable this mechanism, instead of commenting the line, add the noupdate option like so:
1
2
3
4
5
--- /etc/pam.d/login.before
+++ /etc/pam.d/login
@@ -85 +85 @@
-session optional pam_motd.so
+session optional pam_motd.so noupdate

Again, the same change is required in all services using pam_motd.so. You also need to disable the execution on boot with this change:

1
2
3
4
5
--- /etc/init/mounted-run.conf.before
+++ /etc/init/mounted-run.conf
@@ -22 +22 @@
- [ -d "/etc/update-motd.d" ] && run-parts --lsbsysinit /etc/update-motd.d > /run/motd &
+ # [ -d "/etc/update-motd.d" ] && run-parts --lsbsysinit /etc/update-motd.d > /run/motd &

Fedora 15 installer: hard to find :)

On fedoraproject.org, the new prominent installation media with Fedora 15 is the live desktop, eg. Fedora-15-x86_64-Live-Desktop.iso.

Download it, boot it in VirtualBox. Gnome 3 will use the fallback mode. How do you start the installer? Well, turns out it is normally made prominent by a Gnome Shell extension (~/.local/share/gnome-shell/extensions/Installer@shell-extensions.fedoraproject.org/).

Did I mention that in fallback mode, which you can enjoy with most virtualization technologies, Gnome Shell is not running? Instead the installer ends up in the menu. Somehow it took me over 15 minutes to find it. Most people would be ashamed, I blog about it.

In case you wonder how to start the Fedora 15 installer in your virtual machine (SSO)… You can find it in Applications → System Tools → Install to Hard Drive. Alternatively, Alt+F2, liveinst. My bad for not partaking in the testing effort…

It makes me feel old and nerdy, but I love console-based installation processes.

7clock, console 7-segment clock

I wanted a minimal, fullscreen clock I could run without X. In the longer term, I am thinking about a locking console “screensaver” (more about that later). I ended up writing a small library (240 lines and counting) to render a 7-segment display in monospaced text at any “resolution”.

The end result adapts to the console size, instantaneously if resized. It lacks any options, if I’m bored again I might come back to it. It should be quite efficient, though more trivial optimizations are possible.

Older screenshots from various prototypes (the test program is available alongside the clock; as usual, I recommend sticking to the instructions in INSTALL at the root of the repo):

errnos, identify that random error number

As a Technical Support Engineer at Red Hat, I got to read a lot of logs, error messages and code. One of my pain points was errnos.

They rely on magic numbers), which is perfectly understandable given their use cases (eg return values for system calls). But obviously, we’re not going to be using magic numbers everywhere, hence they are defined as constants) for the C preprocessor.

Numbers are not very user-friendly, so why not display them in a human-readable format when a human might want to read it? strerror comes to the rescue! Unfortunately, not every piece of code presenting errors to the user or administrator uses it. There are various reasons for that:

  • It is a libc function, and the Linux kernel didn’t implement something similar (or most of us aren’t aware of its existence);
  • The length of the (hexa)decimal representation of an integer is bound, whereas the strings it returns are not, which can make memory allocation for log messages trickier.

strerror strings are very readable; for example, it will turn ENOMEM into Cannot allocate memory under OSX. I will not lose too much of your time on the annoyances caused by weird usages of errnos, though I encountered quite a few already: they usually do not delay troubleshooting too much if you’re not too trusting. For example ENOTDIR, represented by glibc’s strerror as Not a directory, indicates when returned by keyctl_search(3) that one of the keyrings is a valid key that isn’t a keyring.

The tricky part now? From a number or an strerror description in logs or an output, there is no trivial way to establish which constant to look for. What’s more, C preprocessor constants are not available at runtime, you’d need the headers at hand, cpp, and your own list to go through as there isn’t a standard one! For example, if you see 44, 0x2c or Channel number out of range, what should you git grep for in the affected software, libc and/or kernel? ECHRNG, of course!

I’d love to give you a simple table listing each errno constant, its representation in decimal and hexadecimal and its strerror description, but there are a few reasons why I can’t:

  • Probably mostly for historical reasons, the association between constants and integer values is not common between operating systems; worse, they can also change between hardware architectures.
  • The strerror representation varies between systems, in particular depending on the libc being used.
  • Some operating systems use the same value for multiple constants. This is not a bug per se as long as no standard function can return those different constants for different reasons. On my systems, those are EDEADLK and EDEADLOCK under Linux, EAGAIN and EWOULDBLOCK under OSX.

However, this led to the creation of a simple command-line utility, errnos. Build and run it on the system you investigate, or a similar one (same operating system, same libc, same CPU architecture), and you will get something you can store and grep at will. It could also make an ironic wallpaper for your child’s room, but don’t blame me if they have nightmares of production systems going down.

For once, I used glib as I needed a hashtable of lists and couldn’t be bothered implementing those for the millionth time in history (comp. sci. students who have to do so tonight, I share your frustration). There’s a limited amount of magic involved in the build process, so please just clone the repository and stick to the build instructions unless you have time to lose.

The first column gives the number in its decimal representation, the second in hexadecimal. The third is either its strerror representation between double quotes or a preprocessor constant.

To close this article, here is the end of its output on my Mac:

1
2
3
4
5
6
7
8
9
10
11
12
$ errnos | tail
Stopped looking at 1128
99    0x63    "Not a STREAM"
99    0x63    ENOSTR
100   0x64    "Protocol error"
100   0x64    EPROTO
101   0x65    "STREAM ioctl timeout"
101   0x65    ETIME
102   0x66    "Operation not supported on socket"
102   0x66    EOPNOTSUPP
103   0x67    "Policy not found"
103   0x67    ENOPOLICY

nato, superglob

When you end up having to dictate Unix commands over the phone, you quickly learn from your colleagues that optimism doesn’t compensate your French accent. Turns out cancelling your lovefilm subscription triggers a similar problem.

The most common answer is to involve the NATO phonetic alphabet. Looking up a table pinned to your desk is tedious, getting fluent takes time. That’s where nato kicks in.

Not much to say, really:

1
2
3
$ nato 'lspci|grep VGA'
lima, sierra, papa, charlie, india, pipe, golf, romeo, echo, papa, space, uppercase victor, uppercase golf, uppercase alpha, 
$

After this very useful piece of software (at least to some), let’s have a look at something stupid and useless (I won’t even bother trying to explain the silly reason I needed it). If you ever have a good, real-life reason to run superglob, please E-mail me right away, I’ll owe you the drink of your choice. The idea is to build a glob matching all the arguments provided, but as little more as possible (and we’re far from being clever here).

1
2
3
4
5
6
7
8
9
$ superglob Hello World
[HW][eo][lr]l[do]
$ superglob Foo 'Bar Mitzvah'
[BF][ao][or]*
$ superglob /lib64/libpamc.so.0 \
  /lib64/libpamc.so.0.81.0 /lib64/libpam_misc.so.0 \
  /lib64/libpam_misc.so.0.81.2 /lib64/libpam.so.0 \
  /lib64/libpam.so.0.81.5
/lib64/libpam[._c][.ms][ios][.os][.0c]*

A good example of a really silly behaviour (here, *ahello would likely be the best choice):

1
2
$ ~/usr/bin/superglob ahello bahello cbahello
[abc][abh][aeh][ehl][el][lo]*

mkpasswd

You want passwords that are identical with AZERTY, QWERTY and QWERTZ keyboards? And easier to type on a mobile phone (no uppercase)? Alternatively, with the characters you list and just those?

You don’t want to choose a number of characters, how many digits and special characters should appear (or any nonsensical policy when you know what you are doing), but rather would rely solely on the number of bits of entropy you’ll get?

Generated from /dev/random, as it’s a good source of entropy (under a recent Linux kernel at least)?

Check out mkpasswd.