Blog

  • Host monitoring with Prometheus

    I needed monitoring. The plan was to go for an external service – if our environment breaks down, the monitoring is still functional (at least as far as the remaining environment goes). I started to evaluate sysdig cloud, which comes somewhat recommended from “the internet”.

    But then I was kinda unsatisfied (to be honest – most probably unjustified) with the service, because I really didn’t like the UI, and then one metric which was displayed was just wrong. So I got back to prometheus, which we use for metrics gathering of our running services anyway, and used it for host metric monitoring, too.

    That’s my setup. (sorry for the crappy graphic, WordPress does not support SVG … ?!?)

    monitoring setup diagram

    Cause I have consul running on every host, puppet deploying everything, I can use puppet to register the exporter services to consul, and consul to configure prometheus, which has native consul support.

    The prometheus configuration to pull all this is pretty simple actually, once it works:

    global:
      scrape_interval: 10s
      scrape_timeout: 3s
      evaluation_interval: 10s
    scrape_configs:
      - job_name: consul
        consul_sd_configs:
          - server: consul.internal.net:8500
            services: [prom-pushgateway, cadvisor, node-exporter]
        relabel_configs:
          - source_labels: ["__meta_consul_node"]
            regex: "^(.*)$"
            target_label: node
            replacement: "$1"
          - source_labels: ["__meta_consul_service"]
            regex: "^(.*)$"
            target_label: job
            replacement: "$1"
        metric_relabel_configs:
          - source_labels: ["id"]
            regex: "/([^/]+)/.*"
            target_label: item_type
            replacement: "$1"
          - source_labels: ["id"]
            regex: "/[^/]+/(.*)"
            target_label: item
            replacement: "$1"
          - source_labels: ["id"]
            regex: "/docker/(.{8}).*"
            target_label: item
            replacement: "$1"
    

    Some caveats:

    • Prometheus will not tell you why a relabeling does not work. It will just not do it.
    • Prometheus will not tell you that a regex is faulty on SIGHUP, only on restart.
    • The difference between “metric_relabel_configs” and “relabel_configs” seems to be that the former must be applied to scraped metrics, while the latter can only be applied to metrics which are “already present”, which seems to be only the “\__*”-meta labels (for example “__meta_consul_service”)

    Then it works like a charm.

    And the final bonbon: Directly after I had it running I discovered a problem:

    grafana shows a problem

    Yippieh πŸ˜€

    consul, #monitoring, #prometheus, #puppet

  • CI/CD, the status quo

    … is quickly summed up: Working. πŸ™‚

    So I am – naturally – a happy camper. Although, there are a couple of things I would like to change, which don’t scale or are in a state I’m not happy with for “v1.0” yet.

    Container data persistency

    Rancher does offer convoy, but I’m not sure it really fits my needs, and even if I’m not using it yet. And it starts to hurt not to have this. Badly. (AWS Elastic File System would be excactly what I need, but that’s not going to happen I fear). And even this is the most hurting part, not sure if this is the one I can solve quickest.

    Pressure: 9

    Monitoring.

    It’s embarassing, but – I don’t have host monitoring. I don’t even remember how often I needed to re-create a host cause the disk went full without me noticing (and recreation is just so much easier than fixing, actually, so at least this is a good sign). The current evaluation candiate is sysdig cloud. That might change to DataDog. And don’t get me started on log management.

    Pressure: 8,5

    TeamCity.

    While I like TeamCity, it’s – like all things Java – horribly, completely, fully over-engineered mess with billions of functions which all don’t quite do what you want. In our case it’s also reduced to executing basically 3 scripts. So TeamCity must go, mid term. Replacement? No idea. Gradle, drone.io, wercker, distelli or Travis seem viable candidates. (Also found an interesting article about online CI tools which is well worth a read).

    Pressure: 7

    Intmaniac.

    It’s written by me. And it sucks, and, although it works, it must go.

    Pressure: 7

    Security.

    Currently I am focused on “getting things to usable”. Security is not on that list. I need to address some of the issues which I ignored until now, mainly because they might have architecture impact. (Which I believe I planned pretty well, but who knows until you have to actually do it, right?)

    Pressure: 6

    Puppet.

    I am still using Puppet. Might be overkill, but it’s reliable once set up. That brings me to … the set up. I am still using masterless puppet, one environment, and pulling the repo from each host each time. Simple, robust, working, but not elegant. I see different environments coming up, and then … hm. It’s gonna be complicated.

    Pressure: 6

    Rancher.

    Rancher is awesome. But I’m so desperately waiting for this feature right here to arrive it’s not funny any more πŸ™‚ . I also need the same thing for host-based services (which are not inside of a container orchestration platform). I want to spawn a service somewhere, and an Amazon Route53 entry should appear automatically. Preferably based on a consul readout. So DNS management becomes a non-issue.

    Pressure: 5

    HA.

    If one host breaks, the whole cloud is inoperable. Which one? The NAT host. Needed for any host to get a connection to the outside in AWS. And that’s just one breaking point of a couple. Critical? Not really. Super annoying? You bet.

    Pressure: 4

    Cloudformation.

    Currently I use cloudformation to set everything up. And although it’s an awesome product, it’s kinda limited. If a host is gone, I can’t call “cloudformation fix-it” and this very host will respawn. With tools like terraform this is possible, but terraform relies (or relied the last time I looked) on local state data, which is a big pffft. It also means to really test an updated cloudformation template I have to recreate the full thing, which is just too cumbersome. (Takes about 1 hour to completely migrate everything, if done quickly, which is awesome for a complete outage, but really bad for testing). But maybe I just don’t know enough and my template is way too tightly written.

    Pressure: 3

  • Arch with dm-cryptt on UEFI boot

    Just a collection of links, cause this is something which is not really documented in full. I am also too lazy, but next time I don’t want to search πŸ™‚ .

    • Basic installation instructions: 3rd party article (German), Arch Linux docs (without LUKS)
      • Read the comments on the first article: gummiboot is now bootctl, you have to format the boot partition with FAT32, and you should use a better random generator on cryptsetup. It’s all in there.
      • Note: The former /boot partition is identical to the UEFI boot partition. You don’t need both.
    • Arch Linux original instructions: In German, in English
      • Careful: The setting “FONT_MAP” in /etc/vconsole.conf in theΒ German guide shouldΒ not be applied! It’s obsolete.
      • The English guide does only go into the crypt installation, but that really deep.
    • If you actually added FONT_MAP to /etc/vconsole.conf, that happens
    • The file /vmlinuz-linux comes in the “kernel” package. If it’s missing, just reinstall “pacman -S kernel”
    • You can usually choose between NetworkManager and systemd-networkd as networking management solutions. I chose NetworkManager:
      • systemctl enable NetworkManager
      • systemctl start NetworkManager
      • systemctl disable systemd-networkd
      • systemctl disable systemd-resolved
    • Installing GNOME (and the login manager gdm) is pretty simple:
      • pacman -S gnome gnome-extras
      • systemctl enable gdm

    That should be it.

  • Really annoying thread properties

    This sucks monkey ass, mainly because I didn’t think of that before. And that’s just one example why multi-threaded (soon to be -processing, probably) applications are hard.

    import subprocess as sp
    import time
    import os
    from threading import Thread
    
    
    class MyThread(Thread):
    
        def __init__(self, mydir):
            super().__init__()
            self.mydir = mydir
    
        def run(self):
            os.chdir(self.mydir)
            time.sleep(2)
            print("I'm (%s) in directory %s"
                  % (str(self), os.getcwd()))
    
    if __name__ == "__main__":
        MyThread("/tmp").start()
        time.sleep(1)
        MyThread("/").start()
    

    Result is:

    I'm (<MyThread(Thread-1, started 140195858716416)>) in directory /
    I'm (<MyThread(Thread-2, started 140195850323712)>) in directory /
    
  • Mac software essentials

    Well, after the Firefox Essentials, here’s my list of Mac essentials:

  • My essential Firefox addons

    … can be seen on the Firefox Addons page as collection πŸ™‚ .

    The addon ecosystem of Firefox is what keeps me with it. And the way it’s done. One example: Mouse Gestures are available on Chrome, which seems to be a lot faster and smoother today :'( … but in FF, when I go to the settings page, I can still use mouse gestures, cause it’s just another page.

    In Chrome I can’t. I like the Firefox behavior – it’s consistent, which is highly a highly underrated property these days.

  • jq stuff

    Waah. In case I *ever* need this again.

    aws ec2 describe-security-groups | \
        jq '.SecurityGroups | \
            map(select(has("Tags"))) | \
            map(select(.Tags[].Key|contains("Name"))) | \
            .[] | \
            [ .Tags , .GroupId ] | \
            [ (.[0]|map(select(.Key=="Name")))[0].Value, .[1] ] | \
            join("=") '
    

    What does it do? It makes a list

    $NAME_TAG=$GROUP_ID
    

    … for each security group which has a name tag. Yeah, right.

  • Shortcut with β€œΓΆβ€ (o-umlaut)

    I want my clipboard history on CTRL-Γ–. Don’t ask why.

    Here’s what you can enter in GNOME to get this:

    <Ctrl>odiaeresis
    
  • Rancher IAM role

    Rancher can create instances on EC2. If you want to define a dedicated IAM user for this, refer to the Amazon docs for the a profile template.

    Unfortunately the first thing you get when using those permissions in rancher is “You are not authorized”. Great. I’ll update this when I know the correct permissions.

    (Source: Rancher docs)

  • VPC with NAT to internet on AWS

    … and other TLAs.

    Anyways, as far as I remember OpenStack does not need this, so I thought I document it here. I at least was surprised.

    Situation: You want a private network sement in the cloud (in my case an Amazon VPC), and you don’t want all hosts to be accessible from the internet. So you don’t assign public IPs, and you need a router/gateway.

    Amazon creates a network internet gateway, but this thing does not do one thing: NATting. If your host does not have a private IP, it can’t connect to “the internet”.

    Solution: You actually need to instantiate an EC2 instance, which you have to configure to do NAT (which is forwarding and masquerading) with a public IP address. And you have to set routing tables which point to that instance for all networks which should be inaccessible from the internet.

    Thankfully there’s an article providing an example CloudFormation template.

    Really, thanks.

    Configuring the NAT instance is super-easy then. Amazon mentions in its docs that there are special Amazon Linux instances (“These AMIs include the string amzn-ami-vpc-nat in their names […]”) which come with NATting preconfigured. Just instantiate an instance using the appropriate AMI image, and you’re done. No further configuration needed.