Category: Snippets

  • Shutter can’t edit images on Arch

    Unfortunately shutter does no longer work (or not yet, maybe, hopefully 😉 with Wayland on Arch. But I still use it for image editing, namely screenshot annotations, for which this is the best tool by far I have ever found. Not to mention the most private one, cause everybody and his dog wants you to upload “to the cloud” nowadays.

    On a freshly installed system you will find the “Edit” button grayed out though after you installed Shutter. Reason being there’s a lib missing which is not installed by default. This is how you install it:

    $ yaourt -S perl-goo-canvas
    

    And the editing continues.

    (Original source: this one. Thanks!)

  • JIRA and Python

    I really came to hating JIRA with a passion. And now I have to create about 350 tickets.

    Naturally I don’t do this by hand. But using the JIRA API is kind of … hard, there is a Python library, but usage is rather sparsely documented, and this whole thing is just annoying as hell.

    Although when you did it, it’s quite simple. Here’s an example of how to create a ticket with due date and estimate set from a CSV file:

    from pprint import pprint, pformat
    from getpass import getpass
    
    import jira
    
    
    def do_shit():
        url = read("JIRA url:")
        username = read("JIRA username:")
        password = getpass("JIRA password:")
        jira_inst = jira.JIRA(url, basic_auth=(username, password))
    
        test = {
            "project":          {"key": "TEST"},
            "issuetype":        {"name": "Task"},
    
            "labels":           ["deleteme", "whatisthis"],
    
            "summary":          "woho",
            "description":      "wohooooo",
            "duedate":          "2017-11-11",
    
            "timetracking":    {"originalEstimate": "4d"},
        }
    
        created = jira_inst.create_issues([test])
        pprint(created)
    
    
    if __name__ == "__main__":
        do_shit()
    

    So simple.

  • CUPS is … weird

    CUPS is the printing system developed by Apple which is now also in use on pretty much every other Linux / Unix / *BSD box there is. Unfortunately, the process of managing printers is way more painful than the process on Arch Linux. At least with my setup, which is i3 running in a weirdly crippled GNOME session.

    This is what you have to do to get CUPS working on your machine:

    $ pacman -S cups
    [...]
    $ usermod -aG sys MY_USER
    [...]
    $ systemctl enable org.cups.cupsd
    $ systemctl start org.cups.cupsd
    

    The magic is line 2 – adding your user to the sys group. If you don’t do this, you won’t be able to use the web interface at all. Which is a shame, since then you can’t do anything useful (install printers, manage jobs, etc …).

  • No password prompts for GitHub “hub” client

    GitHub has a little cli tool called “hub” which makes working with repos a bit easier (“hub pull-request” instead of a lot of clicking around on the web). Unfortunately they don’t really tell you how to prevent password inputs every time you use it.

    Unless you dig deep:

  • jq makes AWS “describe-instances” actually useful

    Just so I don’t forget 🙂

    aws ec2 describe-instances | \
      jq '.Reservations[].Instances[] | {IP: .PrivateIpAddress, ID: .InstanceId, Name: .Tags[] | select(.Key=="Name").Value}'
    
  • Linux font rendering sucks, a.k.a “Where is Boohomil”?

    For some reason, the maintainer behind the “*-infinality” packages in Arch Linux “has gone missing” for a while.

    Why is that important to me? Because infinality is a patch set to a bunch of font and rendering packages, which make fonts under Linux look SO much better than the default setup. (Yes, there are still a couple of things that Linux just absolutely cannot compete in with Mac and / or Win, and font rendering is one of them. Ubuntu does a reasonable job of this, every other distro just sucks.

    Except when you were using infinality. And now it’s defunct.

    Anyway, after experiencing the unbelievably ugly phenomenon described in here, I tried this guide here now, and it seems to fix it.

  • Syntax highlighting with wordpress

    This is just a test for syntax highlighting. Which I really really really wanted to have. Even if it’s WordPress and not something cool like hugo.

    So, let’s try:

    import random
    
    class RndTest:
        def __init__(self):
            print("Oy! 'this is python!")
            self.rnd_seed = random.random()
    
        @classmethod
        def do_nothing(cls, nothing):
            print("Doing {} now ;)".format(nothing))
    

    That doesn’t look so bad, right?

    How to do this

    • install the crayon syntax highlight plugin
    • when writing posts, your toolbar will have a new icon looking like this: <>
    • press it, and an “add code” dialog will open
    • do your thing
    • save
    • done

    Like.

    Dislikes

    • I don’t know which JS engine this thing uses (if any public one in particular)
    • I like highlight.js

    (So, none really)

    Alternative plugins (untested)

    To be honest, the only real contender in installation base and features seems to be Syntax Highlighter Evolved. I have not tried it, but if you don’t like crayon, that looks like the one to go.

  • PyCharm, Arch linux & Python 3.6

    Love Python. Love PyCharm. Love Arch Linux.

    Unfortunately Arch sneakily updated Python to 3.6. Cool, new version … but hey, why don’t my debug runs in PyCharm work any more??

    ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory
    

    Yup, pretty confusing. It seems unable to find shared python 3.5 library. Well. After some cursing, turns out the solution is pretty simple (if you know what to do):

    • get pyenv
    • use pyenv to install Python 3.5.2, but with -enable-shared option set
    • use this python version for PyCharm projects (it does not matter if it’s in a virtualenv or not)

    Like this:

    $ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.5.2
    [...]
    $ sudo $HOME/.pyenv/versions/3.5.2/bin/python "/opt/pycharm-professional/helpers/pydev/setup_cython.py" build_ext --inplace
    [...]
    $ _
    

    That solved it for me 🙂

    References:

  • Logs with docker and logstash

    It would be nice to have all container logs from a docker cluster sent to … let’s say, an ELK stack. Right?

    Right.

    So we did:

    • on each host in the cluster, we use the GELF log driver to send all logs to a logstash instance
    • the logstash instance clones each request using type “ELK”
    • to the “ELK” clone, it adds the token for the external ELK service
    • the “ELK” clone goes out to the external ELK cluster
    • the original event goes to S3.

    Here’s how.

    Some thoughts about choosing GELF over the others

    Logstash sees a stream of lines. When the data is sent to it, logstash gets a lot of different data types, depending what software runs in the containers. So to filter out a container / service / instance in Kibana, we need at least the container image name to say “give me the logs of this service type”, and then we don’t even separate by different stages.

    GELF gives us this information out-of-the-box:

    {
           "source_host" => "10...",
                 "level" => 6,
               "created" => "2017-01-09T13:41:06.039364628Z",
               "message" => "This is da log!",
               "version" => "1.1",
               "command" => "./run.sh",
                  "tags" => [],
            "image_name" => "hub/repo/image",
            "@timestamp" => 2017-01-12T11:56:34.395Z,
        "container_name" => "ze-container",
                  "host" => "ip-10-9-8-7",
              "@version" => "1",
                   "tag" => "",
              "image_id" => "sha256:...",
          "container_id" => "..."
    }
    

    This is a lot of information what is useful later.

    Compared to that, syslog looks rather bleak (all according to the examples on the docker pages):

    Aug  7 18:33:19 HOSTNAME docker/hello-world/foobar/5790672ab6a0[9103]: Hello from Docker.
    

    Yes, you can tag that, but you also have to. Not with GELF. Also it’s already beautifully structured in fields, whereas with syslog you’d probably have to agree on a scheme and deconstruct later.

    Over to fluentd:

    2015-09-01 15:10:40 -0600 docker.3fd8678d487e: \
        {"source":"stdout","log":"Hello Fluentd!",\
         "container_id":"3fd8678d487e540c7a303e1613101e746c5012f3317434eda93f24351c1928f7",\
         "container_name":"/angry_kalam"}
    

    This is … not so helpful. Especially the container name and the container ID are super useless, because there’s no way I can get the image name information out of it. You might be able to tag accordingly, but the same now applies as with syslog (except here we already have a document structure it seems). I would have liked to use fluentd, because there’s a low-resource agent on each host, and we don’t need a central collector (i.e. logstash) for lossy UDP traffic. But that made me choose GELF for the first try.

    Cloudwatch now only logs into buckets, which is a big bunch of text. If you want to separate you have to create a cloudwatch bucket for everything, so pretty much one for every dimension (host, container name, …). Either I have not fully understood this, or this is really really useless.

    Now, the story

    (Disclaimer: I am fairly new to the whole ELK ecosystem, so this is all based in my potentially limited knowledge, and might be wrong in places)

    We committed ourselves to just use JSON formatted log output for our log messages. Now, not every program is without flaws, so what we usually have is a couple of stack traces mixed with the output, which are also usually multi-line (read: multiline events in ELK speak).

    So one of my biggest requirements was: To parse the JSON logs as JSON when I have one, and to re-assemble several distinct log messages which form a multi-line event into one.

    The JSON-messages are not a problem. Check whether something starts and stops with a curly brace, and you’re good to go. And it’s always one single line anyway.

    The multiline messages though … . From logstash perspective, there’s just a stream of lines. For each line the information of the container is present, but for logstash that’s just dumb metadata without any meaning. So imagine this sequence of events:

    EVENT 0: CONTAINER_1_HOST_1 JSON-MESSAGE-LINE
    EVENT 1: CONTAINER_2_HOST_2 JSON-MESSAGE-LINE
    EVENT 2: CONTAINER_3_HOST_1 MULTILINE LOG - LINE 0
    EVENT 3: CONTAINER_4_HOST_3 JSON-MESSAGE-LINE
    EVENT 4: CONTAINER_1_HOST_1 MULTILINE LOG - LINE 0
    

    If I naively use the multiline codec like this:

    input {
      stdin {
        codec => multiline {
          pattern => "^\s"
          what => "previous"
        }
      }
    }
    

    I guess that would not work. Why? Because either Event 4 gets appended to the line of Event 3, which is wrong, or (I have no clue how the codecs work) to Event 2, which is equally wrong (different container source).

    Now, the solution to use the GELF input with a multiline codec does not seem to work, because there is a bug in logstash which prevents usage of codecs in the GELF input. Luckily there’s a workaround using the multiline input filter of logstash, outside of the plugin. Fortunately there’s an input plugin we can use, called multiline (yes, just like the filter plugin), which seems to be working. Let’s try this. Unfortunately this plugin is no longer bundled with Logstash 5.x. (Sucks? Yes.) It was “replaced” in favor of the multiline codec, which reads like “deprecated” to me, although I get the motivation. But it makes me feel weird using it. (Sucks even more? Yes.)

    To add insult to injury I deploy logstash using Docker, and the Docker image now does not have this plugin – so I have to build my own logstash image. I don’t like to repeat myself, so I built a logstash image which can dynamically install plugins. Adds ages to startup time, but if I ever have another plugin I need I don’t need to rebuild the image, just restart.

    [ Our Logstash dynamic plugin docker container on quay.io ]

    Our test-config for now, which seems to be working well so far (couple of test runs only), is:

    input { gelf {} }
    
    filter {
    
      # if we have json, let's use it.
      if [message] =~ /^{.*}$/ {
        json {
          source => 'message'
          target => 'data'
        }
    
        mutate {
          remove_field => 'message'
        }
      }
    
      multiline {
        pattern => '^\s'
        what => 'previous'
        stream_identity => "%{container_id}"
      }
    
      # let's clone and add the token field. the ORIGINAL input goes to S3,
      # the CLONED input with the token field goes to logz.io
    
      clone {
        clones => [ "one" ]
        add_field => { "token" => "asdf" }
      }
    
    }
    
    output { # ... }
    

    That works somewhat well. All in all, I am pretty pleased after the whole thing, because once you have overcome the obstacles of the logstash config language, plugins, docker configuration, etc., it is running pretty smoothly.

    Now, I still don’t like this solution for the following reasons:

    • Apparently a huge performance impact if we do this, but this is IMHO the only real solution to the problem – except doing crappy stuff and then maintaining it.
    • It might be that the fluentd-based system is more robust (no single point of failure on my side) and better fitting for the task (multiline, no lossy UDP, which might be a problem in some use cases).
    • Now even if I could use the codec instead of the filter, it does not come with a “stream_identity” parameter. Whether you don’t need it any more (then how does it do it?!) or it went without replacement I don’t know.
    • I have to scale up (anyways) – but I do not have any idea about UDP package loss and resource requirements on the logstash side yet, and when something happens logs will be lost. This is rather unfortunate.
    • Java stack traces are fine. Python and Go stack traces not so much – they contain non-indented lines within the body of the stack trace, so the “starts with whitespace? add to last line” approach does not work. And I have no idea for other approaches to be honest.
    • Changing logstash configs promises to be a pain – because it’s central component, so doing this without logs getting lost you seem to need to …
    • have multiple logstash instances, load-balanced using IP round robin probably, with at least one more than you need for the full load
    • Take one of those instances out of the round-robin
    • Wait until traffic is exactly down to zero (which might take time), …
    • adjust the rules, …
    • test somehow (takes quite a while because of logstash start times, and if you have to test multiple times this is quite annoying) …
    • and save.
    • Then re-add the instance-IP to the IP pool, …
    • … and repeat for all the rest.

    I hope I am doing a couple of things “wrong”, because as long as it runs it runs fine, but changes are not so agile in this description. I still might check out fluentd in more detail, because if that workflow is needed fluentd might be a bit more flexible.

    But all in all a really great first start.

  • Logstash, clone filter & add_field mysteries

    That’s a really great piece of documentation. This does not work:

    # let's clone each event, one goes to somewhere, one goes to somewhere else.
    # note this was copy-pasted from the docs!
    # see here: https://is.gd/QSHNps
    # again. THIS DOES NOT WORK.
    filter {
      clone {
        add_field => { "token" => "ABCDEF" }
      }
    }
    output {
      if [token] {
        # go somewhere
        tcp { ... }
      } else {
        # go to somewhere else
        s3 { ... }
      }
    }
    

    Why? Because the clone filter will not clone anything. And the documentation is super unclear on this. If you know it, you can read it – if you don’t know this, you’ll … google.

    For it to actually clone anything you have to specify the ‘clones => [“one”, …]’ parameter. Then it will clone, and add the token field as expected. Like this:

    filter {
      clone {
        clones => ["logz"]     # NOW it will clone.
        add_field => { "token" => "ABCDEF" }
      }
    }
    

    Interestingly the “clones =>” parameter is optional, which just confuses the shit out of me.

    The reasoning that I don’t just add the field altogether is that this is the access token for our externally hosted ELK service. This should only be there for the external path, and not be put in S3 in parallel.

    References:
    • https://logstash.jira.com/browse/LOGSTASH-2144