Blog

  • Shutter can’t edit images on Arch

    Unfortunately shutter does no longer work (or not yet, maybe, hopefully 😉 with Wayland on Arch. But I still use it for image editing, namely screenshot annotations, for which this is the best tool by far I have ever found. Not to mention the most private one, cause everybody and his dog wants you to upload “to the cloud” nowadays.

    On a freshly installed system you will find the “Edit” button grayed out though after you installed Shutter. Reason being there’s a lib missing which is not installed by default. This is how you install it:

    $ yaourt -S perl-goo-canvas
    

    And the editing continues.

    (Original source: this one. Thanks!)

  • Install Arch with full disk encryption, btrfs and EFI

    Updates:

    • 2022-06-23 | some minor corrections, added wifi link
    • 2018-03-27 | fixed a typo in the HOOKS documentation, clarified kernel boot parameters

    Introduction

    I recently had to re-install my beloved Arch Linux. For security I need (and use) full disk encryption. This is a cheatsheet for the whole procedure, because although the Arch Linux Wiki is excellent, it is also huge and sometimes you must pick your stuff together from many pages.

    This is what I am doing here 🙂

    NOTE: Usually you only have to follow the one subsection I link to!

    Overview

    One after another, we will do the following steps

    • Download and prepare Arch USB stick (skipped, you should know that 😉
    • Prepare the hard disk
    • Prepare the disk partitions
    • Add LVM “inside” the crypted partition
    • Create filesystems & mount partitions
    • Install arch
    • Configure boot manager

    Prepare the hard disk

    Use parted to init the disk and …

    1. init the disk using a GPT partitioning scheme, then create
    2. a GPT boot partition and put 100% of the remaining space in another partition (the first two actions behind the link)

    Prepare the disk partitions

    Basically,

    1. use the cryptsetup command to encrypt the main (big) partition,
    2. and create a file system on the boot partition (remember: it must be FAT32 for EFI boot, and it must be _un_encrypted!)

    Add an LVM “inside” the encrypted partition

    Cause we want “properly” encrypted swap (you can also encrypt swap using a /dev/random key every time, but then you will not persist data between reboots and you can’t do things like suspend-to-disk), we need at least two “partitions” “inside” the crypted volume. Sounds like LVM on LUKS? It does. We already used it 🙂 .

    1. Create LVM partitions inside the encrypted volume (Don’t forget to use cryptsetup luksOpen before, usually in step 1 in the last section 🙂

    NOTE: Do not follow the above link down to “prepare the boot partition”, cause they use ext2 and we need FAT32 for EFI boot partitions. Just don’t.

    I use the name “secure” for the VG, and I use btrfs cause I am so incredibly elite, and so we don’t need to set a specific size for the / and /home “partitions” and can just use btrfs subvolumes, while still being able to wipe the system without the home directories. That’s pretty neat if you need it (I never did, but now I can ;). So that’s the final setup:

    /dev/mapper/secure-swap    40 GB, swap
    /dev/mapper/secure-system  rest, btrfs with 2 subvols: root & home
    

    Create filesystems & mount partitions

    Of course, Arch has already a wiki page section for that. I did it 3 times in a different way until I found it and had to do it again. So here is my summary.

    # CREATE BTRFS & SUBVOLUMES
    $ mkfs.btrfs /dev/mapper/secure-system
    $ mount /dev/mapper/secure-system /mnt
    $ btrfs subvolume create /mnt/@
    $ btrfs subvolume create /mnt/@home
    $ btrfs subvolume create /mnt/@snapshots
    $ umount /mnt
    
    # MOUNT PARTITIONS
    $ mount -o subvol=@ /dev/mapper/crypted-system /mnt
    $ mkdir -p /mnt/home /mnt/boot
    $ mount -o subvol=@home /dev/mapper/crypted-system /mnt/home
    $ mount /dev/sda1 /mnt/boot
    

    NOTE: /boot is not on an encrypted partition 😉 , and the leading “@” is a convention for subvolumes which should be mounted somewhere. I also don’t use compress=...  parameters, cause I don’t need / want transparent compression.

    Install arch

    Set up networking

    Nowadays nobody uses LAN cables any more. And of course, with Arch you can do it over WiFi as well. This is the short form:

    $ iwctl
    [iwd] station list
    # get the name of your "station" here
    [iwd] station wlan0 scan
    [iwd] station wlan0 get-networks
    # see the list of networks
    [iwd] station wlan0 connect NETWORK_NAME
    

    Also useful: IWD docs, connecting to the internet on the arch Wiki.

    Install base system

    Then you follow up with the usual installation procedure, but you stop at the “Initramfs” section. Here we will pick up again. In the chroot, this is what is needed and useful. (Yes, we’re installing multiple network managers, Linux “is about choice” …)

    pacman -S lvm2 vim \
              iwd netctl dialog wpa_supplicant \
              networkmanager \
              dhcclient \
              udev nano zsh sudo iproute2 net-tools \
              git
    

    Configure boot manager & boot files

    We are using systemd-boot. Or bootctl, as the binary is called. It should be already installed. The procedure is also outlined here. We also enable TRIM support, it seems to lessen security, but it raises SSD performance and life time.

    1. First, check if your system EFI is all right.
    2. Optionally install the Intel microcode updater package if you have an Intel CPU by doing pacman -S intel-ucode.
    3. Then run … bootctl -path=/boot install to install systemd-boot.
      (If you ran into this error File system ... has wrong type for EFI System Partition, then you forgot set 1 esp on on your boot partition in parted during partitioning)

    Now create/modify the files below. Remember: /mnt/boot should be a mounted directory, and since you’re in a chroot this should be /boot for you.

    FILE: /boot/loader/entries/arch.conf

    title Arch Linux
    linux /vmlinuz-linux
    initrd /intel-ucode.img       # ONLY FOR INTEL CPUs!!
    initrd /initramfs-linux.img
    options luks.uuid=FS_UUID root=/dev/mapper/secure-system rootflags=subvol=@ rd.luks.options=discard
    

    Some hints:

    • use blkid to get the UUID
    • Note that the FS_UUID is the UUID of the *encrypted luks partition* (/dev/sda2 or similar), and not the filesystem within!
    • use a mouse to copy-paste the UUID\
    • gpm -m /dev/input/mice -t imps2; you may have to disable vim’s mouse support using set mouse= and set ttymouse=
    • the list of normal and dm-crypt related kernel parameters … well, is also in the Arch wiki.

    FILE: /boot/loader/loader.conf

    default arch    # the file above without .conf extension, can have wildcards!!
    timeout 2
    editor  0
    

    FILE: /etc/mkinitcpio.conf

    # Just MODIFY that file, to be precisely this line:
    HOOKS=(base systemd autodetect modconf keyboard sd-vconsole block sd-encrypt sd-lvm2 filesystems fsck)
    

    The key idea is to use the “systemd” parameters instead of the “normal” ones. The full list of hooks is of course also available, and the order is important.

    Install boot manager

    Now execute:

    mkinitcpio -p linux
    

    If you run into some vmlinuz-linux - file not found issues, just re-install linux and intel-ucode while making sure /boot is mounted.

    Reboot

    We should be done here 🙂 …

    … but we’re paranoid:

    • did you install lvm2? (if not – install it and run mkinitcpio again)
    • did you install iwd? (if not – no wifi in the final system)
    • did you set a root password? (if not, set it)

    Then you can safely …

    $ reboot
    

    Further reading 😉

  • JIRA and Python

    I really came to hating JIRA with a passion. And now I have to create about 350 tickets.

    Naturally I don’t do this by hand. But using the JIRA API is kind of … hard, there is a Python library, but usage is rather sparsely documented, and this whole thing is just annoying as hell.

    Although when you did it, it’s quite simple. Here’s an example of how to create a ticket with due date and estimate set from a CSV file:

    from pprint import pprint, pformat
    from getpass import getpass
    
    import jira
    
    
    def do_shit():
        url = read("JIRA url:")
        username = read("JIRA username:")
        password = getpass("JIRA password:")
        jira_inst = jira.JIRA(url, basic_auth=(username, password))
    
        test = {
            "project":          {"key": "TEST"},
            "issuetype":        {"name": "Task"},
    
            "labels":           ["deleteme", "whatisthis"],
    
            "summary":          "woho",
            "description":      "wohooooo",
            "duedate":          "2017-11-11",
    
            "timetracking":    {"originalEstimate": "4d"},
        }
    
        created = jira_inst.create_issues([test])
        pprint(created)
    
    
    if __name__ == "__main__":
        do_shit()
    

    So simple.

  • CUPS is … weird

    CUPS is the printing system developed by Apple which is now also in use on pretty much every other Linux / Unix / *BSD box there is. Unfortunately, the process of managing printers is way more painful than the process on Arch Linux. At least with my setup, which is i3 running in a weirdly crippled GNOME session.

    This is what you have to do to get CUPS working on your machine:

    $ pacman -S cups
    [...]
    $ usermod -aG sys MY_USER
    [...]
    $ systemctl enable org.cups.cupsd
    $ systemctl start org.cups.cupsd
    

    The magic is line 2 – adding your user to the sys group. If you don’t do this, you won’t be able to use the web interface at all. Which is a shame, since then you can’t do anything useful (install printers, manage jobs, etc …).

  • No password prompts for GitHub “hub” client

    GitHub has a little cli tool called “hub” which makes working with repos a bit easier (“hub pull-request” instead of a lot of clicking around on the web). Unfortunately they don’t really tell you how to prevent password inputs every time you use it.

    Unless you dig deep:

  • jq makes AWS “describe-instances” actually useful

    Just so I don’t forget 🙂

    aws ec2 describe-instances | \
      jq '.Reservations[].Instances[] | {IP: .PrivateIpAddress, ID: .InstanceId, Name: .Tags[] | select(.Key=="Name").Value}'
    
  • Linux font rendering sucks, a.k.a “Where is Boohomil”?

    For some reason, the maintainer behind the “*-infinality” packages in Arch Linux “has gone missing” for a while.

    Why is that important to me? Because infinality is a patch set to a bunch of font and rendering packages, which make fonts under Linux look SO much better than the default setup. (Yes, there are still a couple of things that Linux just absolutely cannot compete in with Mac and / or Win, and font rendering is one of them. Ubuntu does a reasonable job of this, every other distro just sucks.

    Except when you were using infinality. And now it’s defunct.

    Anyway, after experiencing the unbelievably ugly phenomenon described in here, I tried this guide here now, and it seems to fix it.

  • Elastic Beanstalk with Docker using Terraform

    I just investigate AWS Elastic Beanstalk. And I want to use terraform for this. This is what I’ve done, and how I’ve got it running. I basically do this because the docs for this are either super-long (and are still missing critical points) or super-short (and are also missing critical points), at least what I’ve found.

    This should get you up and running in very little time. You can also get all the code from a demo github repository.

    General principles

    The Architectural Overview is a good page to read to get an idea of what you’re about to do. It’s not that long.

    In short, Elastic Beanstalk runs a version of an application in an environment. So the process is: Declaring an application, defining a couple of versions and environments, and then combine one specific version with one specific environment of an app to create an actually running deployment.

    The environment is just a set of hosts configured in a special way (autoscaling & triggers, subnets, roles, etc.), whereas the application version is the info about how to deploy the containers on that environment (ports, env variables, etc.). Naturally, you think of having a DEV environment which runs “latest”, and a PROD environment which runs “stable” or so. Go crazy.

    Prerequisites & Preparation

    For the example here you need a couple of things & facts:

    • An AWS account
    • In that account, you need:
    • an S3 bucket to save your app versions
    • a VPC ID
    • subnet IDs for the instance networks
    • an IAM roles for the hosts
    • an IAM service roles elastic beanstalk. (see bottom for how to create that)
    • Terraform 🙂
    • The aws command line client

    Get started

    The files in the repository have way more parameters, but this is the basic set which should get you running (I tried once, then added all that stuff). The main.tf  file below will create the application and an environment associated with it.

    # file: main.tf
    
    resource "aws_elastic_beanstalk_application" "test" {
      name        = "ebs-test"
      description = "Test of beanstalk deployment"
    }
    
    resource "aws_elastic_beanstalk_environment" "test_env" {
      name                = "ebs-test-env"
      application         = "ebs-test"
      cname_prefix        = "mytest"
    
      # the next line IS NOT RANDOM, see "final notes" at the bottom
      solution_stack_name = "64bit Amazon Linux 2016.09 v2.5.2 running Docker 1.12.6"
    
      # There are a LOT of settings, see here for the basic list:
      # https://is.gd/vfB51g
      # This should be the minimally required set for Docker.
    
      setting {
        namespace = "aws:ec2:vpc"
        name      = "VPCId"
        value     = "${var.vpc_id}"
      }
      setting {
        namespace = "aws:ec2:vpc"
        name      = "Subnets"
        value     = "${join(",", var.subnets)}"
      }
      setting {
        namespace = "aws:autoscaling:launchconfiguration"
        name      = "IamInstanceProfile"
        value     = "${var.instance_role}"
      }
      setting {
        namespace = "aws:elasticbeanstalk:environment"
        name      = "ServiceRole"
        value     = "${var.ebs_service_role}"
      }
    
    }
    

    If you run this, at least one host and one ELB should appear in the defined subnets. Still, this is an empty environment, there’s no app running in it. If if you ask yourself, “where’s the version he talked about?” – well, it’s not in there. We didn’t create one yet. This is just the very basic platform you need to run a version of an app.

    In my source repo you can now just use the script app_config_create_and_upload.sh , followed by deploy.sh . You should be able to figure out how to use them, and they should work out of the box. But we’re here to explain, so this is what happens behind the scenes if you do this:

    1. create a file “Dockerrun.aws.json ” with the information about the service (Docker image, etc.) to deploy
    2. upload that file into an S3 bucket, packed into a ZIP file (see “final notes” below)
    3. tell Elastic Beanstalk to create a new app version using the info from that file (on S3)

    That obviously was app_config_create_and_upload.sh . The next script, deploy.sh , does this:

    1. tell EBS to actually deploy that configuration using the AWS cli.

    This is the Dockerrun.aws.json  file which describes our single-container test application:

    {
      "AWSEBDockerrunVersion": "1",
      "Image": {
        "Name": "flypenguin/test:latest",
        "Update": "true"
      },
      "Ports": [
        {
          "ContainerPort": "5000"
        }
      ],
      "Volumes": [],
      "Logging": "/var/log/flypenguin-test"
    }
    

    See “final notes” for the “ContainerPort” directive.

    I also guess you know how to upload a file to S3, so I’ll skip that. If not, look in the script. The Terraform declaration to add the version to Elastic Beanstalk looks like this: (if you used my script, a file called app_version_<VERSION>.tf  was created for you automatically with pretty much this content):

    # define elastic beanstalk app version "latest"
    resource "aws_elastic_beanstalk_application_version" "latest" {
      name        = "latest"
      application = "${aws_elastic_beanstalk_application.test_app.name}"
      description = "Version latest of app ${aws_elastic_beanstalk_application.test_app.name}"
      bucket      = "my-test-bucket-for-ebs"
      key         = "latest.zip"
    }
    

    Finally, deploying this using the AWS cli:

    $ aws elasticbeanstalk update-environment \
      --application-name test-app \
      --version-label latest \
      --environment-name test-env
    

    All done correctly, this should be it, and you should be able to access your app now under your configured address.

    Wrap up & reasoning

    My repo works, at least for me (I hope for you as well). I did not yet figure out the autoscaling, for which I didn’t have time. I will catch up in a 2nd blog post once I figured that out. First tests gave pretty weird results 🙂 .

    The reason why I did this (when I have Rancher available for me) is the auto-scaling, and the host-management. I don’t need to manage any more hosts and Docker versions and Rancher deployments just to deploy a super-simle, CPU-intensive, scaling production workload, which relies on very stable (even pretty conservative) components in that way. Also I learned something.

    Finally, after reading a lot of postings and way to much AWS docs, I am surprised how easy this thing actually is. It certainly doesnt look that way if you start reading up on it. I tried to catch the essence of the whole process in that blog post.

    Final notes & troubleshooting

    1. I have no idea what the aws_elastic_beanstalk_configuration_template  Terraform resource is for. I would like to understand it, but the documentation is rather … sparse.
    2. The solution stack name has semantic meaning. You must set something that AWS understands. This can be found out by using the following command: $ aws elasticbeanstalk list-available-solution-stacks ** ** … or on the AWS documentation. Whatever is to your liking.
    3. If you don’t specify a security group (aws:autoscaling:launchconfiguration  – “SecurityGroups “) one will be created for you automatically. That might not be convenient because this means that on “terraform destroy” this group might not be destroyed automatically. (which is just a guess, I didn’t test this)
    4. The same goes for the auto scaling group scaling rules.
    5. When trying the minimal example, be extra careful when you can’t access the service after everything is there. The standard settings seem to be: Same subnet for ELB and hosts (obviously), and public ELB (with public IPv4 address). Now, placing a public-facing ELB into an internal-only subnet does not work, right? 🙂
    6. The ZIP file: According to the docs you can only upload the JSON file (or the Dockerfile file if you build the container in the process) to S3. But the docs are not extremely clear, and Terraform did not mention this. So I am using ZIPs which works just fine.
    7. The ContainerPort is always the port the applications listens on in the container, it is not the port which is opened to the outside. That always seems to be 80 (at least for single-container deployments)

    Appendix I: Create ServiceRole IAM role

    For some reason on the first test run this did not seem to be necessary. On all subsequent runs it was, though. This is the way to create this. Sorry that I couldn’t figure out how to do this with Terraform.

    • open AWS IAM console
    • click “Create new role”
    • Step 1 – select role type: choose “AWS service role”, and under that “AWS Elastic Beanstalk”
    • Step 2 – establish trust: is skipped by the wizard after this
    • Step 3 – Attach policy: Check both policies in the table (should be “AWSElasticBeanstalkEnhancedHealth”, and “AWSElasticBeanstalkService”)
    • Step 4 – Set role name and review: Enter a role name (e.g. “aws-elasticbeanstalk-service-role”), and hit “Create role”

    Now you can use (if you chose that name) “aws-elasticbeanstalk-service-role” as your ServiceRole parameter.

    Appendix II: Sources

  • The state of things – management

    Yep, this is the challenge why I converted from a freelancer (which I still prefer as a working model) to a “normal” employed person. I am a “manager” now. Well, I just have team responsibility. And it is crazy. This brings *so* many challenges which are so amazing (cause they’re new) and exhausting (cause I need to deal with them in a completely different way).

    Here they are.

    Challenge one – team spirit. That is something I am most happy with, because our spirit is pretty high I think. And I take this on me, shamelessly, but this is also something which is deeply connected to my “leading persona”, whatever that is. And I think this one is far from perfect.

    Challenge two – training the team. I think I know some stuff, and I keep in contact with things. And I want to learn new things. Now I have to deal with maintenance shit all day, and yet want to try out new toys and stuff. This is quite complicated: On the technical side I have to think now about a way in which people can learn the most, while making sure a fuckup can not break everything. (Which it did – once, and badly). Also I have to ensure that people learn, and have fun doing it. Which is surprisingly hard, but also surprisingly cool if you see it actually working.

    Challenge three – employee interviews. I suck at it, period. I started to ask technical questions now, because before I was under the assumption that every applicant can do the job, and it’s just about how he fits in. Bad mistake. Now I learn that personal markers are also important. Which is the next thing that I need in the team, personality-wise? And am I sure of this? And does the next candidate have it? Cra-zy.

    Challenge four – managing the big picure. Or simply put – how do I make sure that the team is always up-to-date on priorities, talk to people enough, and has a good sense of when something should be “done”? And a good sense of driving it there, btw. Which is pretty much the same as

    challenge five – processes. Which process do we choose? We tried SCRUM, didn’t really work that well, so we changed it after a couple of iterations. Now we try (some sort of) Kanban, and already I am seeing transparency risks, and I need metrics. Also you often read that Kanban needs analogue ticket boards (paper, wall) – not some fancy JIRA tooling shit or so. Now what if the company policy is “log your time in tickets”? And, even most important – how do I self-manage? And the team with it? And prioritize features if they hit me like “oh in two weeks this must be done, and sorry this came in just today”?

    For me all of this is hard. I guess I am getting there, but it is a great challenge, and I love it. But soon I will need a break. And I hope some stuff is done by then.

    Update 2017-04-14: changed challenges 1&2 a bit.

  • The state of things – technology

    It’s been a while. I am currently pretty burned out, and the work keeps getting more. This is bad. But let’s talk about some challenges right now. So this is an overview of our …

    Technical state

    We’re still using Rancher. Rancher is super cool, but has the annoying habit of completely crashing about once every two months, leading to a full cluster outage for anything between 1-3 hours. Usually about 2. I still love it, but we matured in our needs, and maybe Rancher needs time to catch up (cause our needs are sometimes a bit “special”). But the Rancher team is making great progress in the right directions, and I am fully competent that Rancher will take a place in the orchestration space. Still we’re thinking about moving to K8s, simply because so much is already there.

    We’re using Prometheus for monitoring now. Rocks. Period.

    We’re still using AWS. Many of our customers would prefer Azure Germany. If you didn’t know – Azure in Germany advertises a “Data Custodian” mode, or “Data Trustee” model, not sure how to translate this and too lazy to look it up. This means that in Germany the data centers are running the true Azure stuff, but they are actually fully operated by Deutsche Telekom.

    Advantage, you ask? Easy. When the DOJ sends one of those super secure letters to Microsoft for “give me your data”, they simply forward this to Deutsche Telekom. They will probably frame it on a wall somewhere, but I don’t think they will actually give out the data. Problem solved. (We all hope :))

    We are almost done with setting the whole cloud up using Terraform. It became a really mature project over the last year, and we are super happy with the progress it’s making. Also, with Azure in the works for us (some cusomers …) this is a cool way to just manage all with the same tooling. Infrastructure as code, eh.

    We try to migrate away from Teamcity to Jenkins. We didn’t succeed yet. Too little manpower.

    But the more interesting thing is in the next post, for me at least 😉