Category: Snippets

  • SARS-CoV-2 (β€œCorona”) Data Sources and APIs

    Last update: 2020-04-01

    You just have to write about something, and it changes. In this case, that’s pretty good πŸ™‚ . So here’s a list of APIs and data sources for the CoV-2 pandemia.

    APIs:

    Data sources:

    Dashboards and visualizations:

  • Configure Python on Windows

    All right, I have a Windows machine. It’s a PITA, but it’s here. And for some reason I started doing some Python testing on it. So this is how I managed to do it:

    Preparation:

    • Install python with choco (choco install -y python)
    • Run PowerShell as Administrator
      • Execute Set-ExecutionPolicy -ExecutionPolicy Unrestricted (we’ll see why in a very short time)

    Now to code it’s pretty similar to *NIX:

    • Create your code folder
    • Set up a python venv (python -m venv .env)
    • In VS Code, choose this interpreter

    So why the PowerShell stuff? Cause to activate the environment VS Code needs to execute a .ps1 script. Which it can’t, cause “executing scripts is disabled on this machine”, which seems to be the default setting.

    All in all, surprisingly straightforward. And I just noticed even the *NIX keyboard shortcuts (CTRL-A, CTRL-K, for example) work in the terminal window now. Crazy.

  • Use autohotkey for text expansion on Windows

    Under macOS I use TextExpander, under Windows there’s the fantastic AutoHotkey. One of the few softwares I can’t live without.

    This is my default configuration:

    ; ---------- "auto reload" ----------
    
    FileGetTime ScriptStartModTime, %A_ScriptFullPath%
    SetTimer CheckReload, 1000, 0x7FFFFFFF ; ms & priority
    
    ; from here: https://stackoverflow.com/a/45488494
    CheckReload() {
        global ScriptStartModTime
        FileGetTime curModTime, %A_ScriptFullPath%
        If (curModTime <> ScriptStartModTime) {
            Loop
            {
                reload
                Sleep 300 ; ms
                MsgBox 0x2, %A_ScriptName%, Reload failed. ; 0x2 = Abort/Retry/Ignore
                IfMsgBox Abort
                    ExitApp
                IfMsgBox Ignore
                    break
            } ; loops reload on "Retry"
        }
    }
    
    
    ; ---------- actual content here ----------
    
    ; removed all my email address shortcuts ...
    
    :*:-dt::
      ; from here: https://is.gd/3u6MKQ
      Send, %A_YYYY%-%A_MM%-%A_DD%
      Return
    
    :*://ts::
      Send, %A_YYYY%%A_MM%%A_DD%_%A_Hour%%A_Min%%A_Sec%
      Return
    
  • Check MK container/k8s deployment

    In the company everybody seems to love Check MK. Me? Not so much, but a better alternative costs time and effort, both resources we don’t really have right now. Yet there’s a positive thing about it – because there’s an official docker container. Since I already coded a helm chart for stateful single container softwares (which I personally find super useful), I just wrote a Check MK YAML and installed it on my K8S cluster.

    And then nothing worked. Turns out, Apache – which is used in that very strange “Open Monitoring Distribution” which Check MK seems to have been at one point – has a slightly sub-optimal configuration for running in a container behind a load balancer using cert-manager.

    In short, you connect to the load balancer using “cmk.my.domain”, and it redirects you to the container port, which to itself is “https://cmk.my.domain:5000/” and just wrong. Which brings me to the question if anybody has ever tried to run the Check MK container in a k8s cluster or behind a load balancer, which brings me to the question that I’d rather use software which actively embraces that, which brings me to the question WHICH ONE?!? which brings us back to “no resources, no time”.

    So, bad luck, Check MK it is. But what about the bug? Reporting it you get an email “DONT CALL US – WE CALL YOU (and we probably won’t)“, with a ticket ID but no link. So probably no help here. So I “forked” the container, fooled around with it, and found a solution. The “fixed” container is now available on docker hub (sources on GitHub) and running nicely in our internal cluster. Let’s see which hidden bugs I have introduced πŸ˜‰ . The stasico-Helm-YAML file I used to deploy Check MK in K8S is also available.

    TL;DR
  • cert-manager too old …

    Today cert-manager stopped issuing certificates, and all requests said “insecure website”. Uncool, since this affected our Confluence and our sign-in mechanism. So let’s find out what was happening, right? Turns out cert-manager considered itself “too old” (“your ACME client is too old”, literally) and wanted to be updated.

    So far, so good. Just perform helm update cert-manager cert-manager, right?

    Wrong.

    • First, I had to upgrade to helm3. All right, I could have used helm2, but helm3 was already on here, and it seemed easy. That went (fairly) easy.
    • Then I wanted to upgrade cert-manager. Turns out for that I actually had to upgrade the running k8s version 1.12.x to 1.13.x … otherwise I’d get errors from the helm chart. That just took ages cause AKS is a bit slow.
    • Finally done I wanted to upgrade cert-manager. Until I realized a lot of stateful pods were stuck in “initialization”. Turns out that AKS had issues moving the volumes around, I still don’t know why. (Did I mention I just don’t like pretty much anything about Azure? It’s just so incredibly cumbersome to use, and nothing is where you expect it). So I had to manually mount the volumes on the host the Pod was currently on and have an open TODO now, which just sucks.
    • Finally done I wanted to upgrade cert-manager. The upgrade went just peachy, until I realized that … nothing happened. Turns out they changed pretty much all API versions and annotation keys. So I had to rewrite / upgrade all ingress annotations, update the ClusterIssuer resources and delete the now obsolete former K8S CRDs.

    And just like that I had my certificates back. Wasn’t that easy? πŸ˜€

  • The 6 ways of returning data from an Azure Function

    Why have one if you can have many? Well, beats me, but I thought I’d collect them here so I have my personal reference. Disclaimer: This is only for JavaScript based functions (most of the examples on the MS pages are C#).

    So, most of them have to do with the file function.jsonΒ  file:

    {
      "bindings": [
        {
          "type": "httpTrigger",
          "name": "req",
          "and": "so on ..."
        },
        {
          "type": "http",
          "name": "thisIsAnOutputBinding",
          "direction": "out"
        },
        {
          "type": "someOtherType",
          "name": "anotherBinding",
          "direction": "out"
        }
      ]
    }
    

    More precisely, with the “out” type binding that is defined in the 2nd “bindings” object. The “out” binding has a “name” property, which is basically relevant for all of the methods.

    !!!!! WARNING !!!!! – It seems the name of the binding property must be camelCase**. At least I consistently get “invalid binding” errors when I use snake_case**

    Method 1 – context property

    The context object has one property per defined “out” binding. So if our property is named “thisIsAnOutputBinding”, our result property is context.thisIsAnOutputBinding , or as MicroSoft puts it:

    Outputs (bindings of direction === "out" ) can be written to by a function in a number of ways. In all cases, the name property of the binding as defined in function.json corresponds to the name of the object member written to in your function.

    Simply put, you just assign the value to a contextΒ  member and you’re done:

    module.exports = function (context, req) {
      context.thisIsAnOutputBinding = {
        my: "return value",
        "in this case": "an object",
      };
      // contect.done() is only needed for synchronous functions context.done();
    };
    

    Source here.

    Method 2 – return an object (async functions only)

    With the bindings given above, you can assign the values to the context property, but you can also return an object whose keys correspond to the binding names:

    module.exports = async function (context) {
      return { thisIsAnOutputBinding: 42, anotherBinding: 43 };
      // of course, instead of 42/43 you can put any js object here.
    };
    

    Source here.

    Method 3 – using context.bindings

    This is basically a variant of method 1. Why does it exist? No one knows. Apparently the context object has a .bindingsΒ  property, which in turn again has properties which name-match the defined out bindings. So this is another possibility, and I think the returnΒ  is unnecessary:

    module.exports = async function (context) {
      let retMsg = "Hello, world!";
      context.bindings.thisIsAnOutputBinding = {
        body: retMsg,
      };
      context.bindings.anotherBinding = retMsg;
      return;
    };
    

    Source here.

    Method 4 – using context.done()

    If you’re using a sync method, you can’t return an object, but you can call context.done(err, obj)Β .

    module.exports = async function (context) {
      context.done(null, {
        thisIsAnOutputBinding: { text: "hello there, world", anotherBinding: 42 },
      });
    };
    

    Source here.

    Method 5 – just return it already (only async functions)

    Probably someone said “Well, if I only have one output binding, why should I explicitly address it?”, which is a valid thought. So another way to return data was invented.

    For this the configuration in function.json looks a bit different:

    {
      "bindings": [
        { "type": "someType", "name": "req", "direction": "in" },
        { "type": "someOutType", "direction": "out", "name": "$return" }
      ]
    }
    

    You have to set exactly one “out” type binding, and the name must be “$return”. Then you can do this:

    module.exports = async function (context, req) {
      return "woohooo!!";
    };
    

    Source here.

    Methods 6 – special for http outputs

    You thought we were done? Noooooo. For http methods, you can diretly return the body object which is then used to create the body

    // actually NO IDEA if this works sync, async or both
    // my guess is: sync, because "context.done() is implicitly called"
    module.exports = async function (context, req) {
      rv = { body: "<html/>", status: 201 };
      context.res.send(rv);
    };
    

    Source here.

    Summary

    Let’s just say it’s a mess, and too many ways ruin the map.

    It works though.

  • Win10 & Veracrypt & systemd-boot

    There are some things seemingly nobody does. For example, …

    • double-booting Win10 and Linux
    • on an UEFI System
    • while the Win10 Partition is encrypted using VeraCrypt.

    Yes, it’s a complex scenario, but since MS in all of his (money-grabbing) wisdom does not include BitLocker in Win10 Home, this is a necessary precaution. I’ll not go over the installation of both systems (pretty straightforward, and Arch Linux has – as always – a nice Wiki entry about it).

    Unfortunately, Win10 likes to break its own boot manager on updates, which is very scary (“Your Windows partition is damaged”), and super annoying, but I think I got the solution now.

    So, the Linux-based (of course) solution for Windows 10 and VeraCrypt is:

    # esp partition - /loader/entries/winvera.conf
    title Windows 10 VeraCrypt
    efi /EFI/VeraCrypt/DcsBoot.efi
    

    This is in fact all you need to do. Now, if Windows fucks up its own boot loader, it seems systemd-boot just ignores everything, loads the correct VeraCrypt bootloader (as it is supposed to be), and all is well.

    It can happen though that Windows places its own boot manager back in front of systemd-boot again, so it’s used as the default one. Then use one of the methods described here, and you should be fine. (This did not happen to me, it always used the correct boot manager but fucked up Windows boot)

  • Windows after 13 years – and nothing changed

    I have a Windows PC again, after about 13 years of abstinence and never looking back. (Why? Gaming. Once in 13 years is OK I guess).

    And nothing changed.

    Step 1: Uninstalling crap

    Uninstall those things from the Windows menu: Candy Crush, Cooking Fever, and three others I forgot to document. It’s a pristine ISO install, nothing from a vendor – I bought components myself, and I assembled myself. So this is Windows and Windows alone that’s to blame.

    And don’t forget all the crap which is in the Windows menu tiles – XBox & co, I mean you. (Removed about 7 super useless things here alone).

    Step 2: change mouse wheel direction

    Step 2: Change mouse wheel direction (sorry, Mac spoiled me). I can configure anything and everything in Windows – not that. Google helps, and I have to – of course – navigate the registry to find keys that look like this:

    HKLM\SYSTEM\CurrentControlSet\Enum\HID\...
    ... VID_046D&PID_C53D&MI_01&COL01\9&12BDBF6B&0&0000\...
    ... DeviceParameters\FlipFlopWheel
    

    (Set this to 1, and get the “VID_0…” whatever string from the “Advanced Settings” of the mouse properties dialogue. Brainfuck.

    Step 3: Disable cortana

    Oh yeah, disabling Cortana is almost easy (set this to 0):

    HKLM\SOFTWARE\Policies\Microsoft\Windows\...
    ... Windows Search\AllowCortana
    

    Step 4: Remove contacts icon from taskbar

    Removing the stupid “Contacts” icon on the task bar is super simple in contrast: Right-click, and uncheck “Show contacts”. Yay!

    Step 5: Re-login / Reboot

    Where the fuck can I log out?!

    Oh right, click the start menu, immediately see the unobtrusive grey junk icon which is supposed to be me in the leftmost area on top of all the other nondescriminate icons, click it, and see the menu pop up which offers to “log out”. How could I miss this.

    Summary

    Well, this is not all. This is just what I did today, after already tuning the system a while ago. In contrast Mac: Unpack, open (Laptops only here), start working. No candy crush removal necessary.

  • Powershell, O365 & Teams PSTN calling

    Unfortunately you need a Windows system to administer Office 365 with PowerShell. It’s only API calls, but it’s not (yet, hopefully) migrated to .NET Core. So Mac & Linux users are out of luck, although .NET Core should be more than capable to do this.

    Anyhow.

    If you want to administer Teams with PowerShell, you … are in trouble. It’s barely documented, and it sucks. Those are the steps to be done:

    > $sess = New-CsOnlineSession ...
    > Import-Module SkypeOnlineConnector ...
    > Import-PSSession $sess
    

    … and this should be it. Now all the PowerShell commands for Teams (in my case: Grant-CsTeamsUpgradePolicy) should be available.

  • Misc Django I – forms

    Custom form errors

    If you want to validate something in the view, and return with a custom error message in the same form, you can use the [Form.add_error][1](fieldname, errorstring) method. And then, of course, return to the previous template.

    class MyView(View):
        def get(self, request):
            data = form.cleaned_data
            if len(res) > 0:
                form.add_error( 'login', "Diese Personalnummer existiert bereits!")
            return render(request, 'my_template.html', {'form': form})
    

    Dynamic choice fields in forms

    You want a form which fills its choice field from the database? And if the database changes, if you reload the page, the form should change as well? Of course! Django got you covered.

    class UserForm(forms.Form):
        def __init__(self, *args, **kwargs):
            super(UserForm, self).__init__(*args, **kwargs)
            self.fields['site'] = forms.ModelChoiceField( label="Site", queryset=Site.objects.all().order_by('name'), )
            for field in ('department', 'office', 'phone'):
                self.fields.move_to_end(field)
    
        login = forms.CharField(label="Login")
        email = forms.EmailField(label="Email")
        site = None # this is set in __init__() :)
        department = forms.CharField(label="Department")
        office = forms.CharField(label="Office")
        phone = forms.CharField(label="Phone")
    

    … now, why the “for field in (‘department’ …)” line you ask?

    Simple. The fields dict is an OrderedDict. If you replace a field it is appended to the end again. So in the form the “Site” input box would be displayed last, although it makes more sense to display it where it is in the original definition.

    Using .move_to_end() you can re-adjust this. If someone knows a better method … feel free to tell me.

    (Sources: here)