Blog

  • Long overdue & my first Firefox addon

    Long overdue – another blog overhaul. It’s building again, now based on the blowfish theme, and if all goes well I will get a better editing experience than this <censored> “open-editor-new-file-add-front-matter” process, which is, frankly, shit. The last theme lasted two full posts. One if you don’t count the one announcing it. Wow.

    Apart from that – I start being a half-fan of “AI”. The possibilities opening up are quite nice.

    For me, that was a successor of a Firefox addon I heavily rely on (“Containerise“), which seemed no longer maintained, and also super cumbersome to use for 2026. Also, ugly.

    So here’s my first Firefox addon: “Container Dispatcher”. You can find more information now on the new project page, and my dispatcher site.

  • Postgres column type inference madness

    Long time no see.

    Anyways.

    This just cost me hours. And it’s “part I” because I just might add all the difficulties I had, which are … plenty.

    DISCLAIMER: In no way I want to bash either Postgres or SQLAlchemy here, which are incredible projects. This is just my personal experience in, well, “experiencing” the learning curve as someone who has basically no clue about all those things.

    Situation: Saving a database table, i get this error:

    Original exception was: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type double precision: "offset_key.251"
    E   LINE 1: ...:JSONB, ARRAY['scheduled-item']::VARCHAR[], NULL, 'offset_key.251',...
    E [SQL: INSERT INTO invoice_items (<<tons truncated>> offset_key, <<tons truncated>> ... offset_by_ref, cancelled_by_ref) SELEC ... 1158 characters truncated ... 7, p18, p19,
    E [parameters: {'offset_key__0': 'offset_key.251', <<again tons truncated>> ..., 'offset_key__1': nan, <<blablablablabla>>
    

    After a lot of debugging aorund I was absolutely sure that the value offset_key.251 was supposed to be written in the column offset_key. Which was perfectly fine, and which is both defined as a Mapped[String], as well as actually created as such during the test run. So the actual table column type was absolutely correct.

    After a lot of haggling with ChatGPT the solution was this (copy-pasted, because that stupid thing explains really well sometimes):

    Ah — now it’s crystal clear. Let’s unpack this carefully.

    (well, thanks and fuck you too)

    Sometimes, if the first row’s parameter is NULL, Postgres tries to infer a type. If SQLAlchemy passes ‘offset_key.251’ as a parameter in a position that Postgres already inferred as double precision (from another row, e.g., nan), you get this error.

    Databases are … weird. Shouldn’t it just use the actually defined column type?!? This seems really, really wrong (not the first time with the ChatGPTool …).

    But I digress.

    Notice: the second row is nan.

    Yes.

    In Python, float(‘nan’) is a FLOAT, so SQLAlchemy passes it as double precision.

    Postgres sees nan in row 1 → infers double precision for that column in the multi-row insert.

    Row 0 has ‘offset_key.251’ → cannot cast to float → error.

    ✅ This is a classic multi-row INSERT typing problem with Postgres.

    WTF.

    I was absolutely sure that could not be it. But I was completely out of options, so I went after the nan value – and it turns out, I read this from a Pandas DataFrame. Okay, that makes absolute sense, Pandas quite dislikes None, even in string columns.

    So I re-mapped this to None, ran the test, …

    … and it worked. Unbelievable.

    Again: HOURS.

  • New theme, again

    Every now and then I get that idiotic idea that I would succeed in doing any kind of web / theme design. So the page was in a truly desolate state for a while now, which I now freely admit, and (hopefully) corrected.

    I found the theme Hinode, which seemed kinda nice, and does all I need – for now. Maybe I’ll switch again, maybe not, but I truly and desperately hope that I will never attempt to do this myself – it’s just a waste of time.

    So here we go, an actually working theme again, have fun.

  • Hetzner dedicated server setup

    Dedicated Server

    I have a dedicated server now, from Hetzner. My old one was getting kinda crammed, and I realized that a “cloud based” server would be massively smaller at the same price – so I have now “invested” in a proper and totally oversized dedicated server.

    The goal: To host my services on it, this time using k8s and all the shiny best-practice tools we have available now. No more fiddling around. Also, no more things that can go “meh”, I spent way too much time reigning my old server back in.

    Server installation

    First of all, i basically followed this document.

    • check for RAID controllers (lspci | grep -i raid)
    • i had none, great, software RAID it is
    • wipe hard drive boot sectors, just in case (wipefs -fa /dev/nvme*n1)
    • exec installimage

    Now the question of questions – which OS? Debian, Ubuntu, or Arch Linux?

    • Debian: Rock solid, a bit dusty, rock solid, supported everywhere.
    • Ubuntu: Most popular, annoying commercial stunts, supported also everywhere.
    • Arch Linux: Least supported, most cool.

    Arch it is? … Hm, let’s opt for “rock solid”. Autoupdate should be a given (it is with Debian), so … yeah. Debian. (sadface, a bit, but still – Debian)

    An unreadable editor opens. (lightgray on lightblue? srsly??)

    Some really serious questions though.

    File system & RAID

    BTRFS? But … on top of software RAID? Or “native” BTRFS-RAID? Conveniently, the Hetzner installation script uses mdadm based RAID by default, so this is what I’m going to use.

    Regarding the file system, some research lead to …

    • don’t do BTRFS on top of SW RAID (which Hetzner is kindly setting up for me, so no BTRFS it si then), and
    • ext4 and XFS “meh”, really not much difference.

    Regarding layout, …

    • I thought long and hard and realize the only question is: Do I want to re-install on hard drive failure, or not?

    … and I don’t. So RAID 1 (“mirroring”), no LVM (because yeah, sure, i could extend transparently, but … gooood get on with it already, and KEEP IT SIMPLE!!).

    So, software-based RAID 1 (mdadm), ext4 filesystem, go.

    ##  Hetzner Online GmbH - installimage - standard config
    
    ##  HARD DISK DRIVE(S):
    DRIVE1 /dev/nvme0n1
    DRIVE2 /dev/nvme1n1
    
    ##  SOFTWARE RAID:
    # activate software RAID?  < 0 | 1 >
    SWRAID 1
    # Choose the level for the software RAID < 0 | 1 | 10 >
    SWRAIDLEVEL 1
    
    ##  HOSTNAME:
    HOSTNAME fly3
    
    ##  NETWORK CONFIG:
    # IPV4_ONLY no
    
    ##  MISC CONFIG:
    USE_KERNEL_MODE_SETTING no
    
    ##  PARTITIONS / FILESYSTEMS:
    PART swap   swap       32G
    PART /boot  ext4     1024M
    PART /      ext4       all
    
    ## your system has the following devices:
    # Disk /dev/nvme0n1: 512.12 GB (=> 476.94 GiB)
    # Disk /dev/nvme1n1: 512.12 GB (=> 476.94 GiB)
    
    ## Based on your disks and which RAID level you will choose you have
    ## the following free space to allocate (in GiB):
    # RAID  0: ~952
    # RAID  1: ~476
    
    ##  OPERATING SYSTEM IMAGE:
    IMAGE /root/.oldroot/nfs/install/../images/Debian-1205-bookworm-amd64-base.tar.gz
    

    The nice thing: If you “just do” and don’t think, the whole process it’s going to take you less than 10 minutes.

    K3S installation

    Well, for installing k3s (the k8s distribution of choice) there are were two valid options:

    Decisions, decisions, again. I did for v2 of my server, so that sounded like a no-brainer option. But I had a look at the ansible roles, and I thought “do I really need this?” … and the answer is – no. The only thing I want are managed upgrades, and this is possible either with ansible, or – cloud native 😁 – using a K8S Operator, namely Rancher’s system-upgrade-controller.

    So I am about to try k3sup, which should give me the most simple K8S cluster there is.

    Let’s go:

    k3sup \
        install
        --cluster \
        --host $HOST_IP \
        --merge \
        --user root \
        --ssh-key $HOME/.ssh/id_ed25519
    

    It’s a bit annoying that you have to actually specify your SSH private key if you don’t use RSA, but hey – otherwise it worked like a charm.

    Running: k3sup install
    2024/06/01 13:46:24 fly3.srv.flypenguin.de
    Public IP: <host-ip>
    [INFO]  Finding release for channel stable
    [INFO]  Using v1.29.5+k3s1 as release
    [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.29.5+k3s1/sha256sum-amd64.txt
    [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.29.5+k3s1/k3s
    [INFO]  Verifying binary download
    [INFO]  Installing k3s to /usr/local/bin/k3s
    [INFO]  Skipping installation of SELinux RPM
    [INFO]  Creating /usr/local/bin/kubectl symlink to k3s
    [INFO]  Creating /usr/local/bin/crictl symlink to k3s
    [INFO]  Creating /usr/local/bin/ctr symlink to k3s
    [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
    [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
    [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
    [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
    [INFO]  systemd: Enabling k3s unit
    Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
    [INFO]  systemd: Starting k3s
    Result: [INFO]  Finding release for channel stable
    [INFO]  Using v1.29.5+k3s1 as release
    [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.29.5+k3s1/sha256sum-amd64.txt
    [INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.29.5+k3s1/k3s
    [INFO]  Verifying binary download
    [INFO]  Installing k3s to /usr/local/bin/k3s
    [INFO]  Skipping installation of SELinux RPM
    [INFO]  Creating /usr/local/bin/kubectl symlink to k3s
    [INFO]  Creating /usr/local/bin/crictl symlink to k3s
    [INFO]  Creating /usr/local/bin/ctr symlink to k3s
    [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
    [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
    [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
    [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
    [INFO]  systemd: Enabling k3s unit
    [INFO]  systemd: Starting k3s
     Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
    
    Merging config into file: /Users/tm/Dev/private/tf-spaghetti/kubeconfig
    Saving file to: /Users/tm/Dev/private/tf-spaghetti/kubeconfig
    
    # Test your cluster with:
    export KUBECONFIG=/Users/tm/Dev/private/tf-spaghetti/kubeconfig
    kubectl config use-context default
    kubectl get node -o wide
    
    🚀 Speed up GitHub Actions/GitLab CI + reduce costs: https://actuated.dev
    

    It also seems the cluster is named default, which is kind of the worst name for a cluster, so after doing this I did do this:

    • vim $HOME/.kube/config and change basically everything (context, cluster & user name) to $MYCLUSTER

    Or in case you like automation, simply use yq (using vim is actually faster in that case, though):

    KC=/home/.kube/config
    yq eval -i '(.clusters[] | select(.name=="default")).cluster.server="https://localhost:56443"' $KC
    yq eval -i '(.clusters[] | select(.name=="default")).name="MYCLUSTER"' $KC
    yq eval -i '(.contexts[] | select(.name=="default")).context.cluster="MYCLUSTER"' $KC
    yq eval -i '(.contexts[] | select(.name=="default")).contest.user="MYCLUSTER"' $KC
    yq eval -i '(.contexts[] | select(.name=="default")).name="MYCLUSTER"' $KC
    yq eval -i '(.users[] | select(.name=="default")).name="MYCLUSTER"' $KC
    

    Firewall

    So to finally protect the server, let’s install and activate ufw (the “uncomplicated firewall”), following this blog post, slightly modifying based on this article, some general ufw docs from digital ocean, and an overview of flannel networking interfaces from here:

    # sudo apt-get install ufw, naturally ;)
    ufw default deny
    ufw default allow routed
    ufw allow ssh
    
    ufw allow in on cni0 from 10.42.0.0/16              comment "K3s kube-system pod traffic"
    ufw allow in on flannel.1 from 10.42.0.0/16 comment "K3s"
    
    ufw allow "www"        comment "K8s Ingress"
    ufw allow "www Secure" comment "K8s Ingress"
    

    I did not enable ufw and copy-pasted this tmux, so if my ssh connectino broke I could restart the server and would not have been locked out. (I don’t have a root password, so …)

    Controller-based upgrades

    Since this is already taking long, let’s quickly install the upgrade controller and schedule a more in-depth review later 🙂 .

    RANCH=https://github.com/rancher/system-upgrade-controller/releases/latest
    kubectl apply -f $RANCH/download/crd.yaml
    kubectl apply -f $RANCH/download/system-upgrade-controller.yaml
    

    Finally, let’s install FluxCD

    From the professional life I recently came in contact with ArgoCD, and I like it. So I had a look around, and found FluxCD – and this seems also cool.

    So I’m going to try this for myself for now. So, to deploy flux, we have to …

    • Create a cluster-state repository (in my case on GitLab)
    • Create a personal access token with full read/write permissions (there are ways to not do this, which I would definitely do the next time, but this time I was too late). Note that you do this in your Profile, and not the repository settings.
    • Execute flux bootstrap

    After I did the first two things, here I went:

    # Create a gitlab PERSONAL access token with all permissions first.
    # that is done IN YOUR PROFILE, not in the repository.
    # (it should be possible without, yet it was kinda too late)
    flux bootstrap gitlab \
      --owner=my-group \
      --repository=my-state-repo \
      --components source-controller,kustomize-controller,helm-controller,notification-controller
    

    The whole thing is … weird. On the one hand, super cool, everything done. On the other hand, super complex, meeeeh. If you look in the repository online you will notice a new folder flux-system, which we will use right now to add a “Kustomization” I want.

    • Clone out the repo,
    • modify the file flux-system/kustomization.yaml and add this content,
    • perform a git push,
    • “wait” or do this: flux reconcile ks flux-system --with-source.

    And while doing this I learned that FluxCD has no Web UI (well, now it has a 3rd party UI but …). So you can’t really see at a glance whether this worked or not. Or click “update” / “refresh” to get an idea. So … the whole “managing FluxCD” part is … annoying already. Let’s see if this becomes less annoying over time.

    So … maybe also ArgoCD?

    So, since the whole FluxCD thingy was … not so cool, I might actually co-install ArgoCD and see where it leads me.

    But for now, that’s it.

  • Create ODT files using ZIP

    So I wanted to test something, for which I needed to create or modify an ODT file. Since ODT files are just ZIPs, I thought “open, edit, zip, done” would be sufficient.

    It’s not:

    • you need to create the ODT file using a special procedure.
    • you may not include extra files into the ODT file (e.g. no “notes.txt” or sth)

    The 2nd thing you have to take care of yourself, for the first I wrote a small Mac automator script, using bin/bash as shell:

    SAVE_DIR="$PWD"
    
    while read input_dir ; do
        [[ -d "$input_dir" ]] || continue
    
        cd "$input_dir"
        touch _odt_create_log.txt
        exec > _odt_create_log.txt
        exec 2>&1
    
        set -x
        ODT_FILE="../$(basename "$input_dir").odt"
        rm -f "$ODT_FILE"
        zip      "$ODT_FILE" -0 mimetype
        zip -ruo "$ODT_FILE"    .         -x mimetype _odt_create_log.txt
        cd "$SAVE_DIR"
        set +x
    done
    
  • Hugo: index vs. _index

    I renamed ./content/about.md to ./content/index.md, and poof! … all tags gone, all blog postings gone.

    What?!

    It makes sense, if you think about it.

    index.md is a special file, which automatically marks the directory it is in as a so-called “leaf-bundle“, which then makes all the files and directories and all directories become “irrelevant” – a leaf bundle has no children.

    So, what I actually wanted to do was to rename it to _index.md, which marks ./content as what it is – a branch bundle.

    Or in short: don’t put index.md files in ./content. If you want to update your home page, use _index.md.

    Sources:

  • Flypenguin Theme …

    The blog gets a new theme: The flypenguin theme.

    The old one (Cactus) served me well, but I wanted to do something on my own. Sometimes you just have to part ways.

    This theme will be mine and mine alone, it will not be designed for re-use, and I hope I will not totally suck in designing it.

    Let’s see.

  • Getting started with Dagster

    Getting started with dagster

    This is a longer one.

    Target audience: “Somewhat” experienced Pythonistas.

    I have an Airflow-shaped problem, but – techie that I am – I wanted to try something new: Dagster. It boldly claims is is “everything that Airflow can’t be any more” (paraphrased by me ;). So – yeah, sounds cool.

    Unfortuntately, really, the docs suck quite a bit. It’s a bit like what I call the “Microsoft illness”: there’s tons of docs, but somehow they are more confusing than helpful.

    So, let’s define my things to try:

    • I want the process chain to be triggered by a queue
    • in my case, preferably AWS SQS
    • I want to process a “thing”, in that case a CSV file
    • during processing, that CSV file is split into smaller files which then should all be processed similarly
    • E.g. a ZIP file, which contains multiple CSV files, and also each CSV file could be split into several smaller based on a column value
    • The processing does not change, though.

    That’s a real-world requirement, which I think is kinda reasonable.

    The overly easy Hello Dagster really doesnt help here. So, let’s get started right “the right way ™”.

    I will follow the tutorial, and start using (almost) exactly their examples, so in the beginning you can cross-reference if you want to. I intend to diverse, though, to get my use cases done.

    I created a repository with all code from this blog post. The commits are organized after the “steps”**.

    Preparations – getting started

    # set up a project with dagster, dagit and pandas as requirements first,
    # i assume you know how to do this. then ...
    dagster
    dagster dev
    

    Side annoyance: the reason dagster dev works is the tool.dagster entry i pyproject.toml. But how to specify more than one module (arrays for module_name don’t work) is being said nowhere apparently). Or do you now have more than one module in a dagster project?

    Before starting, you should have at least this directory structure now:

    .
    ├── pyproject.toml
    ├── requirements.txt
    ├── rwt
    │   ├── __init__.py
    │   └── assets.py           # we will start editing this one.
    ├── rwt_tests
    │   ├── __init__.py
    │   └── test_assets.py
    └── setup.py
    

    Step 1 – follow the original tutorial

    Please try that first, so we can validat that the whole thing is working in general.

    So let’s get some assets.

    # rwt/assets.py
    import requests
    from dagster import asset
    
    
    @asset
    def top_story_ids():
        newstories_url = "https://hacker-news.firebaseio.com/v0/topstories.json"
        return requests.get(newstories_url).json()[:100]
    

    Execute dagster dev (if it’s not running already) and click “update” (bottom left corner) if you don’t see anything. Then you should see the assets.

    Play around with it, and then extend the code to (immediately adding metadata to the output):

    # new imports
    import pandas as pd
    from dagster import get_dagster_logger, Output, MetadataValue
    
    
    # new asset
    @asset
    def top_stories(top_story_ids):
        logger = get_dagster_logger()
        results = []
        for item_id in top_story_ids:
            url = f"https://hacker-news.firebaseio.com/v0/item/{item_id}.json"
            item = requests.get(url).json()
            results.append(item)
        df = pd.DataFrame(results)
        logger.info(f"got {len(results)} top stories")
        return Output(
            value=df,
            metadata={
                "num_records": len(results),
                "preview": MetadataValue.md(df.head().to_markdown()),
            },
        )
    

    Noteworthy about the code changes

    • The top_stories asset takes the asset top_story_ids as input. It’s an asset created out of another asset.
    • You can create a preview from and associate metadata to the output. (If you don’t want that, simply return df)

    Dagster “Definitions”

    What’s that now? Sorry, that interruption is necessary.

    All of this magic with dagster dev worked nice and well only because the Definition was correct.

    What a “Definition” you ask? It’s a way of telling dagster which “things” you have, with “things” being Jobs, Ops, Assets, and Sensors. The Dagster keyword is “code locations”.

    So, before you continue, have a look at that file: rwt/__init__.py, in which there is the default Definition, that automatically loads our assets from the example.

    Naturally, we need to adjust that later.

    Step 2 – thinking about splitting assets into multiples

    Introducing operations

    Now, the only thing we learned up to here are Assets. That’s by far not enough. Also, unfortunately, the docs present all that, but with little overview about how it all fits together.

    So, going back to the beginning: We want to split assets into multiples, that are then processed independently.

    Let me quote some docs here:

    “Asset” is Dagster’s word for an entity, external to ops, that is mutated or created by an op. An asset might be a table in a database that an op appends to, an ML model in a model store that an op overwrites, or even a slack channel that an op writes messages to.

    Op outputs often correspond to assets. For example, an op might be responsible for recreating a table, and one of its outputs might be a dataframe containing the contents of that table.

    (source)

    […] it is possible to communicate with the Dagster framework either by yielding an event, logging an event, or raising an exception.

    (source)

    Jobs are the main unit of execution and monitoring in Dagster. The core of a job is a graph of ops connected via data dependencies.

    (source)

    Asset sensors allow you to instigate runs when materializations occur.

    (source)

    That is everything but simple.

    It seems we need to …

    • have an OP, which creates an asset, which represents our ZIP file
    • have another series of OPs, that then work with this original asset, doing whatever we want them to do
    • the whole thing probably needs to be wrapped in a Job
    • and apparently we need a sensor which “senses” when we have new assets available

    So, as an initial approach, we will **write a couple of OPs that do what we want them to do, and try to trigger them somehow if we “find” a ZIP file.

    Step 3 – defining the ZIP workflow

    Note: Everything in here is simulated, you don’t need any AWS resources.

    The operations

    Let’s start with the operations first. Let’s have an operation for each “step”:

    • downloading a ZIP file from an S3 bucket, returning the download location on disk
    • unpacking the ZIP file, returning a files list
    • selecting some files from that list, returning a smaller list of files

    All of that is wrapped into a @job, that basically calls them one after another. This is how that looks like:

    # i removed rwt/assets.py
    # this file is now rwt/all.py
    
    from random import randint
    
    from dagster import (
        job,
        op,
        sensor,
        RunRequest,
    )
    
    
    @op
    def download_zip_file(context) -> str:
        # LET'S TALK ABOUT CONTEXT SOON
        zip_file_url = context.op_config["s3_key"].split("/")[-1]
        return f"/some/static/location/{zip_file_url}"
    
    
    @op
    def unpack_zip_file(local_zip) -> list[str]:
        local_files = [f"{local_zip}/{i}" for i in range(10)]
        return local_files
    
    
    @op
    def find_relevant_items(files_list: list[str]) -> list[str]:
        return files_list[0 : len(files_list) // 2]
    
    
    @job
    def process_zip_file():
        local_zip = download_zip_file()
        files_list = unpack_zip_file(local_zip)
        find_relevant_items(files_list)
    

    So, this does not look too bad, right?

    To test it, you have to adjust your Definitions:

    # file: rwt/__init__.py
    from dagster import Definitions, load_assets_from_modules
    
    from . import all
    
    all_assets = load_assets_from_modules([all])
    
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
    )
    

    Now, if you reload your code in dagster now, it should show all that.

    The Sensor

    Unfortunately, this is not run, because we don’t trigger it. To “trigger things” Dagster knows the concept of Sensors. They, well, sense things.

    In our case – a ZIP file on an S3 bucket.

    So append that code to your all.py:

    # file: rwt/all.py, changed parts
    # new import
    from dagster import sensor
    
    
    @sensor(job=process_zip_file)
    def check_for_zip_file():
        rstr = "".join([f"{randint(0, 9)}" for _ in range(6)])
        download_zip_file_config = {
            "config": {"s3_bucket": "a-bucket", "s3_key": f"/a/bucket/path/{rstr}"}
        }
        yield RunRequest(
            run_key=f"s3_file_{rstr}",
            run_config={"ops": {"download_zip_file": download_zip_file_config}},
        )
    

    … aaaand the Definitions:

    # file: rwt/__init__.py, changed parts
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
        sensors=[all.check_for_zip_file],
    )
    

    If called, that sensor will “find” a zip file in an S3 bucket.

    How that fits together

    First of all: what I do not like here is that you configure parameters for an OP in a sensor, which doesn’t call the OP, but the job. So the sensor needs to know what the OP requests. Kinda weird.

    But apart from that, fairly simple. As for concepts, those are the ones used:

    • RunRequest tells Dagster to run a Job
    • you can add information to a RunRequest
    • context is a “magic function argument” which is added by Dagster.
    • Operations can have configuration information added from the outside, which can then be found within the context object.

    So this is all that’s happening.

    Step 4 – materializing stuff

    But. We’re still not “there”. Remember we wanted to actually continue processing our split items? Yah, let’s do that now.

    The second job pipeline

    Let’s say our find_relevant_items() operation now creates assets. Each of those assets need to have their own pipeline to be processed. So we need to start a new job pipeline for each one.

    Cool?

    Cool.

    So let’s start defining the second job pipeline like this:

    # file: rwt/all.py, changed parts
    # new import
    import pandas as pd
    
    
    @op
    def load_csv_file(context) -> pd.DataFrame:
        location = context.op_config["s3_key"]
        return pd.DataFrame([[location]], columns=["location"])
    
    
    @op
    def handle_csv_file(df: pd.DataFrame) -> pd.DataFrame:
        apply_func = lambda x: zlib.crc32(x["location"].encode("utf-8"))
        tmp = df.apply(apply_func, axis=1)
        return pd.concat([df, tmp], axis=1)
    
    
    @op
    def save_csv_file(df: pd.DataFrame):
        print(df)
    
    
    @job
    def process_csv_file():
        save_csv_file(handle_csv_file(load_csv_file()))
    

    Triggering the second job pipeline

    Same problem as before – we now have the job, but no trigger. The trigger is now an asset_sensor. That is a special sensor that reacts on freshly created assets.

    Also note the different way to configure runtime information for ooperations. Above we used run_config={...}, here we use run_config=RunConfig(...). Same same, but different.

    # file: rwt/all.py, changed parts
    # new imports
    from dagster import (
        asset_sensor,
        SensorEvaluationContext,
        EventLogEntry,
        AssetKey,
        RunConfig,
    )
    
    
    @asset_sensor(asset_key=AssetKey("csv_file"), job=process_csv_file)
    def check_for_csv_file(context: SensorEvaluationContext, asset_event: EventLogEntry):
        # from here: https://is.gd/BocYt0
        # that _CANNOT_ be the way this is done?!?
        mat_metadata = asset_event.dagster_event.materialization.metadata
        load_csv_file_config = {"load_csv_file": {"s3_key": mat_metadata["s3_key"].value}}
        yield RunRequest(
            run_key=context.cursor, run_config=RunConfig(ops=load_csv_file_config)
        )
    

    Materializing assets, finally

    We now have the pipeline, and the trigger. Unfortunately we still don’t have anything … well, triggering the trigger – we still don’t actually create (“materialize”) any asset.

    To do that, as said, we change our find_relevant_items() op, to this:

    # file: rwt/all.py, changed parts
    @op
    def find_relevant_items(context, files_list: list[str]):
        relevant_items = files_list[0 : len(files_list) // 2]
        for item in relevant_items:
            relevant_part = "/".join(item.split("/")[-2:])
            context.log_event(
                AssetMaterialization(
                    asset_key="csv_file",
                    metadata={
                        "s3_bucket": "a-bucket",
                        "s3_key": f"/second/path/{relevant_part}",
                    },
                )
            )
    

    (Remember to adjust your definitions …)

    Step 5 – a short reflection

    I have no clue whether this is actually correct. It works, but I cannot imagine this part is correct:

    # file: rwt/all.py, in @asset_sensor / check_for_csv_file():
    # [...]
    mat_metadata = asset_event.dagster_event.materialization.metadata
    

    For a couple of reasons:

    • there is not a single example which I could find which uses an asset_sensor
    • dagster_event has about 1000 properties, none of them documented.
    • I would expect that the information is somewhere attached to any kind of “Asset instance”, not to the event metadata
    • metadata is a dict that holds … not strings, but some weird kind of dagster TextValue representation (whyever they need it). This is why we have that incredibly ugly mat_metadata["s3_key"].value construct.

    So … it works, but I can’t guarantee results.

  • pg_dump and pg_restore

    Since I can never remember … . Using pg_dump and pg_restore, tested with AWS RDS.

    First option, pg_dump can create SQL dumps, which can be read from psql.

    # save database; "user" can be admin user (tested) or db_user (untested)
    pg_dump    -p my-db.host   -U user      -d db   > my_backup.sql
    
    # restore database; maybe you can also use the admin user here (untested)
    psql       -p my-db.host   -U db_user   -d db   < my_backup.sql
    

    Second option, let’s now create a pg_dump “custom” dump file, and restore that with pg_restore.

    # save the db ... (add "-v" for more verbosity)
    pg_dump    -p my-db.host   -U user      -d db   -Fc -b -f   my_dump_file.pg_dump
    
    # and restore it. (add "-v" for more verbosity)
    pg_restore -p my-db.host   -U db_user   -d db   -1          my_dump_file.pg_dump
    
  • Terraform for_each and output values

    Let’s say you have conditionally created a “thing” using for_each:

    resource "aws_instance" "this" {
        for_each = var.create_me == true ? toset(["yaaay"]) : toset([])
      # ...
    }
    

    Now you want to use the instance’s IP address, lets say in the module’s “output”, just to have an example (you can use it within the module as well, anywhere you want). I always have to look into existing code to remember, so here are the multiple possible ways I’ve used so far 🙂 :

    output "instance_ip_address_method_1" {
      type        = string
      value       = element(values(aws_instance.this), 0)["public_ip"]
    }
    
    output "instance_ip_address_method_2" {
      type        = string
      value       = element([for v in values(aws_instance.this) : v.public_ip], 0)
    }
    
    output "instance_ip_address_method_3" {
      description = "method 3"
      type        = string
      value       = aws_instance.this["yaaay"].public_ip
    }