Tag: dev

  • Long overdue & my first Firefox addon

    Long overdue – another blog overhaul. It’s building again, now based on the blowfish theme, and if all goes well I will get a better editing experience than this <censored> “open-editor-new-file-add-front-matter” process, which is, frankly, shit. The last theme lasted two full posts. One if you don’t count the one announcing it. Wow.

    Apart from that – I start being a half-fan of “AI”. The possibilities opening up are quite nice.

    For me, that was a successor of a Firefox addon I heavily rely on (“Containerise“), which seemed no longer maintained, and also super cumbersome to use for 2026. Also, ugly.

    So here’s my first Firefox addon: “Container Dispatcher”. You can find more information now on the new project page, and my dispatcher site.

  • Postgres column type inference madness

    Long time no see.

    Anyways.

    This just cost me hours. And it’s “part I” because I just might add all the difficulties I had, which are … plenty.

    DISCLAIMER: In no way I want to bash either Postgres or SQLAlchemy here, which are incredible projects. This is just my personal experience in, well, “experiencing” the learning curve as someone who has basically no clue about all those things.

    Situation: Saving a database table, i get this error:

    Original exception was: (psycopg2.errors.InvalidTextRepresentation) invalid input syntax for type double precision: "offset_key.251"
    E   LINE 1: ...:JSONB, ARRAY['scheduled-item']::VARCHAR[], NULL, 'offset_key.251',...
    E [SQL: INSERT INTO invoice_items (<<tons truncated>> offset_key, <<tons truncated>> ... offset_by_ref, cancelled_by_ref) SELEC ... 1158 characters truncated ... 7, p18, p19,
    E [parameters: {'offset_key__0': 'offset_key.251', <<again tons truncated>> ..., 'offset_key__1': nan, <<blablablablabla>>
    

    After a lot of debugging aorund I was absolutely sure that the value offset_key.251 was supposed to be written in the column offset_key. Which was perfectly fine, and which is both defined as a Mapped[String], as well as actually created as such during the test run. So the actual table column type was absolutely correct.

    After a lot of haggling with ChatGPT the solution was this (copy-pasted, because that stupid thing explains really well sometimes):

    Ah — now it’s crystal clear. Let’s unpack this carefully.

    (well, thanks and fuck you too)

    Sometimes, if the first row’s parameter is NULL, Postgres tries to infer a type. If SQLAlchemy passes ‘offset_key.251’ as a parameter in a position that Postgres already inferred as double precision (from another row, e.g., nan), you get this error.

    Databases are … weird. Shouldn’t it just use the actually defined column type?!? This seems really, really wrong (not the first time with the ChatGPTool …).

    But I digress.

    Notice: the second row is nan.

    Yes.

    In Python, float(‘nan’) is a FLOAT, so SQLAlchemy passes it as double precision.

    Postgres sees nan in row 1 → infers double precision for that column in the multi-row insert.

    Row 0 has ‘offset_key.251’ → cannot cast to float → error.

    ✅ This is a classic multi-row INSERT typing problem with Postgres.

    WTF.

    I was absolutely sure that could not be it. But I was completely out of options, so I went after the nan value – and it turns out, I read this from a Pandas DataFrame. Okay, that makes absolute sense, Pandas quite dislikes None, even in string columns.

    So I re-mapped this to None, ran the test, …

    … and it worked. Unbelievable.

    Again: HOURS.

  • Getting started with Dagster

    Getting started with dagster

    This is a longer one.

    Target audience: “Somewhat” experienced Pythonistas.

    I have an Airflow-shaped problem, but – techie that I am – I wanted to try something new: Dagster. It boldly claims is is “everything that Airflow can’t be any more” (paraphrased by me ;). So – yeah, sounds cool.

    Unfortuntately, really, the docs suck quite a bit. It’s a bit like what I call the “Microsoft illness”: there’s tons of docs, but somehow they are more confusing than helpful.

    So, let’s define my things to try:

    • I want the process chain to be triggered by a queue
    • in my case, preferably AWS SQS
    • I want to process a “thing”, in that case a CSV file
    • during processing, that CSV file is split into smaller files which then should all be processed similarly
    • E.g. a ZIP file, which contains multiple CSV files, and also each CSV file could be split into several smaller based on a column value
    • The processing does not change, though.

    That’s a real-world requirement, which I think is kinda reasonable.

    The overly easy Hello Dagster really doesnt help here. So, let’s get started right “the right way ™”.

    I will follow the tutorial, and start using (almost) exactly their examples, so in the beginning you can cross-reference if you want to. I intend to diverse, though, to get my use cases done.

    I created a repository with all code from this blog post. The commits are organized after the “steps”**.

    Preparations – getting started

    # set up a project with dagster, dagit and pandas as requirements first,
    # i assume you know how to do this. then ...
    dagster
    dagster dev
    

    Side annoyance: the reason dagster dev works is the tool.dagster entry i pyproject.toml. But how to specify more than one module (arrays for module_name don’t work) is being said nowhere apparently). Or do you now have more than one module in a dagster project?

    Before starting, you should have at least this directory structure now:

    .
    ├── pyproject.toml
    ├── requirements.txt
    ├── rwt
    │   ├── __init__.py
    │   └── assets.py           # we will start editing this one.
    ├── rwt_tests
    │   ├── __init__.py
    │   └── test_assets.py
    └── setup.py
    

    Step 1 – follow the original tutorial

    Please try that first, so we can validat that the whole thing is working in general.

    So let’s get some assets.

    # rwt/assets.py
    import requests
    from dagster import asset
    
    
    @asset
    def top_story_ids():
        newstories_url = "https://hacker-news.firebaseio.com/v0/topstories.json"
        return requests.get(newstories_url).json()[:100]
    

    Execute dagster dev (if it’s not running already) and click “update” (bottom left corner) if you don’t see anything. Then you should see the assets.

    Play around with it, and then extend the code to (immediately adding metadata to the output):

    # new imports
    import pandas as pd
    from dagster import get_dagster_logger, Output, MetadataValue
    
    
    # new asset
    @asset
    def top_stories(top_story_ids):
        logger = get_dagster_logger()
        results = []
        for item_id in top_story_ids:
            url = f"https://hacker-news.firebaseio.com/v0/item/{item_id}.json"
            item = requests.get(url).json()
            results.append(item)
        df = pd.DataFrame(results)
        logger.info(f"got {len(results)} top stories")
        return Output(
            value=df,
            metadata={
                "num_records": len(results),
                "preview": MetadataValue.md(df.head().to_markdown()),
            },
        )
    

    Noteworthy about the code changes

    • The top_stories asset takes the asset top_story_ids as input. It’s an asset created out of another asset.
    • You can create a preview from and associate metadata to the output. (If you don’t want that, simply return df)

    Dagster “Definitions”

    What’s that now? Sorry, that interruption is necessary.

    All of this magic with dagster dev worked nice and well only because the Definition was correct.

    What a “Definition” you ask? It’s a way of telling dagster which “things” you have, with “things” being Jobs, Ops, Assets, and Sensors. The Dagster keyword is “code locations”.

    So, before you continue, have a look at that file: rwt/__init__.py, in which there is the default Definition, that automatically loads our assets from the example.

    Naturally, we need to adjust that later.

    Step 2 – thinking about splitting assets into multiples

    Introducing operations

    Now, the only thing we learned up to here are Assets. That’s by far not enough. Also, unfortunately, the docs present all that, but with little overview about how it all fits together.

    So, going back to the beginning: We want to split assets into multiples, that are then processed independently.

    Let me quote some docs here:

    “Asset” is Dagster’s word for an entity, external to ops, that is mutated or created by an op. An asset might be a table in a database that an op appends to, an ML model in a model store that an op overwrites, or even a slack channel that an op writes messages to.

    Op outputs often correspond to assets. For example, an op might be responsible for recreating a table, and one of its outputs might be a dataframe containing the contents of that table.

    (source)

    […] it is possible to communicate with the Dagster framework either by yielding an event, logging an event, or raising an exception.

    (source)

    Jobs are the main unit of execution and monitoring in Dagster. The core of a job is a graph of ops connected via data dependencies.

    (source)

    Asset sensors allow you to instigate runs when materializations occur.

    (source)

    That is everything but simple.

    It seems we need to …

    • have an OP, which creates an asset, which represents our ZIP file
    • have another series of OPs, that then work with this original asset, doing whatever we want them to do
    • the whole thing probably needs to be wrapped in a Job
    • and apparently we need a sensor which “senses” when we have new assets available

    So, as an initial approach, we will **write a couple of OPs that do what we want them to do, and try to trigger them somehow if we “find” a ZIP file.

    Step 3 – defining the ZIP workflow

    Note: Everything in here is simulated, you don’t need any AWS resources.

    The operations

    Let’s start with the operations first. Let’s have an operation for each “step”:

    • downloading a ZIP file from an S3 bucket, returning the download location on disk
    • unpacking the ZIP file, returning a files list
    • selecting some files from that list, returning a smaller list of files

    All of that is wrapped into a @job, that basically calls them one after another. This is how that looks like:

    # i removed rwt/assets.py
    # this file is now rwt/all.py
    
    from random import randint
    
    from dagster import (
        job,
        op,
        sensor,
        RunRequest,
    )
    
    
    @op
    def download_zip_file(context) -> str:
        # LET'S TALK ABOUT CONTEXT SOON
        zip_file_url = context.op_config["s3_key"].split("/")[-1]
        return f"/some/static/location/{zip_file_url}"
    
    
    @op
    def unpack_zip_file(local_zip) -> list[str]:
        local_files = [f"{local_zip}/{i}" for i in range(10)]
        return local_files
    
    
    @op
    def find_relevant_items(files_list: list[str]) -> list[str]:
        return files_list[0 : len(files_list) // 2]
    
    
    @job
    def process_zip_file():
        local_zip = download_zip_file()
        files_list = unpack_zip_file(local_zip)
        find_relevant_items(files_list)
    

    So, this does not look too bad, right?

    To test it, you have to adjust your Definitions:

    # file: rwt/__init__.py
    from dagster import Definitions, load_assets_from_modules
    
    from . import all
    
    all_assets = load_assets_from_modules([all])
    
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
    )
    

    Now, if you reload your code in dagster now, it should show all that.

    The Sensor

    Unfortunately, this is not run, because we don’t trigger it. To “trigger things” Dagster knows the concept of Sensors. They, well, sense things.

    In our case – a ZIP file on an S3 bucket.

    So append that code to your all.py:

    # file: rwt/all.py, changed parts
    # new import
    from dagster import sensor
    
    
    @sensor(job=process_zip_file)
    def check_for_zip_file():
        rstr = "".join([f"{randint(0, 9)}" for _ in range(6)])
        download_zip_file_config = {
            "config": {"s3_bucket": "a-bucket", "s3_key": f"/a/bucket/path/{rstr}"}
        }
        yield RunRequest(
            run_key=f"s3_file_{rstr}",
            run_config={"ops": {"download_zip_file": download_zip_file_config}},
        )
    

    … aaaand the Definitions:

    # file: rwt/__init__.py, changed parts
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
        sensors=[all.check_for_zip_file],
    )
    

    If called, that sensor will “find” a zip file in an S3 bucket.

    How that fits together

    First of all: what I do not like here is that you configure parameters for an OP in a sensor, which doesn’t call the OP, but the job. So the sensor needs to know what the OP requests. Kinda weird.

    But apart from that, fairly simple. As for concepts, those are the ones used:

    • RunRequest tells Dagster to run a Job
    • you can add information to a RunRequest
    • context is a “magic function argument” which is added by Dagster.
    • Operations can have configuration information added from the outside, which can then be found within the context object.

    So this is all that’s happening.

    Step 4 – materializing stuff

    But. We’re still not “there”. Remember we wanted to actually continue processing our split items? Yah, let’s do that now.

    The second job pipeline

    Let’s say our find_relevant_items() operation now creates assets. Each of those assets need to have their own pipeline to be processed. So we need to start a new job pipeline for each one.

    Cool?

    Cool.

    So let’s start defining the second job pipeline like this:

    # file: rwt/all.py, changed parts
    # new import
    import pandas as pd
    
    
    @op
    def load_csv_file(context) -> pd.DataFrame:
        location = context.op_config["s3_key"]
        return pd.DataFrame([[location]], columns=["location"])
    
    
    @op
    def handle_csv_file(df: pd.DataFrame) -> pd.DataFrame:
        apply_func = lambda x: zlib.crc32(x["location"].encode("utf-8"))
        tmp = df.apply(apply_func, axis=1)
        return pd.concat([df, tmp], axis=1)
    
    
    @op
    def save_csv_file(df: pd.DataFrame):
        print(df)
    
    
    @job
    def process_csv_file():
        save_csv_file(handle_csv_file(load_csv_file()))
    

    Triggering the second job pipeline

    Same problem as before – we now have the job, but no trigger. The trigger is now an asset_sensor. That is a special sensor that reacts on freshly created assets.

    Also note the different way to configure runtime information for ooperations. Above we used run_config={...}, here we use run_config=RunConfig(...). Same same, but different.

    # file: rwt/all.py, changed parts
    # new imports
    from dagster import (
        asset_sensor,
        SensorEvaluationContext,
        EventLogEntry,
        AssetKey,
        RunConfig,
    )
    
    
    @asset_sensor(asset_key=AssetKey("csv_file"), job=process_csv_file)
    def check_for_csv_file(context: SensorEvaluationContext, asset_event: EventLogEntry):
        # from here: https://is.gd/BocYt0
        # that _CANNOT_ be the way this is done?!?
        mat_metadata = asset_event.dagster_event.materialization.metadata
        load_csv_file_config = {"load_csv_file": {"s3_key": mat_metadata["s3_key"].value}}
        yield RunRequest(
            run_key=context.cursor, run_config=RunConfig(ops=load_csv_file_config)
        )
    

    Materializing assets, finally

    We now have the pipeline, and the trigger. Unfortunately we still don’t have anything … well, triggering the trigger – we still don’t actually create (“materialize”) any asset.

    To do that, as said, we change our find_relevant_items() op, to this:

    # file: rwt/all.py, changed parts
    @op
    def find_relevant_items(context, files_list: list[str]):
        relevant_items = files_list[0 : len(files_list) // 2]
        for item in relevant_items:
            relevant_part = "/".join(item.split("/")[-2:])
            context.log_event(
                AssetMaterialization(
                    asset_key="csv_file",
                    metadata={
                        "s3_bucket": "a-bucket",
                        "s3_key": f"/second/path/{relevant_part}",
                    },
                )
            )
    

    (Remember to adjust your definitions …)

    Step 5 – a short reflection

    I have no clue whether this is actually correct. It works, but I cannot imagine this part is correct:

    # file: rwt/all.py, in @asset_sensor / check_for_csv_file():
    # [...]
    mat_metadata = asset_event.dagster_event.materialization.metadata
    

    For a couple of reasons:

    • there is not a single example which I could find which uses an asset_sensor
    • dagster_event has about 1000 properties, none of them documented.
    • I would expect that the information is somewhere attached to any kind of “Asset instance”, not to the event metadata
    • metadata is a dict that holds … not strings, but some weird kind of dagster TextValue representation (whyever they need it). This is why we have that incredibly ugly mat_metadata["s3_key"].value construct.

    So … it works, but I can’t guarantee results.

  • Circular imports for Python typing annotations

    Problem:

    # ###############################################################
    # models.py
    from controllers import BookController
    
    class Book:
        def get_controller(self) -> BookController:
            return BookController(self)
    
    # ###############################################################
    # controllers.py
    from models import Book
    
    class BookController:
        def __init__(self, book: Book) -> None:
            self.book = book
    

    Do you see it? Try it out – it will generate a “circular import” Exception.

    The solution is very nicely described in Adam Johnson’s blog post (credit where credit’s due).

    Copy-pasted for myself (still, please visit Adam’s page!!, he explains it much better and with more detail), this is it:

    # ###############################################################
    # models.py
    from __future__ import annotations
    from typing import TYPE_CHECKING
    
    if TYPE_CHECKING:
        from controllers import BookController
    
    class Book:
        def get_controller(self) -> BookController:
            return BookController(self)
    
    # ###############################################################
    # controllers.py
    from __future__ import annotations
    from typing import TYPE_CHECKING
    
    if TYPE_CHECKING:
        from models import Book
    
    class BookController:
        def __init__(self, book: Book) -> None:
            self.book = book
    

    This fucking rocks. :))

  • jq snippets

    Delete any object which contains a "keyname": "keyvalue" pair anywhere in the JSON:

    cat x.json| jq 'del(..|select(objects and .keyname == "keyvalue"))'
    

    If keyname contains a dash (“-“), use one of those notations:

    cat x.json | jq 'del(..|select(objects and .["key-name"] == "keyvalue"))'
    cat x.json | jq 'del(..|select(objects and ."key-name" == "keyvalue"))'
    

    Delete all objects anywhere that contain a certain key:

    cat x.json | jq 'del(..|select(objects and .key?)'
    

    Delete a certain key from all objects everywhere:

    cat x.json | jq 'del(..|.response?)'
    

    Useful sources:

  • JavaScript ISO date with local timezone

    It seems JS has until today no native (!) solution to get an ISO 8601 date & time string in the local time zone (it’s always in UTC). (Super weird, but okay).

    I updated the solution on Stackoverflow a bit using “new” ES2017 functionality like this:

    function getISOLocalString() {
      let date = new Date();
      let tzo = -date.getTimezoneOffset();
    
      if (tzo === 0) {
        return date.toISOString();
      } else {
        let dif = tzo >= 0 ? "+" : "-";
        let pad = function (num, digits = 2) {
          return String(num).padStart(digits, "0");
        };
    
        return (
          date.getFullYear() +
          "-" +
          pad(date.getMonth() + 1) +
          "-" +
          pad(date.getDate()) +
          "T" +
          pad(date.getHours()) +
          ":" +
          pad(date.getMinutes()) +
          ":" +
          pad(date.getSeconds()) +
          dif +
          pad(tzo / 60) +
          ":" +
          pad(tzo % 60) +
          "." +
          pad(date.getMilliseconds(), 3)
        );
      }
    }
    
    console.log(getISOLocalString());
    
  • GOPATH and GOROOT

    go has some unique concepts, one of them is a system-wide (user-wide) central go directory where things “happen” (e.g. modules being downloaded, etc.). For example, if you perform hugo mod get you essentially perform go mod get, which then downloads required modules … into your $GOPATH. Or $GOROOT/something, I really don’t quite get it, and I don’t care unless it stops working. Which it did.

    You often read “Just look into $GOPATH and see if it’s there”. Unfortunately I never set the value so there is no environment variable. The default of $HOME/go does not exist on my mac. Now what?

    Thanks to 0x434d53 I now know to do this, which prints the values used by go:

    go env
    
  • Azure App Service & Python, Part I

    A.k.a. “you’ve got to be fucking kidding me”. Azure App Service sucks, as basically any part of Azure. What’s surprising this time is that terraform actually contributes to the sucking on this occasion – usually the Hashi folks do a very good job at reducing the annoyance. But maybe they’ve given up, I wouldn’t blame them.

    So what happened?

    What confused the sh*t out of me the last days is that I see tutorials (all really, really, really bad) that do use Python versions newer than 3.4 (not surprisingly). But, alas, there’s this: the site_config.python_version field of terraform’s azurerm_app_service resource. It allows the values “2.7” and “3.4” (a Python version which eol’ed in March ’19).

    After a shit ton of googling I found this, though. Turns out, it’s not even the right field to use!! To select the Python runtime you should use linux_fx_version, which nobody, nowhere, ever documented. And naive me just assumed that something with “python” in its name would be the right thing to use. Stupid, I know, we’re talking Azure here …

  • Install flutter on MacOS

    I blog so I don’t forget … this:

    First, the non-CLI actions:

    • install XCode from the App Store
    • install Flutter plugin for Visual Studio Code (if you haven’t installed VS Code, do it now, you can also use “brew” for that)
    • download the latest Flutter SDK, unzip it, and move the unzipped flutter/ folder to $HOME/Dev/frameworks (or any other location you favor)
    • add $HOME/Dev/flutter/bin and $HOME/Dev/flutter/bin/.pub-cache/bin to your path

    That’s about it for now. Then do some CLI magic:

    brew tap dart-lang/dart
    brew install dart android-studio android-sdk
    sudo gem install cocoapods
    pod setup
    sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
    sudo xcodebuild -runFirstLaunch
    flutter doctor --android-licenses
    

    After that some more UI stuff:

    • Start Android Studio. It will want to install a couple of things (Intel drivers pop up, weirdly)

    And finally: A final flutter doctor will tell you if everything is all right.

    The whole process will download about 10-12 GB of data.

  • SARS-CoV-2, a.k.a “Corona Virus”, pseudo “data science” and Twitter

    Some terminology (taken from – who would have thought it – t-online.de) and background information:

    • SARS-CoV-2 is the correct name of the virus. Everybody calls him “the Corona Virus” though, which is technically incorrect, cause there’s only the family of corona viruses – it’s a group, not a single one.
    • COVID-19 is the name of the disease. It’s an acronym made of “COrona VIrus Disease 19“. (It’s basically the same as HIV and AIDS – one is the infection, the other the actual illness).
    • It’s not the flu, but I think everybody has that down by now
    • It’s R0-value (how many people are infected by one person on average) is about 3 (source: RKI Germany)

    All of us are following the development of this very closely, cause we have no choice since we’re all basically locked at home. And even if we would go out everything’s closed. That gives us a ton of time to play DOOM Eternal (me), or write twitter bots who regularly publish the numbers of COVID-19 infections in Germany (me as well). As macabre as it is, this is very good oppotunity to start fooling around with data science. But to do this you need datasets, which are surprisingly hard to get. For the current case numbers  I found those:

    There’s probably more, but like I said – it’s surprisingly hard to find regularly updated, publicly available data sets in machine-readable form. (Also, this is the first time I go looking for this stuff, so maybe I just have no clue).

    Now back to the twitter bot. What does it do exactly? Go look for yourself, but if you are too lazy:

    • It prints the current infection rate at 8h, 12h, 16h, 20h. At the last time it will include a graph, a 1-week-forecast and a small evaluation how are stand today in comparison to the forecast which would have been made a week ago.
    • A friend helped me greatly by providing first a Holt-Winters prediction (which is live now), and then upgraded this to a more intelligent ARIMA prediction, which is still buried in a jupyter notebook and waiting for daylight. (Maybe she will write a guest post here to explain what it does and how it works – but I haven’t asked yet).
    • As for the bot’s code – it’s not (yet) public. Which is unusual for me, really. I should remedy that.

    What I learned so far:

    • It’s surprsingly easy and hard at the same time to get a Twitter developer account
    • Twitter does not permit publishing the exact same tweet multiple times in a short period of time
    • Heroku is really really nice for this, as long as you don’t need to pay for it. I would be interested in alternatives.
    • matplotlib is a lot more complicated than I expected
    • but strangely neither bokeh, nor plotly can actually export png graphics without either a separate electron app (WTF?) or a headless browser and selenium (W-T-F?!?)
    • pandas rocks, or more precisely: pandas data frames rock.
    • there does not seem to be a single properly maintained Twitter library for JavaScript, but there is at least tweepy for Python. (I mean I want to try something in JS, but honestly, if everything I find is outdated, I just stay with good old Python …)
    • The infection growth is intensely exponetial – almost a straight line on a log scale plot.

    Let’s see where this goes, and I hope you all stay healthy.