Category: Reviews & Experiments

  • Getting started with Dagster

    Getting started with dagster

    This is a longer one.

    Target audience: “Somewhat” experienced Pythonistas.

    I have an Airflow-shaped problem, but – techie that I am – I wanted to try something new: Dagster. It boldly claims is is “everything that Airflow can’t be any more” (paraphrased by me ;). So – yeah, sounds cool.

    Unfortuntately, really, the docs suck quite a bit. It’s a bit like what I call the “Microsoft illness”: there’s tons of docs, but somehow they are more confusing than helpful.

    So, let’s define my things to try:

    • I want the process chain to be triggered by a queue
    • in my case, preferably AWS SQS
    • I want to process a “thing”, in that case a CSV file
    • during processing, that CSV file is split into smaller files which then should all be processed similarly
    • E.g. a ZIP file, which contains multiple CSV files, and also each CSV file could be split into several smaller based on a column value
    • The processing does not change, though.

    That’s a real-world requirement, which I think is kinda reasonable.

    The overly easy Hello Dagster really doesnt help here. So, let’s get started right “the right way ™”.

    I will follow the tutorial, and start using (almost) exactly their examples, so in the beginning you can cross-reference if you want to. I intend to diverse, though, to get my use cases done.

    I created a repository with all code from this blog post. The commits are organized after the “steps”**.

    Preparations – getting started

    # set up a project with dagster, dagit and pandas as requirements first,
    # i assume you know how to do this. then ...
    dagster
    dagster dev
    

    Side annoyance: the reason dagster dev works is the tool.dagster entry i pyproject.toml. But how to specify more than one module (arrays for module_name don’t work) is being said nowhere apparently). Or do you now have more than one module in a dagster project?

    Before starting, you should have at least this directory structure now:

    .
    ├── pyproject.toml
    ├── requirements.txt
    ├── rwt
    │   ├── __init__.py
    │   └── assets.py           # we will start editing this one.
    ├── rwt_tests
    │   ├── __init__.py
    │   └── test_assets.py
    └── setup.py
    

    Step 1 – follow the original tutorial

    Please try that first, so we can validat that the whole thing is working in general.

    So let’s get some assets.

    # rwt/assets.py
    import requests
    from dagster import asset
    
    
    @asset
    def top_story_ids():
        newstories_url = "https://hacker-news.firebaseio.com/v0/topstories.json"
        return requests.get(newstories_url).json()[:100]
    

    Execute dagster dev (if it’s not running already) and click “update” (bottom left corner) if you don’t see anything. Then you should see the assets.

    Play around with it, and then extend the code to (immediately adding metadata to the output):

    # new imports
    import pandas as pd
    from dagster import get_dagster_logger, Output, MetadataValue
    
    
    # new asset
    @asset
    def top_stories(top_story_ids):
        logger = get_dagster_logger()
        results = []
        for item_id in top_story_ids:
            url = f"https://hacker-news.firebaseio.com/v0/item/{item_id}.json"
            item = requests.get(url).json()
            results.append(item)
        df = pd.DataFrame(results)
        logger.info(f"got {len(results)} top stories")
        return Output(
            value=df,
            metadata={
                "num_records": len(results),
                "preview": MetadataValue.md(df.head().to_markdown()),
            },
        )
    

    Noteworthy about the code changes

    • The top_stories asset takes the asset top_story_ids as input. It’s an asset created out of another asset.
    • You can create a preview from and associate metadata to the output. (If you don’t want that, simply return df)

    Dagster “Definitions”

    What’s that now? Sorry, that interruption is necessary.

    All of this magic with dagster dev worked nice and well only because the Definition was correct.

    What a “Definition” you ask? It’s a way of telling dagster which “things” you have, with “things” being Jobs, Ops, Assets, and Sensors. The Dagster keyword is “code locations”.

    So, before you continue, have a look at that file: rwt/__init__.py, in which there is the default Definition, that automatically loads our assets from the example.

    Naturally, we need to adjust that later.

    Step 2 – thinking about splitting assets into multiples

    Introducing operations

    Now, the only thing we learned up to here are Assets. That’s by far not enough. Also, unfortunately, the docs present all that, but with little overview about how it all fits together.

    So, going back to the beginning: We want to split assets into multiples, that are then processed independently.

    Let me quote some docs here:

    “Asset” is Dagster’s word for an entity, external to ops, that is mutated or created by an op. An asset might be a table in a database that an op appends to, an ML model in a model store that an op overwrites, or even a slack channel that an op writes messages to.

    Op outputs often correspond to assets. For example, an op might be responsible for recreating a table, and one of its outputs might be a dataframe containing the contents of that table.

    (source)

    […] it is possible to communicate with the Dagster framework either by yielding an event, logging an event, or raising an exception.

    (source)

    Jobs are the main unit of execution and monitoring in Dagster. The core of a job is a graph of ops connected via data dependencies.

    (source)

    Asset sensors allow you to instigate runs when materializations occur.

    (source)

    That is everything but simple.

    It seems we need to …

    • have an OP, which creates an asset, which represents our ZIP file
    • have another series of OPs, that then work with this original asset, doing whatever we want them to do
    • the whole thing probably needs to be wrapped in a Job
    • and apparently we need a sensor which “senses” when we have new assets available

    So, as an initial approach, we will **write a couple of OPs that do what we want them to do, and try to trigger them somehow if we “find” a ZIP file.

    Step 3 – defining the ZIP workflow

    Note: Everything in here is simulated, you don’t need any AWS resources.

    The operations

    Let’s start with the operations first. Let’s have an operation for each “step”:

    • downloading a ZIP file from an S3 bucket, returning the download location on disk
    • unpacking the ZIP file, returning a files list
    • selecting some files from that list, returning a smaller list of files

    All of that is wrapped into a @job, that basically calls them one after another. This is how that looks like:

    # i removed rwt/assets.py
    # this file is now rwt/all.py
    
    from random import randint
    
    from dagster import (
        job,
        op,
        sensor,
        RunRequest,
    )
    
    
    @op
    def download_zip_file(context) -> str:
        # LET'S TALK ABOUT CONTEXT SOON
        zip_file_url = context.op_config["s3_key"].split("/")[-1]
        return f"/some/static/location/{zip_file_url}"
    
    
    @op
    def unpack_zip_file(local_zip) -> list[str]:
        local_files = [f"{local_zip}/{i}" for i in range(10)]
        return local_files
    
    
    @op
    def find_relevant_items(files_list: list[str]) -> list[str]:
        return files_list[0 : len(files_list) // 2]
    
    
    @job
    def process_zip_file():
        local_zip = download_zip_file()
        files_list = unpack_zip_file(local_zip)
        find_relevant_items(files_list)
    

    So, this does not look too bad, right?

    To test it, you have to adjust your Definitions:

    # file: rwt/__init__.py
    from dagster import Definitions, load_assets_from_modules
    
    from . import all
    
    all_assets = load_assets_from_modules([all])
    
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
    )
    

    Now, if you reload your code in dagster now, it should show all that.

    The Sensor

    Unfortunately, this is not run, because we don’t trigger it. To “trigger things” Dagster knows the concept of Sensors. They, well, sense things.

    In our case – a ZIP file on an S3 bucket.

    So append that code to your all.py:

    # file: rwt/all.py, changed parts
    # new import
    from dagster import sensor
    
    
    @sensor(job=process_zip_file)
    def check_for_zip_file():
        rstr = "".join([f"{randint(0, 9)}" for _ in range(6)])
        download_zip_file_config = {
            "config": {"s3_bucket": "a-bucket", "s3_key": f"/a/bucket/path/{rstr}"}
        }
        yield RunRequest(
            run_key=f"s3_file_{rstr}",
            run_config={"ops": {"download_zip_file": download_zip_file_config}},
        )
    

    … aaaand the Definitions:

    # file: rwt/__init__.py, changed parts
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
        sensors=[all.check_for_zip_file],
    )
    

    If called, that sensor will “find” a zip file in an S3 bucket.

    How that fits together

    First of all: what I do not like here is that you configure parameters for an OP in a sensor, which doesn’t call the OP, but the job. So the sensor needs to know what the OP requests. Kinda weird.

    But apart from that, fairly simple. As for concepts, those are the ones used:

    • RunRequest tells Dagster to run a Job
    • you can add information to a RunRequest
    • context is a “magic function argument” which is added by Dagster.
    • Operations can have configuration information added from the outside, which can then be found within the context object.

    So this is all that’s happening.

    Step 4 – materializing stuff

    But. We’re still not “there”. Remember we wanted to actually continue processing our split items? Yah, let’s do that now.

    The second job pipeline

    Let’s say our find_relevant_items() operation now creates assets. Each of those assets need to have their own pipeline to be processed. So we need to start a new job pipeline for each one.

    Cool?

    Cool.

    So let’s start defining the second job pipeline like this:

    # file: rwt/all.py, changed parts
    # new import
    import pandas as pd
    
    
    @op
    def load_csv_file(context) -> pd.DataFrame:
        location = context.op_config["s3_key"]
        return pd.DataFrame([[location]], columns=["location"])
    
    
    @op
    def handle_csv_file(df: pd.DataFrame) -> pd.DataFrame:
        apply_func = lambda x: zlib.crc32(x["location"].encode("utf-8"))
        tmp = df.apply(apply_func, axis=1)
        return pd.concat([df, tmp], axis=1)
    
    
    @op
    def save_csv_file(df: pd.DataFrame):
        print(df)
    
    
    @job
    def process_csv_file():
        save_csv_file(handle_csv_file(load_csv_file()))
    

    Triggering the second job pipeline

    Same problem as before – we now have the job, but no trigger. The trigger is now an asset_sensor. That is a special sensor that reacts on freshly created assets.

    Also note the different way to configure runtime information for ooperations. Above we used run_config={...}, here we use run_config=RunConfig(...). Same same, but different.

    # file: rwt/all.py, changed parts
    # new imports
    from dagster import (
        asset_sensor,
        SensorEvaluationContext,
        EventLogEntry,
        AssetKey,
        RunConfig,
    )
    
    
    @asset_sensor(asset_key=AssetKey("csv_file"), job=process_csv_file)
    def check_for_csv_file(context: SensorEvaluationContext, asset_event: EventLogEntry):
        # from here: https://is.gd/BocYt0
        # that _CANNOT_ be the way this is done?!?
        mat_metadata = asset_event.dagster_event.materialization.metadata
        load_csv_file_config = {"load_csv_file": {"s3_key": mat_metadata["s3_key"].value}}
        yield RunRequest(
            run_key=context.cursor, run_config=RunConfig(ops=load_csv_file_config)
        )
    

    Materializing assets, finally

    We now have the pipeline, and the trigger. Unfortunately we still don’t have anything … well, triggering the trigger – we still don’t actually create (“materialize”) any asset.

    To do that, as said, we change our find_relevant_items() op, to this:

    # file: rwt/all.py, changed parts
    @op
    def find_relevant_items(context, files_list: list[str]):
        relevant_items = files_list[0 : len(files_list) // 2]
        for item in relevant_items:
            relevant_part = "/".join(item.split("/")[-2:])
            context.log_event(
                AssetMaterialization(
                    asset_key="csv_file",
                    metadata={
                        "s3_bucket": "a-bucket",
                        "s3_key": f"/second/path/{relevant_part}",
                    },
                )
            )
    

    (Remember to adjust your definitions …)

    Step 5 – a short reflection

    I have no clue whether this is actually correct. It works, but I cannot imagine this part is correct:

    # file: rwt/all.py, in @asset_sensor / check_for_csv_file():
    # [...]
    mat_metadata = asset_event.dagster_event.materialization.metadata
    

    For a couple of reasons:

    • there is not a single example which I could find which uses an asset_sensor
    • dagster_event has about 1000 properties, none of them documented.
    • I would expect that the information is somewhere attached to any kind of “Asset instance”, not to the event metadata
    • metadata is a dict that holds … not strings, but some weird kind of dagster TextValue representation (whyever they need it). This is why we have that incredibly ugly mat_metadata["s3_key"].value construct.

    So … it works, but I can’t guarantee results.

  • Backup tool selected

    I enacted an act of choosing. I chose restic.

    Reasons why

    Single binary. The contender with the even better stats was duplicati, which is available from the Ubuntu repositories, but requires 130+ MB of packages to be installed. This is a server system, so a single binary is just what I want – and very easy to duplicate if we switch operating systems. It also comes with a first-class-citizen web UI, which can be bad – cause once you need the backup tool you don’t want to have click orgies in web UIs to get stuff back.

    Somewhat good performace according to the comparison from the last post. I don’t expect to back up huge amounts of data, but I will keep an eye on the restore times. Also this is backing up data from a VM in a data center, so the link bandwidth should be sufficient in all directions.

    Best ever update process for a single binary. Just invoke restic self-update, and it does the rest. So cool.

    Nice docs – at least compared to, for example, duplicity.

    The remaining open question is whether it will back up all file metadata, which I will see shortly.

    Noteworthy

    Like all fiels in IT “backup” has the same drawbacks. On one side you have god-almighty-powerful tools for license cost that makes you scream while needing at least 5 certifications on top to operate them, on the other side you have … tar. Luckily, for backups you get at least some middle ground, namely all the tools I mentioned in my first post.

    I think all of them have certain advantages and disadvantages, while all being “good” tools for the job. The reasons I chose restic are above. Personally I would really like to try bupstash, but with backup software it does not pay to go for the “unstable” software which “might lose data”.

    Current state

    I have (blog post(s) with details forthcoming) …

    • created an S3 bucket and an IAM user with limited access to this
    • installed restic on one machine
    • initialized the restic repository in said bucket
    • created a run script which performs backups and handles service states
    • I want to update databases, and the safest way to do this is to stop them before
    • created a cron job which executes this script every 3 days
  • Backup tool evaluation

    (Update 2021-10-17: Read about the final decision here)

    Because now I need backup.

    Tools considered:

    Not considered:

    • arq – Mac/Win only, not free
    • bacula – waaay to complicated, apparently no S3 support
    • bup – no s3 support
    • bupstash – self-called alpha software, no S3 support
    • clonezilla – no S3 support, only block device backups
    • duplicacy – only free for non-commercial use

    Links:

  • Hugo migration experiences

    The goal was to switch to hugo as a static blogging engine. My reasons were:

    • super easy hosting (planned was S3 & CloudFront, it is now Netlify – which is insanely easy and I don’t need my homepage-builder image any longer)
    • no database any more, everything in git, including git push deployments
    • super quick page loads

    While trying this I found a couple of drawbacks, mainly …

    • way less themes than WordPress (well, big surprise), in by far not the same quality
    • theme switching is, if it gets improved a lot, still a PITA (right now let’s just not talk about it)
    • it is insanely hard to debug your page if something does not work
    • currently no comment engine

    It is a double-edged sword. During the migration process I often wished for a quick WYSIWYG “new blog post” editor like in WordPress, at the same time I loved the security git gave me – it’s nearly impossible to lose images, posts, whatever in your blog, it will always be “somewhere” in git. But there are already ways to get around that (it seems forestry.io is exactly that).

    So, here we are now. The state is:

    • currently, no comments available
    • cause I hate ads and don’t want to introduce ad-supported Disqus, and I also don’t want to pay for it
    • the plan is to add a self-hosted commenting engine though
    • all posts transferred, including all images (as far as I can tell)
    • a somewhat final theme (50:50 that this will be changed again)

    … and all of this took WAY longer that I expected.

  • How to try Superset

    Exactly needed like this (you have to clone the repo):

    git clone https://github.com/apache/superset.git
    cd superset
    docker-compose -f docker-compose-non-dev.yml up
    

    Thoughts:

    • superset/docker is mounted into the container at runtime and needed
    • this should not be like this
    • maybe try a small k8s deployment
  • Bluetooth Headsets

    I’ve had a few now. Still do. Here’s a review.

    Bose QuietComfort QC 35 II

    The best one, hands down. I still try to move away. Why? Several reasons.

    • Bluetooth Connectivity. I had situations where I could barely move 2m away from the laptop until people started complaining. Line-of-sight, that is. Not 2m of concrete wall or something stupid.
    • I had 3 replacements in 2 years of usage, because the connectivity got worse over time.
    • ANC works perfectly – for me. If I fill my coffee maker in my kitchen is next to my home office workplace people hear the Niagra Falls.

    The good parts:

    • ANC. Just fucking awesome.
    • Super comfy earpads.

    Bose 700

    I assume the ANC is just as good. Those are the reasons I brought them back after 3 days:

    • Same Bluetooth issues.
    • They changed the headphone design, which is just annoying. The size adjustment is continuous, which to me is a big step back because the one from the QC worked flawlessly.
    • The touch controls suck – if you use them the earframe basically acts as a drum and produces very annoying and loud sounds from your fingers.
    • Same issues with surrounding noise in calls as well – it’s good for me, but not for my communication partners.

    Sennheiser Momentum 3 Wireless

    Ugh. I am disappointed. Solidly more expensive than the Boses, and a lot of disadvantages (firmware version 3.5.1):

    • Very base-heavy sound. You have to like that. The punch is cool, but can be annoying. Overall sound is great, though.
    • Heavy!! 304g, which is about 55g heavier than Bose 700, and you feel it. It really makes a difference!!
    • Uncomfortable. The grip is very, very strong and it hurts your head. I solved that by bending up the metal arc which forms the headphone. Also the earpads are much less soft than the Bose ones, which is also uncomfortable after a short while.
    • Bluetooth connectivity is good, but Bluetooth in general is awful. After taking a call I have to restart Spotify because it just won’t play any more. Connecting can also take a long time, sometimes. Also sometimes it’s “connected”, but sound is not played back from the device, and then the connection drops again.
    • It’s “Phone 1 connected”, “Phone 2 connected”, etc. where Bose actually reads the names of your devices so you know precisely which device you’re connected to.
    • The app is just awful.
    • Firmware updates take ages (about 15-20 minutes), and the app needs at least one restart and maybe a re-pairing to pick up the change.
    • No on/off button. You switch them off by folding them (if they accidentally unfold the battery will drain), and if you want to use them as passive headphones you have to unfold them while holding the control button. Very intuitive.
    • The ANC is just not good.
    • The head size adjustment is also continuous, basicalls the over-ear-part is sliding in a metal rail, which makes them always feel slightly wrong-fitted, especially if you reduced the grip-pressure before.
    • And the aforementioned issue with the noise for the other parties in the call.

    Jabra Elite 65t

    Jabra is starting to make a lot of things right. The only issues I have are:

    • My ears are tiny. The things are a bit big and uncomfortable (those are in-ear headphones).
    • And of course the issue with the ambient noise and the communication partners.

    The good points?

    • OK ANC.

    Apple AirPods Pro

    An incredible piece of hardware, absolutely stunning. Yet:

    • Comparatively short battery time (yes, this can be an issue in long and subsequent calls).
    • I quote: “Oh I can always tell if you use your AirPods – all the ambient noise is getting so much louder then“. (I kid you not)
    • If you want to hold them while talking on speaker, you have to be very careful. If just one of them thinks it’s “in your ear” because you block one of the “ear sensors” they will grab the call. Yes this is intended behavior, still I have found this annoying at enough occasions to remember.

    The good points?

    • Amazing sound for that size.
    • Amazing ANC for that size.

    Compared to the over-ears they lose of course. That is to be expected, or I would assume an even heavier price tag.

    Now what?

    I have isolated the Jabra Evolve2 85 to try. Why?

    • Made for conferencing – it has a hardware microphone in front of your mouth. It seems tech still needs this. Also if you pull it up you go mute, which is perfect.
    • Bluetooth.
    • I wanted a mic, and so I read up on gaming headsets. Unfortunately all of them are extremely specialized for gaming, so most come with additional hardware (e.g. a dongle) and use separate connections (Bluetooth vs. custom 2.4GHz dongle) for sound vs. speech. Also they are sometimes tied to a single hardware platform, so the risk of buying an expensive good one to use on the iPhone is substantial.
    • Apparently a whole apparatus of 11 (!) microphones for speech vs. noise computation.
    • Supposedly very good sound quality.
    • ANC is supposed to be good.
    • They come in beige, enough with boring black tech gear. (all right this is by no means a technical feature …)

    If those suck as well … I’m currently out of options.