Tag: data processing

  • Getting started with Dagster

    Getting started with dagster

    This is a longer one.

    Target audience: “Somewhat” experienced Pythonistas.

    I have an Airflow-shaped problem, but – techie that I am – I wanted to try something new: Dagster. It boldly claims is is “everything that Airflow can’t be any more” (paraphrased by me ;). So – yeah, sounds cool.

    Unfortuntately, really, the docs suck quite a bit. It’s a bit like what I call the “Microsoft illness”: there’s tons of docs, but somehow they are more confusing than helpful.

    So, let’s define my things to try:

    • I want the process chain to be triggered by a queue
    • in my case, preferably AWS SQS
    • I want to process a “thing”, in that case a CSV file
    • during processing, that CSV file is split into smaller files which then should all be processed similarly
    • E.g. a ZIP file, which contains multiple CSV files, and also each CSV file could be split into several smaller based on a column value
    • The processing does not change, though.

    That’s a real-world requirement, which I think is kinda reasonable.

    The overly easy Hello Dagster really doesnt help here. So, let’s get started right “the right way ™”.

    I will follow the tutorial, and start using (almost) exactly their examples, so in the beginning you can cross-reference if you want to. I intend to diverse, though, to get my use cases done.

    I created a repository with all code from this blog post. The commits are organized after the “steps”**.

    Preparations – getting started

    # set up a project with dagster, dagit and pandas as requirements first,
    # i assume you know how to do this. then ...
    dagster
    dagster dev
    

    Side annoyance: the reason dagster dev works is the tool.dagster entry i pyproject.toml. But how to specify more than one module (arrays for module_name don’t work) is being said nowhere apparently). Or do you now have more than one module in a dagster project?

    Before starting, you should have at least this directory structure now:

    .
    ├── pyproject.toml
    ├── requirements.txt
    ├── rwt
    │   ├── __init__.py
    │   └── assets.py           # we will start editing this one.
    ├── rwt_tests
    │   ├── __init__.py
    │   └── test_assets.py
    └── setup.py
    

    Step 1 – follow the original tutorial

    Please try that first, so we can validat that the whole thing is working in general.

    So let’s get some assets.

    # rwt/assets.py
    import requests
    from dagster import asset
    
    
    @asset
    def top_story_ids():
        newstories_url = "https://hacker-news.firebaseio.com/v0/topstories.json"
        return requests.get(newstories_url).json()[:100]
    

    Execute dagster dev (if it’s not running already) and click “update” (bottom left corner) if you don’t see anything. Then you should see the assets.

    Play around with it, and then extend the code to (immediately adding metadata to the output):

    # new imports
    import pandas as pd
    from dagster import get_dagster_logger, Output, MetadataValue
    
    
    # new asset
    @asset
    def top_stories(top_story_ids):
        logger = get_dagster_logger()
        results = []
        for item_id in top_story_ids:
            url = f"https://hacker-news.firebaseio.com/v0/item/{item_id}.json"
            item = requests.get(url).json()
            results.append(item)
        df = pd.DataFrame(results)
        logger.info(f"got {len(results)} top stories")
        return Output(
            value=df,
            metadata={
                "num_records": len(results),
                "preview": MetadataValue.md(df.head().to_markdown()),
            },
        )
    

    Noteworthy about the code changes

    • The top_stories asset takes the asset top_story_ids as input. It’s an asset created out of another asset.
    • You can create a preview from and associate metadata to the output. (If you don’t want that, simply return df)

    Dagster “Definitions”

    What’s that now? Sorry, that interruption is necessary.

    All of this magic with dagster dev worked nice and well only because the Definition was correct.

    What a “Definition” you ask? It’s a way of telling dagster which “things” you have, with “things” being Jobs, Ops, Assets, and Sensors. The Dagster keyword is “code locations”.

    So, before you continue, have a look at that file: rwt/__init__.py, in which there is the default Definition, that automatically loads our assets from the example.

    Naturally, we need to adjust that later.

    Step 2 – thinking about splitting assets into multiples

    Introducing operations

    Now, the only thing we learned up to here are Assets. That’s by far not enough. Also, unfortunately, the docs present all that, but with little overview about how it all fits together.

    So, going back to the beginning: We want to split assets into multiples, that are then processed independently.

    Let me quote some docs here:

    “Asset” is Dagster’s word for an entity, external to ops, that is mutated or created by an op. An asset might be a table in a database that an op appends to, an ML model in a model store that an op overwrites, or even a slack channel that an op writes messages to.

    Op outputs often correspond to assets. For example, an op might be responsible for recreating a table, and one of its outputs might be a dataframe containing the contents of that table.

    (source)

    […] it is possible to communicate with the Dagster framework either by yielding an event, logging an event, or raising an exception.

    (source)

    Jobs are the main unit of execution and monitoring in Dagster. The core of a job is a graph of ops connected via data dependencies.

    (source)

    Asset sensors allow you to instigate runs when materializations occur.

    (source)

    That is everything but simple.

    It seems we need to …

    • have an OP, which creates an asset, which represents our ZIP file
    • have another series of OPs, that then work with this original asset, doing whatever we want them to do
    • the whole thing probably needs to be wrapped in a Job
    • and apparently we need a sensor which “senses” when we have new assets available

    So, as an initial approach, we will **write a couple of OPs that do what we want them to do, and try to trigger them somehow if we “find” a ZIP file.

    Step 3 – defining the ZIP workflow

    Note: Everything in here is simulated, you don’t need any AWS resources.

    The operations

    Let’s start with the operations first. Let’s have an operation for each “step”:

    • downloading a ZIP file from an S3 bucket, returning the download location on disk
    • unpacking the ZIP file, returning a files list
    • selecting some files from that list, returning a smaller list of files

    All of that is wrapped into a @job, that basically calls them one after another. This is how that looks like:

    # i removed rwt/assets.py
    # this file is now rwt/all.py
    
    from random import randint
    
    from dagster import (
        job,
        op,
        sensor,
        RunRequest,
    )
    
    
    @op
    def download_zip_file(context) -> str:
        # LET'S TALK ABOUT CONTEXT SOON
        zip_file_url = context.op_config["s3_key"].split("/")[-1]
        return f"/some/static/location/{zip_file_url}"
    
    
    @op
    def unpack_zip_file(local_zip) -> list[str]:
        local_files = [f"{local_zip}/{i}" for i in range(10)]
        return local_files
    
    
    @op
    def find_relevant_items(files_list: list[str]) -> list[str]:
        return files_list[0 : len(files_list) // 2]
    
    
    @job
    def process_zip_file():
        local_zip = download_zip_file()
        files_list = unpack_zip_file(local_zip)
        find_relevant_items(files_list)
    

    So, this does not look too bad, right?

    To test it, you have to adjust your Definitions:

    # file: rwt/__init__.py
    from dagster import Definitions, load_assets_from_modules
    
    from . import all
    
    all_assets = load_assets_from_modules([all])
    
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
    )
    

    Now, if you reload your code in dagster now, it should show all that.

    The Sensor

    Unfortunately, this is not run, because we don’t trigger it. To “trigger things” Dagster knows the concept of Sensors. They, well, sense things.

    In our case – a ZIP file on an S3 bucket.

    So append that code to your all.py:

    # file: rwt/all.py, changed parts
    # new import
    from dagster import sensor
    
    
    @sensor(job=process_zip_file)
    def check_for_zip_file():
        rstr = "".join([f"{randint(0, 9)}" for _ in range(6)])
        download_zip_file_config = {
            "config": {"s3_bucket": "a-bucket", "s3_key": f"/a/bucket/path/{rstr}"}
        }
        yield RunRequest(
            run_key=f"s3_file_{rstr}",
            run_config={"ops": {"download_zip_file": download_zip_file_config}},
        )
    

    … aaaand the Definitions:

    # file: rwt/__init__.py, changed parts
    defs = Definitions(
        assets=all_assets,
        jobs=[all.process_zip_file],
        sensors=[all.check_for_zip_file],
    )
    

    If called, that sensor will “find” a zip file in an S3 bucket.

    How that fits together

    First of all: what I do not like here is that you configure parameters for an OP in a sensor, which doesn’t call the OP, but the job. So the sensor needs to know what the OP requests. Kinda weird.

    But apart from that, fairly simple. As for concepts, those are the ones used:

    • RunRequest tells Dagster to run a Job
    • you can add information to a RunRequest
    • context is a “magic function argument” which is added by Dagster.
    • Operations can have configuration information added from the outside, which can then be found within the context object.

    So this is all that’s happening.

    Step 4 – materializing stuff

    But. We’re still not “there”. Remember we wanted to actually continue processing our split items? Yah, let’s do that now.

    The second job pipeline

    Let’s say our find_relevant_items() operation now creates assets. Each of those assets need to have their own pipeline to be processed. So we need to start a new job pipeline for each one.

    Cool?

    Cool.

    So let’s start defining the second job pipeline like this:

    # file: rwt/all.py, changed parts
    # new import
    import pandas as pd
    
    
    @op
    def load_csv_file(context) -> pd.DataFrame:
        location = context.op_config["s3_key"]
        return pd.DataFrame([[location]], columns=["location"])
    
    
    @op
    def handle_csv_file(df: pd.DataFrame) -> pd.DataFrame:
        apply_func = lambda x: zlib.crc32(x["location"].encode("utf-8"))
        tmp = df.apply(apply_func, axis=1)
        return pd.concat([df, tmp], axis=1)
    
    
    @op
    def save_csv_file(df: pd.DataFrame):
        print(df)
    
    
    @job
    def process_csv_file():
        save_csv_file(handle_csv_file(load_csv_file()))
    

    Triggering the second job pipeline

    Same problem as before – we now have the job, but no trigger. The trigger is now an asset_sensor. That is a special sensor that reacts on freshly created assets.

    Also note the different way to configure runtime information for ooperations. Above we used run_config={...}, here we use run_config=RunConfig(...). Same same, but different.

    # file: rwt/all.py, changed parts
    # new imports
    from dagster import (
        asset_sensor,
        SensorEvaluationContext,
        EventLogEntry,
        AssetKey,
        RunConfig,
    )
    
    
    @asset_sensor(asset_key=AssetKey("csv_file"), job=process_csv_file)
    def check_for_csv_file(context: SensorEvaluationContext, asset_event: EventLogEntry):
        # from here: https://is.gd/BocYt0
        # that _CANNOT_ be the way this is done?!?
        mat_metadata = asset_event.dagster_event.materialization.metadata
        load_csv_file_config = {"load_csv_file": {"s3_key": mat_metadata["s3_key"].value}}
        yield RunRequest(
            run_key=context.cursor, run_config=RunConfig(ops=load_csv_file_config)
        )
    

    Materializing assets, finally

    We now have the pipeline, and the trigger. Unfortunately we still don’t have anything … well, triggering the trigger – we still don’t actually create (“materialize”) any asset.

    To do that, as said, we change our find_relevant_items() op, to this:

    # file: rwt/all.py, changed parts
    @op
    def find_relevant_items(context, files_list: list[str]):
        relevant_items = files_list[0 : len(files_list) // 2]
        for item in relevant_items:
            relevant_part = "/".join(item.split("/")[-2:])
            context.log_event(
                AssetMaterialization(
                    asset_key="csv_file",
                    metadata={
                        "s3_bucket": "a-bucket",
                        "s3_key": f"/second/path/{relevant_part}",
                    },
                )
            )
    

    (Remember to adjust your definitions …)

    Step 5 – a short reflection

    I have no clue whether this is actually correct. It works, but I cannot imagine this part is correct:

    # file: rwt/all.py, in @asset_sensor / check_for_csv_file():
    # [...]
    mat_metadata = asset_event.dagster_event.materialization.metadata
    

    For a couple of reasons:

    • there is not a single example which I could find which uses an asset_sensor
    • dagster_event has about 1000 properties, none of them documented.
    • I would expect that the information is somewhere attached to any kind of “Asset instance”, not to the event metadata
    • metadata is a dict that holds … not strings, but some weird kind of dagster TextValue representation (whyever they need it). This is why we have that incredibly ugly mat_metadata["s3_key"].value construct.

    So … it works, but I can’t guarantee results.

  • How to try Superset

    Exactly needed like this (you have to clone the repo):

    git clone https://github.com/apache/superset.git
    cd superset
    docker-compose -f docker-compose-non-dev.yml up
    

    Thoughts:

    • superset/docker is mounted into the container at runtime and needed
    • this should not be like this
    • maybe try a small k8s deployment
  • SARS-CoV-2 (“Corona”) Data Sources and APIs

    Last update: 2020-04-01

    You just have to write about something, and it changes. In this case, that’s pretty good 🙂 . So here’s a list of APIs and data sources for the CoV-2 pandemia.

    APIs:

    Data sources:

    Dashboards and visualizations:

  • SARS-CoV-2, a.k.a “Corona Virus”, pseudo “data science” and Twitter

    Some terminology (taken from – who would have thought it – t-online.de) and background information:

    • SARS-CoV-2 is the correct name of the virus. Everybody calls him “the Corona Virus” though, which is technically incorrect, cause there’s only the family of corona viruses – it’s a group, not a single one.
    • COVID-19 is the name of the disease. It’s an acronym made of “COrona VIrus Disease 19“. (It’s basically the same as HIV and AIDS – one is the infection, the other the actual illness).
    • It’s not the flu, but I think everybody has that down by now
    • It’s R0-value (how many people are infected by one person on average) is about 3 (source: RKI Germany)

    All of us are following the development of this very closely, cause we have no choice since we’re all basically locked at home. And even if we would go out everything’s closed. That gives us a ton of time to play DOOM Eternal (me), or write twitter bots who regularly publish the numbers of COVID-19 infections in Germany (me as well). As macabre as it is, this is very good oppotunity to start fooling around with data science. But to do this you need datasets, which are surprisingly hard to get. For the current case numbers  I found those:

    There’s probably more, but like I said – it’s surprisingly hard to find regularly updated, publicly available data sets in machine-readable form. (Also, this is the first time I go looking for this stuff, so maybe I just have no clue).

    Now back to the twitter bot. What does it do exactly? Go look for yourself, but if you are too lazy:

    • It prints the current infection rate at 8h, 12h, 16h, 20h. At the last time it will include a graph, a 1-week-forecast and a small evaluation how are stand today in comparison to the forecast which would have been made a week ago.
    • A friend helped me greatly by providing first a Holt-Winters prediction (which is live now), and then upgraded this to a more intelligent ARIMA prediction, which is still buried in a jupyter notebook and waiting for daylight. (Maybe she will write a guest post here to explain what it does and how it works – but I haven’t asked yet).
    • As for the bot’s code – it’s not (yet) public. Which is unusual for me, really. I should remedy that.

    What I learned so far:

    • It’s surprsingly easy and hard at the same time to get a Twitter developer account
    • Twitter does not permit publishing the exact same tweet multiple times in a short period of time
    • Heroku is really really nice for this, as long as you don’t need to pay for it. I would be interested in alternatives.
    • matplotlib is a lot more complicated than I expected
    • but strangely neither bokeh, nor plotly can actually export png graphics without either a separate electron app (WTF?) or a headless browser and selenium (W-T-F?!?)
    • pandas rocks, or more precisely: pandas data frames rock.
    • there does not seem to be a single properly maintained Twitter library for JavaScript, but there is at least tweepy for Python. (I mean I want to try something in JS, but honestly, if everything I find is outdated, I just stay with good old Python …)
    • The infection growth is intensely exponetial – almost a straight line on a log scale plot.

    Let’s see where this goes, and I hope you all stay healthy.