Tag: self-hosting

  • Backup tool selected

    I enacted an act of choosing. I chose restic.

    Reasons why

    Single binary. The contender with the even better stats was duplicati, which is available from the Ubuntu repositories, but requires 130+ MB of packages to be installed. This is a server system, so a single binary is just what I want – and very easy to duplicate if we switch operating systems. It also comes with a first-class-citizen web UI, which can be bad – cause once you need the backup tool you don’t want to have click orgies in web UIs to get stuff back.

    Somewhat good performace according to the comparison from the last post. I don’t expect to back up huge amounts of data, but I will keep an eye on the restore times. Also this is backing up data from a VM in a data center, so the link bandwidth should be sufficient in all directions.

    Best ever update process for a single binary. Just invoke restic self-update, and it does the rest. So cool.

    Nice docs – at least compared to, for example, duplicity.

    The remaining open question is whether it will back up all file metadata, which I will see shortly.

    Noteworthy

    Like all fiels in IT “backup” has the same drawbacks. On one side you have god-almighty-powerful tools for license cost that makes you scream while needing at least 5 certifications on top to operate them, on the other side you have … tar. Luckily, for backups you get at least some middle ground, namely all the tools I mentioned in my first post.

    I think all of them have certain advantages and disadvantages, while all being “good” tools for the job. The reasons I chose restic are above. Personally I would really like to try bupstash, but with backup software it does not pay to go for the “unstable” software which “might lose data”.

    Current state

    I have (blog post(s) with details forthcoming) …

    • created an S3 bucket and an IAM user with limited access to this
    • installed restic on one machine
    • initialized the restic repository in said bucket
    • created a run script which performs backups and handles service states
    • I want to update databases, and the safest way to do this is to stop them before
    • created a cron job which executes this script every 3 days
  • Pi-Hole in Docker

    disable systemd-resolved stub

    First, disable systemd-resolved name server stub. You can use this command to do this:

    # This just changes "#DNSStubListener=yes" to "DNSStubListener=no"
    sudo sed -r -i.orig 's/#?DNSStubListener=yes/DNSStubListener=no/g' /etc/systemd/resolved.conf
    

    You also have to remove the symlink /etc/resolv.conf, and replace it with a file:

    rm /etc/resolv.conf
    echo "nameserver 1.1.1.1" >  /etc/resolv.conf
    echo "nameserver 8.8.8.8" >> /etc/resolv.conf
    

    Then do systemctl restart systemd-resolved. (Source)

    run pi-hole in docker-compose

    I am using this docker-compose.yml file for pi-hole:

    version: "3"
    
    # More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
    services:
      pihole:
        container_name: pihole
        image: pihole/pihole:latest
    
        ports:
          - "53:53/tcp"
          - "53:53/udp"
          - "67:67/udp"
          - "80:80/tcp"
    
        environment:
          ServerIP: "192.168.1.2" # ADJUST FOR YOUR PI-HOLE IP OF COURSE
          TZ: "Europe/Berlin"
          WEBPASSWORD: "mysupersecretpassword"
    
        volumes:
          - "./data/etc-pihole/:/etc/pihole/"
          - "./data/etc-dnsmasq.d/:/etc/dnsmasq.d/"
    
        dns: # THIS IS IMPORTANT
          - 127.0.0.1 # SPECIALLY THIS LINE. I DON'T KNOW WHY.
          - 172.198.179.1
    
        # Recommended but not required (DHCP needs NET_ADMIN)
        #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
        cap_add:
          - NET_ADMIN
    
        restart: unless-stopped
    

    Update pi-hole entries

    Then update the pi-hole when it runs under: WebUI -> Tools -> Update Gravity -> Press “Update”.

  • Proxmox and home assistant

    A couple of hints to get proxmox and a home assistant VM up and running on a home server.

    Installing Proxmox

    Tip: Youtube “First 11 Things on Proxmox“.

    Graphical installer doesn’t run

    If the graphical installer does not come up (after DHCP) try this:

    # FIRST START PROXMOX INSTALLATION IN   D E B U G   M O D E   ! ! !
    
    # then in the shell starting before the installer runs, do ...
    $ Xorg -configure
    
    # COULD also be under /root/xorg.conf.new - read the command output
    $ cp /xorg.conf.new /etc/X11/xorg.conf
    
    # now replace all ocurrences of `amdgpu` with `fbdev` in /etc/X11/xorg.conf
    
    # exit the shell
    $ <CTRL-D>
    

    Now the installer should just come up.

    Source: This blog post and that comment below.

    Enable updates without subscription

    Change /etc/apt/sources.list.d/pve-enterprise.list to this:

    # /etc/apt/sources.list.d/pve-enterprise.list
    deb http://download.proxmox.com/debian/pve buster pve-no-subscription
    

    Install HASS.io

    SSH into your proxmox box and use this command:

    # (I didn't need sudo, I connected as root)
    bash -c "$(wget -qLO - https://raw.githubusercontent.com/Kanga-Who/home-assistant/master/hassos_install.sh)"
    
  • Nextcloud cron job

    I switched to system cron for my nextcloud cron jobs. Using docker, this is my cron job command:

    */5  *  *  *  *    /usr/bin/docker exec -u 33 srv_nc_1 /bin/bash -c 'php -f /var/www/html/cron.php'
    
  • Nextcloud out-of-memory PHP error

    I got this error when upgrading to nextcloud 21:

    PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) [...]
    

    Turns out this is the way to solve it: Prefix PHP_MEMORY_LIMIT=512M to the commands. In combination with docker it looks like this:

    /usr/bin/docker exec -u 33 srv_nc_1 /bin/bash -c 'PHP_MEMORY_LIMIT=512M php occ db:add-missing-indices'
    
    ... OR USING -e ...
    
    /usr/bin/docker exec -u 33 -e PHP_MEMORY_LIMIT=512M srv_nc_1 /bin/bash -c 'php occ db:add-missing-indices'
    
    ... OR MANUALLY ...
    
    ~$ docker exec -tiu 33 srv_nc_1 /bin/bash
    ~# PHP_MEMORY_LIMIT=512M php occ db:add-missing-indices
    

    Source: https://is.gd/mua5wv

  • nextcloud and Docker and reverse proxies

    I have a nextcloud setup like described here (docker-compose, let’s encrypt proxy companion, postgres and nextcloud). And for a while I couldn’t connect any new nextcloud clients to the installation.

    This fixed it:

    <?php
    $CONFIG = array (
      # manually added because it's not picked up from
      # the env vars once set ... it seems ...
    
      # the docker IP range
      'trusted_proxies' => ["172.16.0.0/12"],
    
      # the hostname of the server
      'overwritehost'   => "my.super.secret.server",
    
      # the ENDUSER->PROXY protocol, NOT the proxy-> nextcloud protocol!
      'overwriteprotocol' => "https",
    
      #
      # AAAND NOW back to the original config file ...
      # ...
    )
    

    Some notes:

  • Databases with dokku

    This is part 2 of a couple of blog posts about dokku, a amazing little Heroku clone.

    In the previous post I showed how to set up Dokku on a DigitalOcean droplet, and deployed a little hello-world container with a single git push. The reason why I wanted dokku thoug was the need of a database. As said – hosting comes cheap, databases usually come either expensive or with limited flexibility, or just too annoying configuration effort.

    Dokku is the perferct middle ground. Let’s see why.

    For me it was the existing postgres plugin which you can simply install and use. The whole process is incredibly easy, takes wbout two commands, and looks like this (let’s assume our “hello world” container uses a database):

    $ sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres
    

    That’s it, again.

    $ dokku postgres:create hello-world
    
    dokku postgres:create hello-world
    Waiting for container to be ready
    Creating container database Securing connection to database
    =====> Postgres container created: hello-world
    =====> Container Information
           Config dir: /var/lib/dokku/services/postgres/hello-world/config
           Data dir: /var/lib/dokku/services/postgres/hello-world/data
           Dsn: postgres://postgres:bd6b0725d710bb5a662bb628eee787b1@dokku-postgres-hello-world:5432/hello_world
           Exposed ports: -
           Id: 785ef252c748ed85739d1d6ad375a1e1bd66e925ac79358e9ffaa30ab852d6c0
           Internal ip: 172.17.0.9
           Links: -
           Service root: /var/lib/dokku/services/postgres/hello-world
           Status: running
           Version: postgres:10.2
    
    $ docker ps
    
    CONTAINER ID   IMAGE                      COMMAND                  CREATED         STATUS         PORTS      NAMES
    cc99cccacf2c   dokku/hello-world:latest   "/bin/sh -c 'php-fpm…"   2 minutes ago   Up 2 minutes   80/tcp     hello-world.web.1
    785ef252c748   postgres:10.2              "docker-entrypoint.s…"   5 minutes ago   Up 5 minutes   5432/tcp   dokku.postgres.hello-world
    [...]
    

    This creates a database container with postgres 10.2, as you can see. You can influence a lot of behavior by using environment variables, see the GitHub page for more info.

    Then you link the container to the running app:

    $ dokku postgres:link hello-world hello-world
    -----> Setting config vars
           DATABASE_URL: postgres://postgres:bd6b0725d710bb5a662bb628eee787b1@dokku-postgres-hello-world:5432/hello_world
    -----> Restarting app hello-world
    -----> Releasing hello-world (dokku/hello-world:latest)...
    -----> Deploying hello-world (dokku/hello-world:latest)...
    -----> Attempting to run scripts.dokku.predeploy from app.json (if defined)
    -----> No Procfile found in app image
    -----> DOKKU_SCALE file found (/home/dokku/hello-world/DOKKU_SCALE)
    =====> web=1
    -----> Attempting pre-flight checks
           For more efficient zero downtime deployments, create a file CHECKS.
           See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples
           CHECKS file not found in container: Running simple container check...
    -----> Waiting for 10 seconds ...
    -----> Default container check successful!
    -----> Running post-deploy
    -----> Configuring hello-world.my-paas.for-myself.com...(using built-in template)
    -----> Creating http nginx.conf
    -----> Running nginx-pre-reload Reloading nginx
    -----> Setting config vars DOKKU_APP_RESTORE: 1
    -----> Found previous container(s) (14c349cb496d) named hello-world.web.1
    =====> Renaming container (14c349cb496d) hello-world.web.1 to hello-world.web.1.1535285386
    =====> Renaming container (cc99cccacf2c) serene_bassi to hello-world.web.1
    -----> Attempting to run scripts.dokku.postdeploy from app.json (if defined)
    -----> Shutting down old containers in 60 seconds
    =====> 14c349cb496d95cc4be1833f2e7f6ef2bef099a37c2a22cd4dcdb542f09bea0f
    =====> Application deployed:
           http://hello-world.my-paas.for-myself.com
    

    And done.

    What happened? You have now the environment variable $DATABASE_URL set in the hello-world app, that’s why the restart was necessary (which you can postpone, if you want, but you probably need it now, right?).

    Let’s check:

    $ docker exec -ti hello-world.web.1 /bin/sh
    
    [now in the container]
    
    # env | grep DATABASE
    DATABASE_URL=postgres://postgres:bd6b0725d710bb5a662bb628eee787b1@dokku-postgres-hello-world:5432/hello_world
    

    That’s it. Super easy. Now if you’re using Django, you could use kennethreitz/dj-database-url to automatically parse and use it, and you’re done. (Probably every framework has something similar, so just have a look).

  • Build your own PaaS with Dokku

    I was looking for some “play” deployment method for a couple of things I want to try out. Most of them require a database. And it should be cheap, cause I don’t have any load on them and don’t earn any money, so I don’t want to spend basically no money if possible. The usual suspects are too expensive – AWS, Heroku, etc.

    So I looked around and found Dokku.

    Dokku is a set of – hang on – shell scripts – which basically emulate Heroku on a machine of your own. It’s integrated with Digital Ocean droplets out of the box, if you want it. And the whole thing is 5 € / month, which is perfect. It also integrates with a Dockerfile based deployment, so you do git push and everything just works.

    It’s amazing.

    This is how you get started. But before you can get started, you need a domain you control, either on AWS or any other hoster. This is for routing traffic to your deployments later. You also need a public SSH key, or better a public / private key pair. Once you have both you can …

    1. create a Digital Ocean account, and …
    2. add your SSH public key to your account, and …
    3. in that account, create a new droplet with a “Dokku” image preinstalled.
    4. Wait until the droplet finished provisioning.

    create new droplet

    While the droplet is being created, you can also create a project locally to test it later:

    $ mkdir dokku-test
    $ cd dokku-test
    $ git init
    $ echo "FROM tutum/hello-world" > Dockerfile
    $ git add Dockerfile
    $ git commit -m "Initial commit"
    

    In this little test project we only create a Dockerfile from an hello-world image which displays “Hello world” in a browser so we can verify it worked.

    Once the droplet is done, you can start setting up your personal little PaaS. First, you have to configure your DNS. We will set up a wildcard entry for our deployments, and a non-wildcard entry for git. Let’s assume your domain is for-myself.com, then you would add …

    • my-paas.for-myself.com , type “A” (or “AAAA” if you are IPv6) to your droplet IP
    • *.my-paas.for-myself.com just the same

    Then you SSH into your droplet, and create your dokku project. (This is something you have to do for every project). All you have to do for this is:

    $ ssh root@DROPLET_IP
    ~# dokku apps:create hello-world
    -----> Creating hello-world... done
    ~# _
    

    Done.

    Now you configure a git remote URL for your project, and push it:

    $ git remote add dokku dokku@my-paas.for-myself.com:hello-world
    

    Again – done. If you push your project now (assuming DNS is already set), everything should happen automagically:

    $ git push --set-upstream dokku master
    X11 forwarding request failed
    Enumerating objects: 3, done.
    Counting objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 241 bytes | 241.00 KiB/s, done.
    Total 3 (delta 0), reused 0 (delta 0)
    -----> Cleaning up...
    -----> Building hello-world from dockerfile...
    remote: build context to Docker daemon  2.048kB
    Step 1/1 : FROM tutum/hello-world
    latest: Pulling from tutum/hello-world
    658bc4dc7069: Pulling fs layer
    [... TRUNCATED ...]
    983d35417974: Pull complete
    Digest: sha256:0d57def8055178aafb4c7669cbc25ec17f0acdab97cc587f30150802da8f8d85
    Status: Downloaded newer image for tutum/hello-world:latest
     ---> 31e17b0746e4
    Successfully built 31e17b0746e4
    Successfully tagged dokku/hello-world:latest
    -----> Setting config vars
           DOKKU_DOCKERFILE_PORTS:  80/tcp
    -----> Releasing hello-world (dokku/hello-world:latest)...
    -----> Deploying hello-world (dokku/hello-world:latest)...
    -----> Attempting to run scripts.dokku.predeploy from app.json (if defined)
    -----> No Procfile found in app image
    -----> DOKKU_SCALE file not found in app image. Generating one based on Procfile...
    -----> New DOKKU_SCALE file generated
    =====> web=1
    -----> Attempting pre-flight checks
           For more efficient zero downtime deployments, create a file CHECKS.
           See http://dokku.viewdocs.io/dokku/deployment/zero-downtime-deploys/ for examples
           CHECKS file not found in container: Running simple container check...
    -----> Waiting for 10 seconds ...
    -----> Default container check successful!
    -----> Running post-deploy
    -----> Creating new /home/dokku/hello-world/VHOST...
    -----> Setting config vars
           DOKKU_PROXY_PORT_MAP:  http:80:80
    -----> Configuring hello-world.my-paas.for-myself.com...(using built-in template)
    -----> Creating http nginx.conf
    -----> Running nginx-pre-reload
           Reloading nginx
    -----> Setting config vars
           DOKKU_APP_RESTORE:  1
    =====> Renaming container (14c349cb496d) amazing_snyder to hello-world.web.1
    -----> Attempting to run scripts.dokku.postdeploy from app.json (if defined)
    =====> Application deployed:
           http://hello-world.my-paas.for-myself.com
    
    To my-paas.for-myself.com:hello-world
     * [new branch]      master -> master
    Branch 'master' set up to track remote branch 'master' from 'dokku'.
    

    And if you open your URL now (which is hello-world.my-paas.for-myself.com) you should see this image:

    browser display of deployed example site

    Now, for 5 € / month you get:

    • A heroku-like, no-nonsense, fully automated, git-based deployment platform
    • A server which you control (and have to maintain, okay, but on which you can deploy …)
    • A database (or many of them – dokku provides great integration for databases btw; more on that in another post)
    • Publicly reachable deployments (for customers, testing, whatever)
    • Let’s Encrypt certificates (dokku provides support for these as well, again more in a later post)
    • And for 1 € more (it’s always 20% of the base price) you get backups of your system)

    That’s absolutely incredible. Oh, and did I mention that the maintainers are not only friendly, but also super responsive and incredibly helpful on Slack?

  • Syntax highlighting with wordpress

    This is just a test for syntax highlighting. Which I really really really wanted to have. Even if it’s WordPress and not something cool like hugo.

    So, let’s try:

    import random
    
    class RndTest:
        def __init__(self):
            print("Oy! 'this is python!")
            self.rnd_seed = random.random()
    
        @classmethod
        def do_nothing(cls, nothing):
            print("Doing {} now ;)".format(nothing))
    

    That doesn’t look so bad, right?

    How to do this

    • install the crayon syntax highlight plugin
    • when writing posts, your toolbar will have a new icon looking like this: <>
    • press it, and an “add code” dialog will open
    • do your thing
    • save
    • done

    Like.

    Dislikes

    • I don’t know which JS engine this thing uses (if any public one in particular)
    • I like highlight.js

    (So, none really)

    Alternative plugins (untested)

    To be honest, the only real contender in installation base and features seems to be Syntax Highlighter Evolved. I have not tried it, but if you don’t like crayon, that looks like the one to go.

  • Docker registry, S3 and permissions

    There are a couple of bazillion blog posts saying “yah just did my docker registry on S3”.

    It’s not so easy, though. Cause what if you want to limit access to a certain IAM user? Yup, you need to go deep (well, a bit) into the policy thing of Amazon. Which sounds simple, but isn’t.

    I got “HTTP 500” errors from the docker registry when I first deployed. My configuration, which was wrong, looked like this:

    "RegistryIAMUser" : {
      "Type" : "AWS::IAM::User"
    },
    "RegistryIAMUserAccessKey" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : { "UserName" : { "Ref" : "RegistryIAMUser" } }
    },
    "Bucket" : {
      "Type" : "AWS::S3::Bucket",
      "Properties" : { "BucketName" : "flypenguin.docker-registry" }
    },
    
    "RegistryPrivateAccess" : {
      "Type" : "AWS::S3::BucketPolicy",
      "Properties" : {
        "Bucket" : {"Ref":"Bucket"},
        "PolicyDocument": {
          "Statement":[{
            "Action":[ "s3:*" ],
            "Effect":"Allow",
            "Resource":  { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "Bucket" } , "/*" ]]},
            "Principal": {"AWS" : {"Fn::GetAtt":["RegistryIAMUser","Arn"]}}
          }]
        }
      }
    }
    

    Since this didn’t work really well, I googled my a** off and found a little post, which used a UserPolicy (instead of a bucket policy, which is basically the other way around), but did one thing different. My **working** configuration is now … (let’s see if you can see the difference):

    [... same as above ...]
    
    "UserPolicyRegistryPrivateAccess" : {
      "Type" : "AWS::IAM::Policy",
      "Properties" : {
        "PolicyName" : "AccessToDockerBucket",
        "Users" : [ {"Ref":"RegistryIAMUser"}],
        "PolicyDocument" : {
          "Version" : "2012-10-17",
          "Statement" : [{
            "Effect":"Allow",
            "Action":[ "s3:*" ],
            "Resource": [
              { "Fn::Join" : ["", ["arn:aws:s3:::", {"Ref":"Bucket"} , "/*" ]]},
              { "Fn::Join" : ["", ["arn:aws:s3:::", {"Ref":"Bucket"} ]]}
            ]
          }]
        }
      }
    }
    

    See it?

    It’s the two resources now. You need not only “resource/*” as a target, you also need “resource” itself as a target. Which makes sense if you know it and think about it. If you don’t … it’s a bit annoying. And time-consuming.