Tag: data storage

  • Managing S3 bucket permissions with terraform

    This is what I did today, and I found it useful. And yes, sometimes I over-engineer a bit.

      # IN FILE: locals_constants.tf
    
      iam_s3_perms_list = ["s3:List*",]
      iam_s3_perms_read = ["s3:List*", "s3:Get*",]
      iam_s3_perms_write = ["s3:List*", "s3:Get*", "s3:PutObject",]
    
      s3_permissions_rw = distinct(concat(
        local.iam_s3_perms_list,
        local.iam_s3_perms_read,
        local.iam_s3_perms_write,
      ))
    
      s3_permissions_ro = distinct(concat(
        local.iam_s3_perms_list,
        local.iam_s3_perms_read,
      ))
    
    
    # IN FILE: locals.tf
    
    team_dev_bucket_configs = {
      "ReadWrite" = {
        actions = local.s3_permissions_rw
        buckets = [
          "yo-corp-yoman-dev",
          "yo-corp-yoman-test",
          "yo-corp-playground",
        ]
      }
      "ReadOnly" = {
        actions = local.s3_permissions_ro
        buckets = [
          "yo-corp-yoman-prod",
          "yo-corp-yoman-archive",
        ]
      }
    }
    
    
    # IN FILE: permissions.tf
    
    # so this policy is now assigned to a group with every dev team member.
    # easy. :)
    resource "aws_iam_policy" "s3_team_dev_bucket_permissions" {
      name        = "s3-team-dev-bucket-permissions"
      description = "Dev team S3 bucket permissions"
      policy      = data.aws_iam_policy_document.s3_team_dev_bucket_permissions_document.json
    }
    
    data "aws_iam_policy_document" "s3_team_dev_bucket_permissions_document" {
      dynamic "statement" {
        for_each = local.team_dev_bucket_configs
        content {
          sid     = "TeamDevBuckets${statement.key}"
          effect  = "Allow"
          actions = statement.value["actions"]
          resources = flatten(
            [
              for bucket in statement.value["buckets"] :
              [
                "arn:aws:s3:::${bucket}", "arn:aws:s3:::${bucket}/*"
              ]
            ]
          )
        }
      }
    }
    
  • Nextcloud cron job

    I switched to system cron for my nextcloud cron jobs. Using docker, this is my cron job command:

    */5  *  *  *  *    /usr/bin/docker exec -u 33 srv_nc_1 /bin/bash -c 'php -f /var/www/html/cron.php'
    
  • Nextcloud out-of-memory PHP error

    I got this error when upgrading to nextcloud 21:

    PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) [...]
    

    Turns out this is the way to solve it: Prefix PHP_MEMORY_LIMIT=512M to the commands. In combination with docker it looks like this:

    /usr/bin/docker exec -u 33 srv_nc_1 /bin/bash -c 'PHP_MEMORY_LIMIT=512M php occ db:add-missing-indices'
    
    ... OR USING -e ...
    
    /usr/bin/docker exec -u 33 -e PHP_MEMORY_LIMIT=512M srv_nc_1 /bin/bash -c 'php occ db:add-missing-indices'
    
    ... OR MANUALLY ...
    
    ~$ docker exec -tiu 33 srv_nc_1 /bin/bash
    ~# PHP_MEMORY_LIMIT=512M php occ db:add-missing-indices
    

    Source: https://is.gd/mua5wv

  • nextcloud and Docker and reverse proxies

    I have a nextcloud setup like described here (docker-compose, let’s encrypt proxy companion, postgres and nextcloud). And for a while I couldn’t connect any new nextcloud clients to the installation.

    This fixed it:

    <?php
    $CONFIG = array (
      # manually added because it's not picked up from
      # the env vars once set ... it seems ...
    
      # the docker IP range
      'trusted_proxies' => ["172.16.0.0/12"],
    
      # the hostname of the server
      'overwritehost'   => "my.super.secret.server",
    
      # the ENDUSER->PROXY protocol, NOT the proxy-> nextcloud protocol!
      'overwriteprotocol' => "https",
    
      #
      # AAAND NOW back to the original config file ...
      # ...
    )
    

    Some notes:

  • Docker registry, S3 and permissions

    There are a couple of bazillion blog posts saying “yah just did my docker registry on S3”.

    It’s not so easy, though. Cause what if you want to limit access to a certain IAM user? Yup, you need to go deep (well, a bit) into the policy thing of Amazon. Which sounds simple, but isn’t.

    I got “HTTP 500” errors from the docker registry when I first deployed. My configuration, which was wrong, looked like this:

    "RegistryIAMUser" : {
      "Type" : "AWS::IAM::User"
    },
    "RegistryIAMUserAccessKey" : {
      "Type" : "AWS::IAM::AccessKey",
      "Properties" : { "UserName" : { "Ref" : "RegistryIAMUser" } }
    },
    "Bucket" : {
      "Type" : "AWS::S3::Bucket",
      "Properties" : { "BucketName" : "flypenguin.docker-registry" }
    },
    
    "RegistryPrivateAccess" : {
      "Type" : "AWS::S3::BucketPolicy",
      "Properties" : {
        "Bucket" : {"Ref":"Bucket"},
        "PolicyDocument": {
          "Statement":[{
            "Action":[ "s3:*" ],
            "Effect":"Allow",
            "Resource":  { "Fn::Join" : ["", ["arn:aws:s3:::", { "Ref" : "Bucket" } , "/*" ]]},
            "Principal": {"AWS" : {"Fn::GetAtt":["RegistryIAMUser","Arn"]}}
          }]
        }
      }
    }
    

    Since this didn’t work really well, I googled my a** off and found a little post, which used a UserPolicy (instead of a bucket policy, which is basically the other way around), but did one thing different. My **working** configuration is now … (let’s see if you can see the difference):

    [... same as above ...]
    
    "UserPolicyRegistryPrivateAccess" : {
      "Type" : "AWS::IAM::Policy",
      "Properties" : {
        "PolicyName" : "AccessToDockerBucket",
        "Users" : [ {"Ref":"RegistryIAMUser"}],
        "PolicyDocument" : {
          "Version" : "2012-10-17",
          "Statement" : [{
            "Effect":"Allow",
            "Action":[ "s3:*" ],
            "Resource": [
              { "Fn::Join" : ["", ["arn:aws:s3:::", {"Ref":"Bucket"} , "/*" ]]},
              { "Fn::Join" : ["", ["arn:aws:s3:::", {"Ref":"Bucket"} ]]}
            ]
          }]
        }
      }
    }
    

    See it?

    It’s the two resources now. You need not only “resource/*” as a target, you also need “resource” itself as a target. Which makes sense if you know it and think about it. If you don’t … it’s a bit annoying. And time-consuming.

  • Synology and Git, II

    Don’t use the synology git package.

    But then Git’s not installed.

    I found a compromise:

    • Install the package
    • disable it
    • switch back the shells in /etc/passwd from to /bin/sh

    Be happy.

  • Synology Git Server

    Ich besitze ab neulich eine wundervolle Synology DS414. Schönes Teil.

    Und da ich aktuell ein wenig programmiere zu Hause wollte ich mir hier ein Git repository server einrichten. Das ist nicht ganz trivial, und wer sich github- oder gitlab-ähnlichen Komfort erhofft … ah, nö. Es ist commandline only, und davon viel. Aber zum Punkt – der schnellstmöglichen Einrichtung eines Git-Repo-Servers auf einer Diskstation.

    • Der SSH-Zugriff unter Systemsteuerung > Terminal muss aktiviert sein.
    • Es müssen Benutzer angelegt sein, und diese müssen ein Home-Verzeichnis haben (für die ssh-Schlüssel, ohne die es keinen Spaß macht). Das home-Verzeichnis aktiviert man in der Systemsteuerung > Benutzer, dann oben ein Knopf Benutzerbasis (der 2. von rechts bei mir)
    • Wenn die Benutzer vorhanden sind legt man eine Freigabe an, das macht die Sache einfacher. Ich nutze /volume1/gitrepos.
    • Den eigenen ssh-Schlüssel ins home-Verzeichnis kopieren. Das sollte unter /volume1/homes/<Benutzer> erreichbar sein, das Volume hat man vorhin in Schritt (1) ausgewählt. (natürlich muss .ssh/ 0600 und . sein, und .ssh/authorized_keys 0700 und ., aber das muss ich ja nicht extra erwähnen, oder?)
    • Jetzt den Git-Service aktivieren, dem gewünschten Benutzer den Zugriff erlauben.
    • Ein repository anlegen ist leider Handarbeit:
    • Anmelden als root, dann nach /volume1/gitrepos wechseln, ein Verzeichnis anlegen …
    • … git init --bare ausführen, … und
    • und jetzt entweder alles word-writable machen (bäh), oder dem entsprechenden Benutzer mittels chown zuweisen (yup).
    • Beispiel: /volume1/gitrepos/my_repo
    • Schließlich auf dem Arbeitsrechner noch git das neue Repo übergeben: git remote add origin <user>@<diskstation>:/volume1/gitrepos/my_repo
    • … und das wars.

    Ganz einfach, oder? Naja.

    Quellen: Synology Wiki, Synology Forum