Prepare disk image

As a base, we'll use the x86_64 system configuration.

Create a new project, make sure to set MIX_TARGET=x86_64 and also make sure the following drivers are built into the linux kernel. Check the documentation for details on how to build a custom kernel.

• CONFIG_NVME_CORE
• CONFIG_BLK_DEV_NVME
• CONFIG_PCI_MSI
• CONFIG_ENA_ETHERNET

This will make storage (NVME) and networking (Elastic Network Adapters) work in AWS.

Then do the usual dance to build an img file:

export MIX_TARGET="x86_64"
mix deps.get
mix firmware
qemu-img create -f raw disk.img 1G
fwup -d disk.img _build/x86_64_dev/nerves/images/<your_project>.fw

Create amazon machine image (AMI)

We need to do the following things:

• Create a S3 bucket (or use one that already exists)
• Upload the disk image to the bucket
• Create a role vmimport for the import-snapshot job
• Create an EBS snapshot from the disk image
• Create an AMI from the EBS snapshot

Once everything is done you can use the AMI to create new VMs running nerves.

Preparation

Export bucket name and AWS region

export BUCKET_NAME="<bucket-name>"
export AWS_REGION="<region>"

Create a s3 bucket

aws s3 mb s3://$BUCKET_NAME

Upload disk image to s3 bucket

aws s3 cp disk.img s3://$BUCKET_NAME

Create vmimport role

We need to create a role which has the required permissions to import a disk image as a snapshot or image. By default the aws ec2 import-snapshot and aws ec2 import-image commands use a role called vmimport. You can change the name on both commands using the --role-name parameter.

For this guide we stick with the name vmimport.

First we create a role with this trust policy which you should save to a file called trust-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "vmie.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:Externalid": "vmimport"
        }
      }
    }
  ]
}

Then create a role using this policy:

aws iam create-role --role-name vmimport --assume-role-policy-document file://trust-policy.json

Now we need to allow this role to do certain things. Save the following JSON to a file called role-policy.json. Make sure to replace YOUR_BUCKET_NAME with the name of your bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:GetBucketAcl"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:ModifySnapshotAttribute",
        "ec2:CopySnapshot",
        "ec2:RegisterImage",
        "ec2:Describe*"
      ],
      "Resource": "*"
    }
  ]
}

And finally we assign the policy to the role we already created.

aws iam put-role-policy --role-name vmimport --policy-name vmimport --policy-document file://role-policy.json

Create an EBS snapshot from the disk image

This command will import a snapshot from the disk.img file. The vmimport role will be used by default as shown in the documentation.

aws ec2 import-snapshot \
  --description "nerves-$(date +"%Y%m%d%H%M%S") \
  --disk-container '{
    "Format": "RAW",
    "UserBucket": {
      "S3Bucket": "'${BUCKET_NAME}'",
      "S3Key": "disk.img"
    }
  }'

Check the status of the import via this command:

aws ec2 describe-import-snapshot-tasks

The output might look like this:

{
    "ImportSnapshotTasks": [
        {
            "Description": "nerves-20250929164312",
            "ImportTaskId": "import-snap-bea76eae3f6140f2t",
            "SnapshotTaskDetail": {
                "DiskImageSize": 1073741824.0,
                "Format": "RAW",
                "Progress": "19",
                "SnapshotId": "",
                "Status": "active",
                "StatusMessage": "downloading/converting",
                "UserBucket": {
                    "S3Bucket": "mnesia-test",
                    "S3Key": "disk.img"
                }
            },
            "Tags": []
        }
    ]
}

Wait until the Status field shows completed and take note of the SnapshotId value which will appear once the job completed.

export SNAPSHOT_ID="snap-0538c2e855cb369b0"

Create an AMI from the EBS snapshot

aws ec2 register-image \
    --name nerves-$(date +"%Y%m%d%H%M%S") \
    --root-device-name /dev/sda1 \
    --block-device-mappings DeviceName=/dev/sda1,Ebs={SnapshotId=${SNAPSHOT_ID}}

Now you'll have an AMI from which you can boot nerves on a new EC2 instance.

The other day I created a basic OpenStreetMap related example using the phoenix framework. This example uses protomaps and maplibre-gl to draw maps in the browser without any external resources.

The example app also uses a small static GeoJSON file to draw some streets which I downloaded via overpass-turbo. This works well for small datasets.

Short video: https://youtu.be/AnjkZntIxW0

But what if I wanted to draw the streets of a complete country instead of just the city center of Frankfurt? I assume the GeoJSON file would probably be very big and create slow results.

Instead of a static file I wanted to give PostGIS a try to only return the streets in an area around a specific point.

Setup

Create a Postgres database and enable PostGIS:

CREATE DATABASE streets;
CREATE EXTENSION IF NOT EXISTS postgis;

Now we need data to import. I downloaded a slice of the world from Geofabrik.

There are many ways to import the data into the database. I choose ogr2ogr for no particular reason.

ogr2ogr -f "PostgreSQL" PG:"dbname=streets user=postgres" -sql "SELECT osm_id FROM lines WHERE highway IN ('motorway','trunk','primary','secondary','residential','living_street','tertiary')" -nln <table name> <filename>

Replace <table name> with the table name you want to use and <filename> with the name of the pbf file you just downloaded.

Once the command has finished you should have a rather boring table containing three columns.

• ogc_fid: A PostGIS internal ID
• osm_id: The corresponding OpenStreetMap ID
• wkb_geometry: The streets we want in WKB representation

The problem

The import with ogr2ogr2 creates an index for the geometries that looks something like this:

streets=# \d index_test
                                           Table "public.index_test"
    Column    |           Type            | Collation | Nullable |
Default
--------------+---------------------------+-----------+----------+---------------------------------------------
 ogc_fid      | integer                   |           | not null | nextval('index_test_ogc_fid_seq'::regclass)
 osm_id       | character varying         |           |          |
 wkb_geometry | geometry(LineString,4326) |           |          |
Indexes:
    "index_test_pkey" PRIMARY KEY, btree (ogc_fid)
    "index_test_wkb_geometry_geom_idx" gist (wkb_geometry)

Running queries with this index results in poor performance, because it looks like the index is just ignored.

streets_dev=# SELECT osm_id FROM index_test WHERE ST_DWithin(wkb_geometry,
'POINT(8.68417 50.11552)'::geography, 100) ORDER BY wkb_geometry <->
'POINT(8.68417 50.11552)'::geography;

  osm_id
-----------
 5069357
 4834447
 686648950
(3 rows)

Time: 417.628 ms

Adding explain (analyze true, verbose true, buffers true) in front of the query shows a rather long query plan which does not involve the index.

The solution

Luckily an answer on StackOverflow explains very well what the issue is (I replaced the column names to fit my example):

The problem is that you are mixing geometry and geography, and PostgreSQL casts wkb_geometry to geography so that they match.

Now you have indexed wkb_geometry, but not wkb_geometry::geography, which is something different.

Either use 'SRID=4326;POINT(-0.145872 51.509691)'::geometry as second operand or create the GiST index on ((geom_bounding_box::geography)) (note the double parentheses).

After dropping the old index and creating a new one, the query look much better.

streets=# DROP INDEX index_test_wkb_geometry_geom_idx;
DROP INDEX
Time: 6.262 ms

streets=# CREATE INDEX index_test_wkb_geometry_geom_idx ON index_test USING
GIST ((wkb_geometry::geography));
CREATE INDEX
Time: 1154.404 ms (00:01.154)

streets=# SELECT osm_id FROM index_test WHERE ST_DWithin(wkb_geometry,
'POINT(8.68417 50.11552)'::geography, 100) ORDER BY wkb_geometry <->
'POINT(8.68417 50.11552)'::geography;
  osm_id
-----------
 5069357
 4834447
 686648950
(3 rows)

Time: 1.509 ms

This reduced the query time from ~410ms to ~1.5ms. Yay.

One of my clients runs an old database. How old? It's MySQL 5.7.

That is a problem because MySQL 5.7 has been end of life since October 2023 and the FreeBSD port has been removed on 2023-12-31.

But we need a test system. Something that looks and feels like the production system. And we need to find a way to install MySQL 5.7.

The solution

Turns out you can download an old version of the ports tree and start there. There is no guarantee it will work but it is a start.

The FreeBSD ports are hosted on multiple platforms. You can choose whatever you like most.

• cgit
• Codeberg
• GitLab
• GitHub

The name of the port I'm using in this example is mysql57-server.

1. Search for your port on freshports
2. Open the last commit which removed the port. It's probably the first entry on freshports. (cgit, Codeberg, GitLab, GitHub)
3. Go to parent commit where the port has not been removed yet (cgit, Codeberg, GitLab, GitHub)
4. On cgit you can now download a .tar.gz or .zip of the parent commit. On the other platforms, you need to click on "Browser files" or "Browse source" before you can download the source.
5. Extract the archive
6. Move to the right directory: cd databases/mysql57-server
7. Compile and install it: ALLOW_UNSUPPORTED_SYSTEM="YES"; BATCH="YES"; make install clean

ALLOW_UNSUPPORTED_SYSTEM let's you install the port on a different system than it was planned to be installed on.

BATCH uses the default values for every prompt that might come up while installing a port.

Now you just need to wait for the port and all of its dependencies to be compiled.

Update from 2024-04-23:

I was made aware about a couple of fixes to make mysql57-server compile on 14.0:

1. Replace -DWITH_EDITLINE=system with -DWITH_EDITLINE=bundled in Makefile
2. Install openssl (pkg install openssl)
3. Set default to openssl in make.conf (echo "DEFAULT_VERSIONS+=ssl=openssl" >> /etc/make.conf)

Thank you Chr. Schmidt!

The SVG in question is a map of the world.

When you open the link you can already see the problem that also happens when you try to embed it into a website. It will be shown in its full size.

The file starts with the following <svg> tag:

<svg xmlns="http://www.w3.org/2000/svg" height="1200" width="2370">

The obvious change would be to alter width and height but this resulted in the SVG only showing parts of the drawn graphic. It kind of zoomes into the image.

Then I tried to transform the content with an additional element:

<g transform="scale(0.8)">

But this turned out to be a bit static. Viewing the website on a mobile device did not change the size of the SVG at all. So it was too small for a desktop size screen and too big for a mobile device screen.

The solution was to add a div element and some attributes for the SVG tag:

<div style="max-width:300px;">
<svg xmlns="http://www.w3.org/2000/svg" width="100%" viewBox="0 0 2370 1200" preserveAspectRatio="xMinYMax meet">
[...]

The initial width and height attributes are now part of a viewBox attribute. The width is set to 100% to make it as big as the parent div element which has a max-width of 300px. Finally, the preserveAspectRatio="xMinYMax meet" is something that I do not fully understand yet, but it was the solution to align the SVG the the left side of the screen.

Sources:

• https://stackoverflow.com/questions/19484707/how-can-i-make-an-svg-scale-with-its-parent-container
• https://stackoverflow.com/questions/8639383/how-do-i-center-a-svg-in-a-div

To speed up this process, I wrote a one-liner using ffmpeg:

$ which img2webp
img2webp () {
        ffmpeg -hide_banner -loglevel error -i $1 -c:v libwebp ${1/\.*/\.webp} >> /dev/null
        ls -lah $1
        ls -lah ${1/\.*/\.webp}
}

Let's see it in use:

$ img2webp IMG_2228.jpeg
-rw-r--r--@ 1 maltekrupa  staff   2.4M Jan 26 08:15 IMG_2228.jpeg
-rw-r--r--  1 maltekrupa  staff   570K Jan 26 08:16 IMG_2228.webp

It usually takes less than a second on a three year old M1 Mac and since ffmpeg is such an amazing tool, it works with basically any image format.

First, check the current version of every jail:

$ for i in $(jls name); do sudo freebsd-version -j $i; done
[..]
14.0-RELEASE-p1
14.0-RELEASE-p1
14.0-RELEASE-p1

Then, run an unattended update on all jails:

$ for i in $(jls name); do sudo PAGER="cat" freebsd-update -j $i fetch install; done

PAGER is usually set to less which results in a dialog you have to close manually. With cat the output is just printed and everything continues without interaction.

Once the updates are done, restart all the jails.

Keep in mind: This command will stop all jails in the order as they appear in /etc/jail.conf. Once all of them are stopped, they are started again.

$ sudo service jail restart

Done.

To rotate the credentials they:

1. enable the new role with a new password
2. switch all applications to the new credentials
3. disable the old role

Enable/Disable means to allow or prevent the role from logging in.

I usually do the same ... but with a twist. Instead of having e.g. role_a and role_b I'll create a new role whenever the credentials need to be rotated. This new role will look something like this: role_2023_10_04_09_00_00.

The timestamp makes it very clear how long credentials have been in use and it's a good reminder to rotate them very soon.

Helper function

For this I use a helper function called create_rotating_role. This function takes three arguments:

• username: An arbitrary username which is the prefix for the timestamp
• passwd: The password for the new user
• parent: The name of the parent role the new user should become part of

CREATE OR REPLACE FUNCTION public.create_rotating_role(
    username character varying,
    passwd character varying,
    parent character varying
)
RETURNS character varying
LANGUAGE plpgsql
AS $function$
DECLARE
    username_complete character varying;
BEGIN
    username_complete = username || '_' || to_char(current_timestamp, 'YYYY_MM_DD_HH_MM_SS');
    execute format($$CREATE ROLE %I WITH LOGIN PASSWORD '%s' IN ROLE %s$$, username_complete, passwd, parent);
    execute format($$ALTER ROLE %I SET ROLE %s$$, username_complete, parent);
    return username_complete;
END;
$function$;

The function does two things. It creates the new user and changes the default role for said user to the role of the parent. Every role will therefore act as the parent role. This is important because it allows all the roles to alter and delete tables which have been created by previously created roles which might not exist anymore.

It's probably best to add the function in the postgres database. Otherwise you'll need it in every database where you plan to use it (I think).

Workflow

Initial setup

Let's assume we start from zero.

We'll create a new database and a new role which we grant the required permissions:

CREATE DATABASE app_db;
CREATE ROLE app_role_parent WITH NOLOGIN;
GRANT ALL PRIVILEGES ON DATABASE app_db TO app_role_parent;

This new role is the parent and all future changes to permissions will be done to it.

It's redundant but more explicit: We specified WITH NOLOGIN on the role, which is the default for roles create via CREATE ROLE (compared to CREATE USER). This role will not be used to login. That's what we create the other roles for.

Repeating task

With the new function in place we can create a new role by running:

SELECT create_rotating_role('app_role', <password>, 'app_role_parent');

     create_rotating_role
------------------------------
 app_role_2023_10_04_12_34_56
(1 row)

The output of the function will contain the name of the newly created role.

Now you need to change the credentials in whatever app you're using and once everything is migrated to the new role you can delete the old one.

DROP ROLE app_role_2023_09_03_01_23_45;

That's it.

What I learned along the way

Imagine working in IT for way too long and still learning the basics. :)

• Use Single Quotes for strings
• Use Double Quotes for columns, tables or roles
• Use Dollar Quoting to make text in single quotes more readable
• How to write functions and declare variables

Sources

Other people posted about this YEARS ago (in order of how resourceful I think their posts are):

• http://davidhollenberger.com/2017/03/16/postgres-credential-rotation/
• https://thnee.se/postgresql-rotate-password/
• https://www.jannikarndt.de/blog/2018/08/rotating_postgresql_passwords_with_no_downtime/

I misread the task and confused "terms in the Fibonacci sequence whose values do not exceed four million" with "fourth million Fibonacci term".

Instead of calculating to the 33th term I calculated to the 4_000_000th term. :D

15 minutes later I got a rather big number on the screen and realised I did something wrong.

Initially I created a growing list containing all elements from 1 to n which then was iterated a last time to calculate the sum of all even-valued terms. But because of a failure in the exit condition the tests ran into a timeout.

But I didn't see that yet and believed the problem was the list with four million entries, so I refactored the solution to only work with a two element tuple and calculate the sum of the even-valued terms on every iteration.

Since this also resulted in a timeout, I checked the exit condition again and found the real problem. And since I like the two-element-tuple solution more than the initial one I kept it. :)

A quick benchmark suggests that the version using lists is 1.78 times slower than the ones using tuples.

$ mix run benchmarks/problem_2.exs
Operating System: macOS
CPU Information: Apple M1
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.15.6
Erlang 26.0.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 14 s

Benchmarking list ...
Benchmarking tuple ...

Name            ips        average  deviation         median         99th %
tuple        5.06 M      197.69 ns ±16067.94%         125 ns         250 ns
list         2.84 M      351.74 ns  ±8009.82%         292 ns         459 ns

Comparison:
tuple        5.06 M
list         2.84 M - 1.78x slower +154.05 ns

This should now be fixed.

That being said, even after years of using hugo it still feels strange from time to time because I'm not really a web developer nor am I planning to become one (at least not on the frontend side). Still ... hugo made it possible for me to at least be a bit happy about the web sites I build. <3

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23. Find the sum of all the multiples of 3 or 5 below 1000.

My solution: https://github.com/maltekrupa/euler/blob/main/lib/problem_1.ex

I choose to solve this with some recursion which is making use of tail-call optimization.

My biggest "issue" was to remember the Enum.to_list/1 function because my text editor was only showing part of the Enum functions in the code completion. :)

I'll track my work in a public git repository: https://github.com/maltekrupa/euler

In this repository I created a new elixir application via mix new euler and then prepared a mix task to create the needed files for every new problem and download the corresponding problem description into the @moduledoc field of the new module.

I bet the problems get way too hard for me in a matter of days. Let's see.

Setup

Assuming the following configuration after installing vm-bhyve in /etc/rc.conf:

vm_enable="YES"
vm_dir="zfs:zroot/vms"

And a vm-bhyve mountpoint set to /usr/local/vms.

Howto

First we create a new VM. In this guide we use the name testvm.

vm create -t alpine -c 1 -m 2G testvm

Meaning:
-t alpine: use the alpine template
-c 1: one core
-m 2G: 2GB of memory

Delete the disk image file as we will not need it.

rm /usr/local/vms/testvm/disk0.img

Create a ZFS dataset underneath the dataset used for the VM:

zfs create -V20G -o volmode=dev zroot/testvm/disk0

Meaning:
-V20G: Create dataset with a size of 20GB
disk0: Name of the dataset which needs to be referenced in the configuration file

Alter the configuration in /usr/local/vms/testvm/testvm.conf to look like this:

loader="uefi"
cpu="1"
memory="2G"
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_dev="zvol"
disk0_name="disk0"

This is it. Now you can start installing an OS as you would normally do via vm install testvm <iso>.

If you plan on installing your OS via a GUI, you might want to add the following to you configuration to be able to connect via VNC to your VM (requires UEFI!):

xhci_mouse="yes"
graphics="yes"
graphics_res="800x600"

The sample template for alpine linux in the vm-bhyve project is using grub to boot the OS.

With this template I got the following error on every boot:

error: unknown filesystem.
error: you need to load the kernel first.

Solution Workaround

After moving from grub to uefi everything worked fine.

Additionally, I changed virtio-blk to nvme because multiple sources mentioned an enhancement in disk speed.

The template should look something like this:

loader="uefi"
cpu=1
memory=512M
network0_type="nvme"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"

The changes in detail:

• replace loader="grub" with loader="uefi"
• remove all lines containing the word grub_
• replace disk0_type="virtio-blk" with disk0_type="nvme"

You will also need to install sysutils/bhyve-firmware if you haven't done so already.

When I deployed my first elixir based application to a FreeBSD jail I had some issues stopping the service. Starting worked like a charm, but whenever I tried to stop it, the following error appeared:

# service app stop
app not running? (check /var/run/app/app.pid).

This was confusing to me, because the process identifier (PID) in the mentioned file was correct and when I tried to start the process again (while it was already running) I received this error:

# service app start
Starting app.
daemon: process already running, pid: 74473
/usr/local/etc/rc.d/app: WARNING: failed to start app

Something was off and after reading some documentation I found out what it was.

Background

Starting an elixir application is usually done by executing a script. I am using the Phoenix Framework which creates this script using the mix phx.gen.release command. The way I designed the deployment, the generated script eventually lands at the following location on a FreeBSD host: /usr/local/app/v1.2.3/bin/server.

When executed, the script will start the erlang virtual machine (BEAM) with a very long command. It might look something like this:

/usr/local/app/v1.2.3/erts-13.2/bin/beam.smp -- -root /usr/local/app/v1.2.3 -bindir /usr/local/app/v1.2.3/erts-13.2/bin -progname erl -- -home /home/app -- -noshell -s elixir start_cli -mode embedded -setcookie I_AM_A_COOKIE -sname app -config /usr/local/app/v1.2.3/releases/v1.2.3/sys -boot /usr/local/app/v1.2.3/releases/v1.2.3/start -boot_var RELEASE_LIB /usr/local/app/v1.2.3/lib -- -extra --no-halt

Somehow we need to make the rc script aware of this long command. Or at least parts of it.

The solution

Assuming our service is called app, we will place the following rc script at /usr/local/etc/rc.d/app.

#!/bin/sh

# PROVIDE: app
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown

. /etc/rc.subr

name="app"
rcvar="app_enable"

command="/usr/sbin/daemon"
pidfile="/var/run/app/${name}.pid"

task="/usr/local/app/v1.2.3/bin/server"
procname="*beam.smp*"

app_chdir="/usr/local/app"

command_args="-p ${pidfile} -t ${name} -u app -o /var/log/app.log ${task}"

load_rc_config $name
run_rc_command "$1"

To get a general understanding of rc files I can recommend the "Practical rc.d scripting in BSD"-Guide.

The important part for an elixir application and this scenario is the combination of command, command_args, task and procname.

• command defines what is executed on start
• command_args is passed to the command as argument
• task only exists to enhance readability of command_args
• procname is by default set to the same value as command. It must contain the process name at runtime (that is: after it is started).

You might already guess the problem: command contains the script which is created by mix phx.gen.release. And procname must contain the command which is eventually created by the script in command.

When you do not set procname to e.g. *beam.smp* you can still run service app start to start the service, but running service app stop will fail because it will check two things:

• find the process matching the PID specified in the pidfile
• check if the process name matches whatever is specified in procname

The first check will succeed because the PID exists. But the second check will fail because the process name generated to run the BEAM is different from the value in command and - if not set - procname.

To solve this issue, I set procname to a globbed string containing beam.smp which works fine. It might become an issue when you have multiple BEAM processes fighting for PIDs.

Should this becomes an issue, try to make the procname more restrictive like e.g.: /usr/local/app/v1.2.3/erts-13.2/bin/beam.smp*.

Finally, to enable this service, add the following line to /etc/rc.conf:

app_enable="YES"

Now you can start and stop your elixir application using the usual service app <start|stop|restart> commands.

TL;DR:

# pkg install elixir-devel erlang-runtime25
# setenv PATH /usr/local/lib/erlang25/bin:$PATH

I'm still relatively new around FreeBSD, so some things are not as obvious to me as others. One thing that took me quite some time was the installation of a relatively recent version of elixir on FreeBSD.

Looking at a pkg search for elixir shows the following packages:

elixir-1.13.4_1                Functional, meta-programming aware language built on top of Erlang VM
elixir-devel-1.14.0            Functional, meta-programming aware language built on top of Erlang VM
[..]

Of course I only saw the first line and didn't bother to look any further. m(

After spending an afternoon learning howto build or update a FreeBSD packages myself using poudriere I came across elixir-devel on freshports. Not my proudest moment.

This elixir version 1.14.0 isn't the latest but it is close enough for now.

Another thing you'll need is erlang. Let's see what pkg search erlang has to offer:

erlang-24.3.4.5,4              Functional programming language from Ericsson
[..]
erlang-runtime21-21.3.8.24_1   Functional programming language from Ericsson
erlang-runtime22-22.3.4.26     Functional programming language from Ericsson
erlang-runtime23-23.3.4.16     Functional programming language from Ericsson
erlang-runtime24-24.3.4.4      Functional programming language from Ericsson
erlang-runtime25-25.1          Functional programming language from Ericsson

Searching for -devel does not help here but a little bit down the list we can find -runtime versions of the package. They are available in different versions and even a relatively recent version of Erlang/OTP 25 is available.

This is all I need, so there is no need to change the FreeBSD release channel for now. Hint: Changing the release channel from quarterly to latest might be a way to get even newer packages without starting to build our own packages.

After installing erlang-runtime25 we're greeted with the following message:

To use this runtime port for development or testing, just prepend
its binary path ("/usr/local/lib/erlang25/bin") to your PATH variable.

We can do so via the following command:

# setenv PATH /usr/local/lib/erlang25/bin:$PATH

We can now use the iex command to start a shell:

# iex
Erlang/OTP 25 [erts-13.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns] [dtrace] [sharing-preserving]

Interactive Elixir (1.14.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

Hint: Yes, these commands were run in a root shell which you can identify by the # in front of every command. I ran all of this in a testing jail which was deleted even before this post was created.

Today I needed a to find a way to run a freebsd-update command exactly once on a certain jail.

Background

Let's take a step back shortly to understand why I need this.

Previously the process to build jails looked as follows:

Do once:

• Download base.txz

Do for every new jail:

• Create a ZFS dataset
• Extract base.txz into the dataset
• Run freebsd-update on the jail
• Configure sshd and set authorized_keys
• Copy resolv.conf
• Configure jails.conf with IPv6/IPv4 addresses
• Start jail

This felt tedious and seemed like something that can be improved. Since ZFS supports very fast and supposedly lightweight snapshots I figured they could be the right tool. After a quick search I found a blog post called "FreeBSD jails the hard way" which describes exactly this scenario. Funny thing: This post is from 2015 and still works like a charm. Do you remember the last time a seven year old blog post helped you solve an issue with kubernetes? It was probably seven years ago.

After a short refactoring I was able to change the process to this:

Do once:

• Download base.txz
• Create a ZFS dataset
• Extract base.txz into the dataset
• Run freebsd-update for the jail
• Configure sshd and set authorized_keys
• Copy resolv.conf
• Create a ZFS snapshot

Do for every new jail:

• Create a ZFS dataset clone from previous snapshot
• Configure jails.conf with IPv6/IPv4 addresses
• Start jail

Check the commit if you're interested in the details..

The solution

And since I do not want to run the slow freebsd-update command on every invocation of ansible, I needed to find a way to only run it once.

This command obviously changes files on the file system. To find these files, I made use of ZFS snapshots and the diff functionality of ZFS.

It's as easy as it sounds.

1. Create snapshot of dataset
2. Run command that alters files
3. Create another snapshot of the same dataset
4. Run zfs diff snapshot1 snapshot2 to get a diff

And of course, after searching through the list of files that were changed, I found this thing:

root@w0:~ # zfs diff zroot/jails/test2/root@pre-install zroot/jails/test2/root@post-install
[..]
+       /usr/local/jails/test2/root/boot/kernel.old/.freebsd-update
[..]

The + at the beginning of the line suggests that this is a file that is present in the second but not the first snapshot. A perfect candidate for our task.

Finally, the ansible task to run the update looks as follows:

- name: Update
  ansible.builtin.command:
    cmd: "freebsd-update -b /usr/local/jails/{{ jails_version }}/root fetch install --not-running-from-cron"
    creates: "/usr/local/jails/{{ jails_version }}/root/boot/kernel.old/.freebsd-update"

Source on github

Someone on the FreeBSD Discord recently mentioned that they're making use of ZFS deduplication for all of their full-jails. While I do not really want to go the route of using deduplication yet I liked the idea to save a lot of space when having multiple jails. Maybe on another day.

Yesterday I tried to find a way to create jails more or less automatically from an ansible configuration.

Today I wanted to have a quick look inside of these jails to understand a bit more about how they work different from the host OS.

But even before I was able to run the first command inside of a jail I had to deal with an issue that makes it very hard to work with jails. Slow startup time.

Have a look at this restart of a jail:

root@w0:~ # time service jail restart test1
Stopping jails: test1.
Starting jails: test1.
0.176u 0.139s 1:49.75 0.2%      151+217k 0+31io 0pf+0w

Stopping the jail was very fast. But starting the jail took the majority of the nearly two minutes.

This is not a playground you can work with, so we need to debug this somehow. After a minute with google I found a couple of hints in the direction of IPv6 and DNS.

To test both IPv6 and DNS I ran a couple of drills:

root@test1:/ # time drill nafn.de aaaa @2620:fe::fe
[..]
nafn.de.        3525    IN      AAAA    2a00:d0c0:200:0:3493:eeff:fe94:1775
0.000u 0.004s 0:00.00 0.0%      0+0k 0+0io 0pf+0w

Seems to work fine. What didn't work so fine was the same command using a IPv4 address.

root@test1:/ # time drill nafn.de aaaa @9.9.9.9
Error: error sending query: Could not send or receive, because of network error
0.003u 0.000s 0:15.21 0.0%      0+0k 0+0io 0pf+0w

At this point I knew two things:

• IPv4 traffic is somehow bugged
• a jail without DNS resolving capabilities will start very slow

Firewall and NAT

Since I already worked with a system like this about 14 years ago (back then I failed misserably), I knew it might have something to do with the firewall configuration.

The server is running at Hetzner and I'm not interested in buying more IPv4 addresses so IPv4 inside of jail is making use of NAT. And while running these drill commands, I remembered that I haven't spent much time with pf yet.

I think I just took the first example I could find in /usr/share/examples/pf/ and stripped it down to something that I can work with and understand.

Of course I only took care of the things in the configuration file that I immediately needed at that time, so the file looked something like this:

ext_if="em0"
[..]
nat on $ext_if from lo0:network to any -> ($ext_if)
[..]

Before we get into why this cannot work, lets see what this line actually means:

"Translate all packets coming from lo0:network to the address associated with $ext_if when they are going out on $ext_if"

This cannot work for multiple reasons. First, the jail IPs are not associated to lo0. Also, $ext_if has multiple addresses associated which kind of makes this bogus. Completely useless so far.

Let's have a look at a configuration that actually works:

ext_if="em0"
ext_ipv4="185.26.156.224"
int_network_range="10.0.0.0/8"
[..]
nat on $ext_if from $int_network_range to any -> $ext_ipv4
[..]

Meaning:

"Translate all packets coming from $int_network_range to the address $ext_ipv4 when they're going out on $ext_if"

It's a bit more static but since all the relevant information is available via ansible, it is automatically changed when running on different hosts.

Did it solve the problem?

It did. After applying this new configuration to the firewall, the jails started very quick and IPv4 traffic from inside of the jail was working too.

root@test1:/ # time drill nafn.de aaaa @9.9.9.9
[..]
nafn.de.        3600    IN      AAAA    2a00:d0c0:200:0:3493:eeff:fe94:1775
[..]
0.000u 0.003s 0:00.03 0.0%      0+0k 0+0io 0pf+0w

root@w0:~ # time service jail restart test1
Stopping jails: test1.
Starting jails: test1.
0.185u 0.162s 0:01.00 34.0%     108+194k 0+31io 0pf+0w

\o/

Wow, a couple days later I see this option on the unarchive module. m(

Original post

While writing an ansible role (source) that would create and setup FreeBSD jails, I needed to find a way to only extract the base.txz file (containing the base operating system) to a directory when it hasn't been done before.

This should only be done once to prevent overwritten files and save time.

My first attempt was a naive check if the directory or a file in the directory already exist.

- name: Check jails for extracted base.txz
    ansible.builtin.stat:
      path: "/usr/local/jails/{{ item }}/root/etc/passwd"
    loop:
      "{{ jails_create.keys() | flatten(levels=1) }}"
    register: jails_stat

This works quite nice. The next step would be to extract the base.txz file under certain conditions.

- name: Extract base.txz
  ansible.builtin.unarchive:
    src: /tmp/base.txz
    dest: "/usr/local/jails/{{ item.item }}/root"
    remote_src: yes
  loop: "{{ jails_stat.results }}"
  when:
    - item is not skipped
    - not item.stat.exists

Instead of looping over the jails_create list, we iterate over the results of the previous check. This allows us to access the required information for the conditions more easily.

Both of these took me about an hour to figure out (of course with the help of stackoverflow and the ansible documentation). And when I finally ran both of these without --check --diff, to actually get a result, I ran into the following error:

Failed to find handler for "/tmp/base.txz".
Make sure the required command to extract the file is installed.

Command "/usr/bin/tar" detected as tar type bsd. GNU tar required.

Unable to find required 'zipinfo' binary in the path.

🧘

So, finally, I replaced both of these with something that is way shorter, easier to read and less complicated:

- name: Extract base.txz to jails
  ansible.builtin.command: "tar -xvf /tmp/base.txz -C /usr/local/jails/{{ item }}/root"
    args:
      creates: "/usr/local/jails/{{ item }}/root/etc/passwd"
    loop:
      "{{ jails_create.keys() | flatten(levels=1) }}"

That's it. Took me less than five minutes and makes me way more happy.

And of course, there are better solutions to managing jails. But I'll take one step after another.

To script the installation of a FreeBSD server (not a jail) you can make use of bsdinstall script. Sadly this doesn't work for jails, because bsdinstall script wants to configure disks and other things that aren't necessary and/or available in jails.

The correct command would be bsdinstall jail, which up until now does not support any scripting. A PR for this feature is already merged into current, but has not yet landed anywhere else.

It took me quite a while to figure out why this change from over a year ago is not yet available on the FreeBSD 13.1-RELEASE machines I have running.

Turns out a quick look into the handbook would've explained it.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#updating-src-obtaining-src

What is named current is the "bleeding-edge" release channel of FreeBSD. It corresponds with the main branch of the freebsd-src repository.

The next best thing is the stable/X branch (where the change also hasn't been seen yet 😢) and finally there is the releng/X.Y branch which holds everything that is part of the OS I'm currently running.

Long story short: It will probably take quite some time until this patch is available in a FreeBSD RELEASE version.

Have a look at this example where I wanted to use psql to connect to a postgres database on a Ubuntu 20.04 LTS server.

$ psql

Command 'psql' not found, but can be installed with:

apt install postgresql-client-common
Please ask your administrator.

So far so good. Let's install it like the computer tells us to:

$ sudo apt-get install postgresql-client-common
[...]

In a perfect world you would imagine that we're done now and can continue our life.

Sadly:

$ psql
Warning: No existing cluster is suitable as a default target. Please see man
pg_wrapper(1) how to specify one.
Error: You must install at least one postgresql-client-<version> package

What? Wtf did I just install?

man 1 pg_wrapper

After wasting time reading this manpage I can tell you:

Apparently I installed a perl wrapper script which intercepts all postgres related commands. It tries to be smart about something that I didn't even knew I needed: Having multiple versions of postgres installed at the same time. Cool, thanks computer.

Eventually, this saved the day:

$ sudo apt-get install postgresql-client-12

Sure, why bother writing about this, right? For years I just ignored every little failure of whatever Linux distribution I was using at the time, but after spending some time with FreeBSD I have to say that it's just a pain to go back to Linux now. Maybe that would've been the better blog post.

I like challenges and I like them even more if I can work on them whenever I want instead of having the pressure to do so immediately.

While trying to find a solution for an exercism exercise called "top-secret" where your task is to decode hidden messages in code, I realized that it requires me to work more than usual to understand what is going on and how to solve it.

To quote Angelika Tyborska while describing the exercises on exercism: "solving a practice exercise takes anywhere from five minutes to five hours". After I spent five hours I thought I could also write a blogpost and dump even more hours into this topic.

Solving the "top-secret" exercise involves something called meta programming which I have heard of before but never interested me enough to take a look at. Maybe I was scared of the complexity that the first sentence of the Wikipedia implies:

Metaprogramming is a programming technique in which computer programs have the ability to treat other programs as their data.

It sounds intriguing but also scary. I already have a hard time reading code, how am I supposed to make the computer read and act on code?! And to be honest, I haven't read more that the first paragraph of the Wikipedia page yet.

This post describes my journey of learning what I think is required to solve the challenge.

The challenge

Your task is to extract secret messages from code. You do so by making use of the name and the arity (amount of arguments to a function) of a function. The arity dictates how many letters of the function name should be taken into account for the message.

An example:

def invert(list), do: Enum.reverse(list)

def lines(list, nth_element) do
  list
  |> Enum.take_every(nth_element)
  |> length
end

def keep(list, only_keep) do
  list
  |> Enum.filter(fn x -> x == only_keep end)
end

def teams(list, team_size, keep_smaller_teams \\ true) do
  list
  |> Enum.split(team_size)
  # TODO: implement keep_smaller_teams
end

Hint: In elixir it is common to refer to a function by appending the arity to it.

Going from top to bottom we have: invert/1 -> i lines/2 -> li keep/2 -> ke teams/3 -> tea

Assembling the messages yields iliketea. So far so good. Doing this for the first couple test cases was easy, but the edge cases made me refactor the code over and over again and breaking what worked before which was a sign for me that I missed something along the way and my approach is probably very wrong.

But let's start at the beginning.

Abstract Syntax Trees

The last time I dealt with Abstract Syntax Trees (AST) was in university. At least I think I dealt with them. Maybe they were different trees. Who knows. As far as I understood it, you might think of ASTs as a way of representing code as data.

Elixir refers to ASTs as quoted expressions and offers a macro called quote which turns some code into a tuple with three elements.

iex(1)> quote do: sum(1,2,3)
{:sum, [], [1, 2, 3]

The first element (:sum) is an atom describing the operation, the second argument is a list describing metadata and the last element is a list which contains other nodes.

quote works a bit different atoms, floats, integers, strings, lists and tuples where it just returns the value. These seem to be the leafs of the tree.

iex(1)> quote do: "hello"
"hello"
iex(2)> quote do: [1,2,3]
[1, 2, 3]
iex(3)> quote do: :hello 
:hello

This already feels like everything we need to move code into data.

On a side node: quote somehow cleared up some things for me (not all of course) when dealing with expressions in Elixir.

iex(1)> quote do: 1+1
{:+, [context: Elixir, import: Kernel], [1, 1]}
iex(2)> quote do: 1==1
{:==, [context: Elixir, import: Kernel], [1, 1]}

The first approach

Let's start by taking the first function of the example from the beginning and see what we get after feeding it to quote:

iex(1)> quote do: def invert(list), do: Enum.reverse(list)
{:def, [context: Elixir, import: Kernel],
 [
   {:invert, [context: Elixir], [{:list, [if_undefined: :apply], Elixir}]},
   [
     do: {{:., [], [{:__aliases__, [alias: false], [:Enum]}, :reverse]}, [],
      [{:list, [if_undefined: :apply], Elixir}]}
   ]
 ]}

This is already quite nested and looks complicated. The metadata field contains values and makes things harder to read and understand. The function name invert became an operation and somewhere at the end we find the actual function call we're running.

In the past I did a lot of web scraping using things like Beautifulsoup and xpath to get what I need from a nested data structure. Naturally I tried the same with elixir and ended with a lot of pipelines looking something like this:

arity = data |> elem(2) |> Enum.at(0) |> elem(2) |> length
message = data |> elem(2) |> Enum.at(0) |> elem(0) |> Atom.to_string() |>
String.slice(0..arity-1)

It works. Sure. But not for long. To account for different structures, I added a couple of case statements but eventually resigned because I wasn't able to make it work and have a solution that is readable.

What next?

Now I'm sitting here and have done two things:

1. I failed to finish the exercise ("in time")
2. I started a blog post without knowing how to continue once I reach the "What next?" point

More to come. Hopefully.

Initially I found my way into this bubble via the sprachnachrichten podcast and a youtube video where a person explained their way of "taking notes".

Let's start with the things I immediately didn't like about logseq:

• Promotes "privacy-first" but you have to manually disable tracking for multiple error-tracking-application-analytics-monitoring-sites after the first start
• You also have to disable tracking on every device separately
• Has keyboard shortcuts that just don't work for me. E.g. 🍎+, is not working
• In general it feels way too mouse-centric
• vim mode is available via a plugin which makes everything a bit more complicated
• Does not allow to disable auto pairing for brackets and back-ticks (I even had to tweet about it)

Let's compare it to what I like about Obsidian:

• Has a working vim mode included
• Only opens one connection to releases.obsidian.md when opening the application
• Has decent keyboard shortcuts that make sense to me
• Allows to disable auto pairing for brackets and back-ticks

After fifteen minutes with obisidian I was ready to pay them even tough it is free for personal use.

One would think that it is quite easy to configure a static IPv4 address on a hetzner cloud server on FreeBSD, but that is not the case. It's not hard either, but took me a couple of minutes to understand what the issue is.

The documentation contains a paragraph about this topic but only mentions DHCP for IPv4:

ifconfig_vtnet0="DHCP"
ifconfig_vtnet0_ipv6="inet6 <one IPv6 address from your subnet, e.g.  2001:db8:0:3df1::1>/64"
ipv6_defaultrouter="fe80::1%vtnet0"

They did the same for NetBSD, so my guess was that this is an uncommon configuration.

This post assumes the following:

• IPv4 address of your host: 1.2.3.4/32
• IPv4 address of the gateway: 172.31.1.1
• Network interface: vtnet0

Issue

Without the static route, you'll run into the following error message when trying to add the default gateway.

root@hetzner-cloud-test:~ # route add default 172.31.1.1
route: writing to routing socket: Network is unreachable
add net default: gateway 172.31.1.1 fib 0: Network is unreachable

Howto

Usually it is enough to configure your /etc/rc.conf to something like:

ifconfig_vtnet0="inet 1.2.3.4 netmask 255.255.255.255"
defaultrouter="172.31.1.1"

But, to get around the Network is unreachable error, we also need to create a static route like so:

static_routes="vtnet0"
route_vtnet0="172.31.1.1 -iface vtnet0"

That's it. Reboot and everything should work.

Alternatively, to make it work at runtime, run the following commands:

ifconfig vtnet0 inet 1.2.3.4 netmask 255.255.255.255
route add 172.31.1.1 -iface vtnet0
route add default 172.31.1.1

I tested it multiple times with RDS/Aurora instances and it works as expected, even without the --apply-immediately parameter (at least for the instance class modifications).

Full Example

Let's change the instance class for an Aurora instance called database-2-instance-1 from db.t3.medium to db.r4.large and undo it again afterwards. None of these commands do impact the availability of the database.

Note: Using jq here to only output the important parts.

Check instance class

aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].DBInstanceClass'
"db.t3.medium"

Validate that there is no pending modification

aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].PendingModifiedValues'
{}

Modify instance class

This modification will result in a change of the instance class in the next maintenance window.

aws rds modify-db-instance --db-instance-identifier database-2-instance-1 --db-instance-class db.r4.large | jq '.DBInstance.PendingModifiedValues'
{
  "DBInstanceClass": "db.r4.large"
}

Validate again

Just to be sure, check if everything looks as expected.

aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].PendingModifiedValues'
{
  "DBInstanceClass": "db.r4.large"
}

Undo modify of instance class

This is the important part which modifies the instance class back to the old value. The documentation describes that a --apply-immediately is required, but it turns out that that is not the case. At least in this example.

aws rds modify-db-instance --db-instance-identifier database-2-instance-1 --db-instance-class db.t3.medium | jq '.DBInstance.PendingModifiedValues'
{}

Validate the removed pending modification

aws rds describe-db-instances --db-instance-identifier database-2-instance-1 | jq '.DBInstances[].PendingModifiedValues'
{}

At this point the pending modification is gone.

Over time a time machine backup will throw errors that either state that the backup cannot be done because of various reasons or the backup needs to be re-created.

I fixed all errors with the following steps and need to write them down to prevent me from searching for this information again in the future.

Error messages

Recreate backup

Time Machine completed a verification of your backups.
To improve reliability, Time Machine must create a new backup for you.

Backup cannot be done

Time Machine could not complete the backup. (error 45)

Solution

To speed things up it is recommended to copy the backupbundle (formerly sparsebundle) file to your workstation, attach the volume and then run the disk repair command.

Attach the volume

$ hdiutil attach -nomount -readwrite foobar.backupbundle
/dev/disk2              GUID_partition_scheme
/dev/disk2s1            EFI
/dev/disk2s2            Apple_HFS

Remember the value of the first column on the the Apple_HFS line! (/dev/disk2s2)

This is the partition we're going to repair.

Run the disk repair command

fsck_hfs -rf /dev/disk2s2

fsck_hfs is the "HFS file system consistency check" tool.

• -r: Rebuild the catalog btree.
• -f: Force fsck_hfs to check and repair journaled HFS+ file systems.

The command takes ages to complete and might look something like this:

$ fsck_hfs -rf /dev/disk2s2
** /dev/rdisk2s2
   Executing fsck_hfs (version hfs-522.0.9).
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is Time Machine Backups
** Checking extents overflow file.
** Checking catalog file.
** Rebuilding catalog B-tree.
** Rechecking volume.
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is Time Machine Backups
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking multi-linked directories.
** Checking volume bitmap.
** Checking volume information.
   Invalid volume file count
   (It should be 10384127 instead of 10079428)
   Invalid volume directory count
   (It should be 1557312 instead of 1514027)
   Invalid volume free block count
   (It should be 370750423 instead of 376119386)
   Volume header needs minor repair
(2, 0)
** Repairing volume.
** Rechecking volume.
** Checking Journaled HFS Plus volume.
** Detected a case-sensitive volume.
   The volume name is Time Machine Backups
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking multi-linked directories.
** Checking volume bitmap.
** Checking volume information.
** Trimming unused blocks.
** The volume Time Machine Backups was repaired successfully.
$

Now you need to detach the volume and move it back to the location where it came from (external disk, NAS ...).

Detach volume

$ hdiutil detach /dev/disk2
"disk2" ejected.
$

Reset plist

If your error message states that you need to create a new backup, you must also change two files inside the backupbundle/sparsebundle file.

Inside the bundle file, you'll find these files:

• com.apple.TimeMachine.MachineID.plist
• com.apple.TimeMachine.MachineID.bckup

In both files, you'll need to remove the following two lines (if they're present):

        <key>RecoveryBackupDeclinedDate</key>
        <date>2020-01-04T09:41:40Z</date>

You'll also need to change the following line:

Before:

        <key>VerificationState</key>
        <integer>2</integer>

After:

        <key>VerificationState</key>
        <integer>0</integer>

Done

Good luck.

On 01.04.2019 I found a SQL injection vulnerability on the CMS login page of the gruenberg.de website and immediately forwarded this information to the CERT Hessen.

I tend to contact the corresponding CERT instead of the owner of the website, because it's hard to deal with them in most cases. Sometimes they do not answer at all, sometimes they just ignore you and the issue, sometimes they do not accept emails from outside their domain (this happened multiple times already!) and most of the time they do not understand what you want from them.

Timeline

• 01.04.2019: Found issue and contacted CERT Hessen
• 02.04.2019: CERT Hessen acknowledged the issue
• 10.04.2019: CERT Hessen informed me about the issue beeing fixed

CERT Hessen rating

The CERT Hessen is easy to contact because everything you need you can find with a simple google search, they react fast and inform you as soon as the problem is fixed. That's an awesome experience when dealing with this kind of situation where it is not clear what is going to happen.

Aus dem Abstract:

Terraform erlaubt es dir Infrastruktur in Code abzubilden und diesen Code auf verschiedene Anbieter anzuwenden.

Die Möglichkeit, versionierten Code als Lageplan einer Infrastruktur vorzuhalten, ist ein Mehrgewinn der seinesgleichen sucht. Egal ob die gesamte Infrastruktur bei AWS liegt oder ueber mehrere Anbieter verteilt ist: terraform ist hier eine helfende Hand, um die steigende Komplexitaet ansatzweise im Griff zu behalten.

Und da bei Synyx keine halben Sachen gemacht werden, wurde das ganze auch direkt aufgezeichnet. Danke vielmals fuer diese Erfahrung!

Aufzeichnung: https://www.youtube.com/watch?v=Ki1mJTmQhqM
Code+Folien: https://github.com/temal-/terraform_talk

The simulation is available here and the code is on github.

Update a page via curl:

curl -u $user -H 'Content-Type: application/json' -d '{"id":"1234567890","type":"page","title":"Stuff","space":{"key":"SPACE"},"body":{"storage":{"value":" foo ","representation":"storage"}},"version":{"number":7}' https://confluence.example.com/rest/api/content/1234567890

Update a child page via curl:

curl -u $user -H 'Content-Type: application/json' -d '{"id":"1234567890","type":"page","ancestors":[{"type":"page","id":987654321}],"title":"Stuff","space":{"key":"SPACE"},"body":{"storage":{"value":" foo ","representation":"storage"}},"version":{"number":7}' https://confluence.example.com/rest/api/content/1234567890

via Scott Dudley

$ curl -O https://foo.com/bar-1.23.112-alpha-RC0.1.tar.gz
$ tar xzf bar* && cd bar*
$ ./configure
$ make
$ sudo make install

Then, I guess, you also know this.

$ sudo make uninstall
make: *** No rule to make target `uninstall'.  Stop.

To prevent this, you could use fpm to create a package for your distribution which is way easier to manage.

After a

$ make install DESTDIR=/tmp/pkg

you can create a deb or rpm with a:

$ fpm -s dir \
    -t deb \
    -n "git" \
    -v 1.9.0 \
    --prefix /usr/local \
    -C /tmp/pkg \
    bin lib libexec share
Created deb package {:path=>"git_1.9.0_amd64.deb"}

• "-s dir" defines the source as a directory.
• "-t deb" defines the target as a debian package.
• "-n 'git'" defines the name of the package.
• "-v 1.9.0" defines the version of the package.
• "--prefix /usr/local" defines a prefix for all files installed from the created package
• "-C /tmp/pkg" defines a path to which fpm changes before searching for the folders at the end of the command.
• "bin lib libexec share" defines the names of the directories to pack into the package.

This command creates a deb in your $PWD named git_1.9.0_amd64.deb. If you want a rpm, just change -t deb to -t rpm.

Thanks a lot to Jordan Sissel for this timesaver!

In a night shift, in mid 2012, I played around with a couple of apps on my mobile, a http proxy (mitmproxy) and tcpdump to find a way to MITM some of these apps.

After a dozen of apps (most of them failed because of strict certificate checks on the HTTPS front) and way to much coffee, my shift was going to an end. So I started my favorite bike-renting app and looked for the nearest bike which should bring me home. After finding one, I noticed that I still had a proxy configured on my phone and that the app didn't even remotely complain about the wrong certificate (this is fixed now). Jackpot ... I thought ... and went home to bed because it was 8am and I was tired enough. But that should be a good starting point for some "data collection and visualization".

The next night started with some traffic analyzing and a fast discovery of a pair of credentials for the (more or less) public API endpoint. "More or less", because it needs some very basic authentication, based on your mobile device. You'll know what I mean if you look at this snippit from the Ik4ru5/pyABike repository.

    userData.User = 't_cab_android' #from android app
    userData.Password = 'DELETED TO PREVENT PROBLEMS' #from android app

Source: Github

And since I have an iPhone, the credentials on my phone are a bit different. Here is a SOAP-Request I use to gather my data. And since I only need one API endpoint for my queries, I didn't start bothering with a python SOAP library and just copy/paste the query from my phones traffic.

  def createRequest(self, lati, longi, radius, amount):
    """ Returns an XML-Set of all the bikes around the specific coordinates """
    data = """<?xml version="1.0" encoding="utf-8"?>
    <SOAP:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
    <SOAP:Body>
    <foo.listFreeBikes xmlns="https://xml.dbcarsharing-buchung.de/hal2_cabserver/">
    <CommonParams>
    <UserData>
    <User>t_iphone_cab</User>
    <Password>DELETED TO PREVENT PROBLEMS</Password>
    </UserData>
    <LanguageUID>1</LanguageUID>
    <RequestTime>%s</RequestTime>
    <Version>2</Version>
    </CommonParams>
    <SearchPosition>
    <Longitude>%.6f</Longitude>
    <Latitude>%.6f</Latitude>
    </SearchPosition>
    <maxResults>%i</maxResults>
    <searchRadius>%i</searchRadius>
    </foo.listFreeBikes>
    </SOAP:Body>
    </SOAP:Envelope>"""% (datetime.datetime.now().strftime("%Y-%m-%dT%H:%M:%S"), longi, lati, amount, radius)
    return data

Do you see the professional consistency of the usernames? "t_cab_android" vs "t_iphone_cab". Amazing ...

To make it clear: These credentials are only useful to query the backend for bike-data and not user-data.

Nonetheless, I used a couple selfmade python modules to gather this data periodically and save it to a database.

The application logic is very simple:

Look at a certian position (longitude, latitude) for bikes in a range of n meters. If a bike is found, check the database for an entry of that bike. If we don't have an entry for that bike, we've never seen it before and create a new entry. If - on the other hand - we know that bike already, we set the current position and add the last position to the history.

With this little logic we can keep track of every bike for however long we're interested in it and it's path through the city.

And here is an entry of a bike in the database:

> db.bikes.findOne()
{
        "_id" : ObjectId("500c6b57ea0f273df592c45a"),
        "bikenr" : 6935,
        "currentPos" : [
                50.10824,
                8.672381
        ],
        "history" : [
                {
                        "pos" : [
                                50.121815,
                                8.614588
                        ],
                        "time" : ISODate("2012-07-23T00:16:10.997Z")
                },
                {
                        "pos" : [
                                50.121986,
                                8.614226
                        ],
                        "time" : ISODate("2012-07-23T07:30:47.639Z")
                },
                {
                        "pos" : [
                                50.1220567603,
                                8.6142222359
                        ],
                        "time" : ISODate("2012-07-23T07:55:46.248Z")
                }
        ],
        "lastseen" : ISODate("2012-07-23T10:47:06.573Z")
}
> db.bikes.count()
745
}

As you can see, the positions are sometimes nearly the same, so I guess this is the 24h theft-protection of the bike, which sends it's position to the backend once every day if it hasn't moved.

Finaly, a OpenStreetMap-leaflet combination to visualize the data:

Watch it here in full. The Boroughs are colored by the amount of bikes that are currently available and the heatmap shows some data about the requests I made against the Bike-API. This data is static, as I don't want to have my unsecure REST-API available on the internet.

Credit for the GeoJSON of the Frankfurt boroughs goes to the folks at Frankfurt Gestalten and DailyMo. Thanks also to Ik4ru5, because without the knowledge of other people playing with the Bike-API, I'd never have started to visualize the data.

If I'll ever find fun in this type of data again, I'd definitely use ElasticSearch as a replacement for MongoDB and Kibana3 as a replacement for my DIY-Leaflet map. Just to try it out.

EDIT: I deleted all signs of brands and so on to save myself from problems.