Spokes Update v1.4.6

Published at June 10, 2024 ·  5 min read

We’re pleased to announce Spokes v1.4.6. This release add new features and fixes some small bugs.

Get the release!

You can download the RPM here. Our container is available at our packetriot repo and now the Terrapin Labs repo as well.

docker pull packetriot/spokes:1.4.6

# alternatively

docker pull terrapinlabs/spokes:1.4.6

External Database Support

Spokes has used SQLite as it’s backend database since the beginning and we made that choice in it’s design so that we can reduce dependencies to zero and make deployment easier and faster.

We worked with a customer to setup a high-available (HA) instance of Spokes and they wanted to use MySQL as the backend database since their team had a lot of expertise in setting up a Master-Master configuration in MySQL. This particular database setup made operations of this HA setup extremely easy since database replication was managed by MySQL.

Spokes supports these external databases: MySQL, MariaDB, and Postgres.

Updates to the configuration file were necessary to support these new databases. A property in the JSON configuration called database must be added and include properties of it’s own: driver and dsn.

The driver property can include these values: mysql and postgres. MariaDB is a fork of MySQL and the wire protocol for MySQL and MariaDB are identical.

The term DSN stands for data source name, and it essentially a URL for connecting to the database. It includes the hostname or IP, username, password, the database name, and any options. The DSN format is different for MySQL and Postgres databases and we’ll provide examples below.

MySQL and MariaDB

The MySQL driver used in Spokes is the go-gorm driver for MySQL. This page includes more details on the DSN, it’s format and options. Below is an example of a database property with the driver and DSN.

{
	"database": {
		"driver": "mysql",
		"dsn": "root:foobar@tcp(127.0.0.1:3306)/spokes?charset=utf8mb4\u0026parseTime=true"
	},
}

Postgres

We are using the go-gorm driver for Postgres. The DSN for connecting to postgres is much different. You’ll find some example in driver page we linked to, but below is an example that you can use as well.

{
	"database": {
		"driver": "postgress",
		"dsn": "host=localhost user=spokes password=foobar dbname=spokes port=9920 sslmode=disable TimeZone=America/New_York"
	},
}

Before using a MySQL/MariaDB or Postgres database, you must ensure that it is set up and accessible with the values you’ll be specifying in the DSN. You need to make sure that authentication is configured and that the database exists.

We have been using the database name spokes but you can choose another name and customize it.

In the future we’d like to explore a tutorial that documents how to set up MySQL or Postgres in a Master-Master replication mode since it’s a very robust configuration and makes operating an HA instance of Spokes more simple.

Connection Tracking is Optional

This release introduces an optional to disable connection tracking in Spokes. Each connection to a service hosted behind a tunnel is logged and stored in the database.

Depending on how many connections are served, the database can grow quite large and it the accumulation of connection records outpaces our scheduled garbage collection of stale metrics, it may consume disk space and contribute to a fully consumed disk.

You can visit the /var/lib/spokes directory and inspect the size of the spokes-metrics.db file to determine if this is a concern.

This isn’t a concern for vast majority of our customers, but its a useful option.

If you have setup webhooks and publish connection events to an external log collector, like an Opensearch or splunk instance, you will continue to get those events and that data published to those endpoints.

To toggle storing connection metrics, log into your Spokes instance and visit the Settings page, then select the Tunnel tab. You can turn connection logging off and on from here.

Data Export & Import

We’ve added new commands to the spokes CLI to enable admins to export data from the Spokes database to a flat JSON file. Data in the JSON file format can be imported in Spokes as well. This is helpful if an organization is migrating to a different database, e.g. going from SQLite to using Postgres. It’s also helpful is a team is consolidating multiple instances into one larger Spokes instance.

These new commands are part of the migrate commands, below is the output:

[user@host ~]$ spokes migrate
Usage:
  spokes migrate [flags]
  spokes migrate [command]

Available Commands:
  db-export   Export data from the database to JSON
  db-import   Import data from a JSON to the database

Flags:
  -h, --help   help for migrate

Global Flags:
      --config string   path to configuration file

Use "spokes migrate [command] --help" for more information about a command.

You can use the db-export and db-import sub-commands for these export/import operations.

The db-export command supports flags to indent the outputted JSON file so that it is more readable. You can use the --pretty flag to enable that output. Alerts and messages stored by Spokes can be numerous, so exporting those records is optional. To include them use the --messages flag.

Additional details on the db-export command are below.

[user@host ~]$ spokes migrate db-export --help
Export data from the database to JSON

Usage:
  spokes migrate db-export [flags]

Flags:
  -h, --help          help for db-export
      --json string   filename for JSON output
      --messages      export messages, events and alerts
      --pretty        output JSON in readable format

Global Flags:
      --config string   path to configuration file

The db-import command requires one parameter to be passed in which is the JSON output from db-export.

spokes migrate db-import --json /path/to/export.json

Be sure to run these commands as the spokes user or as root if you’re deploying Spokes as a container.

We are planning to enhance this functionality further by adding export and import buttons into the web dashboard UI so that CLI access is not necessary. We may further enhance the export functionality by included necessary artifacts to backup the Spokes server such as the configuration, license files, and TLS certificates.

Thanks!

Thanks to our customers who’ve been sending us these feature requests. Our roadmap is heavily influenced by customer needs and requests, and they help us plan improvements that are relevant to your needs.

We also appreciate the bug reports too! Let us know if we can focus on any new features or improvements we can add to Spokes.

Cheers!