Spokes v1.4.0

Published at January 14, 2023 ·  8 min read

We’re pleased to announce Spokes v1.4.0! This release introduces new features, improvements, and fixes to user-reported issues.

Configuration Defaults

We removed the need to include the database property in the configuration. Since we use SQLite for our database and use the dataPath property is used to identify where all application data is stored, we just use that path as the root path for the database.

Older configurations that still use the database property are unaffected. This small change to defaults in the configuration is meant to simplify configurations.

Extra Certificates & TLS Termination

TLS termination at Spokes is a new and useful feature since it allows users to deploy a single wildcard certificate that can be used across many tunnels without distributing the certificate or private key.

It’s more efficient than using individual certificates, and in deployments where Let’s Encrypt is utilized heavily and rate-limits reached quickly, this provides a solution.

Spokes use a new location in the application directory /var/lib/spokes/certs-extra to store these additional certificates. Individual site certificates or wildcard certificates can be stored here. Self-signed certificates can be utilized for serving and terminating TLS as well.

When an incoming HTTPS request is received by Spokes, it will determine if it can perform a TLS handshake with that client. If so, it will terminate TLS and relay the traffic to a corresponding tunnel. The request will be modified to include additional HTTP headers to help the upstream understand the client’s IP address and that the original request was an HTTPS request.

Spokes uses standard HTTP headers, here’s an example of them:

X-Forwarded-For: <ip address>
X-Forwarded-Host: subdomain.domain.com
X-Forwarded-Proto: https

To use this features, admins will need to deploy TLS certificates and their corresponding keys in the /var/lib/spokes/certs-extra directory using the following conventions:

• Each directory should represent a single site or a wildcard certificate
• Certificates and the private key are required and need to be PEM encoded
• Certificates must use the extension .crt
• Private keys must use the extension .key

TLS traffic is proxied to plain-text rule over TLS-tunnel to clients.

Here is an example tree output of the the certs-extra directory:

certs-extra/
├── site-one.domain.com
│     ├── site-one.domain.com.crt
│     └── site-one.domain.com.key
├── wildcard_.domain-two.com
│     ├── wildcard_.domain-two.com.crt
│     └── wildcard_.domain-two.com.key

It doesn’t matter what the name of the sub-directories under certs-extra are as long as there is a certificate and private key that are PEM encoded and use the right extensions.

Extra certificates used for TLS termination at Spokes can be copied into /var/lib/spokes/certs-extra and picked up automatically by Spokes after an hour (at most).

Very important to note is that by default, some partial matching will be performed if the certificates share the same root domain as an incoming request but do not match perfectly. In the future, this behavior will be an option.

Upcoming releases will include additions to the admin API endpoints for listing extra certificates, adding, updating, and removing them. We will also upgrade the admin UI dashboard to support those functions too.

Tunnel Details UI Updates in Dashboard

We’ve made minor changes to our tunnels details page in the admin dashboard to improve the user experience and add new features.

We’ve renamed the Shutdown button to Terminate since that is what describes this activity’s behavior. Initially, we wanted to use the same terminology to manage virtual machines, such as Create, Start, Stop, and Shutdown. However, we’ve had users confuse Shutdown with just stopping the tunnel. This has created many accidental tunnel deletions (we’re sorry). We hope the actions are more clear now.

The renamed label on the Terminate button clarifies what will happen when you click it.

A new button added to the tunnel page is the Stop Session button. This button will terminate the tunneling session and cause the tunnel to disconnect and auto-reconnect to the Spokes server. This might be helpful to diagnose the tunnel or reset the state of the tunnel without killing the remote process or having to log into the host running the tunnel to restart it.

User Management and Restrictions

In this release, we added the admin API and UI dashboard ability to set or update per-user traffic restrictions. This is useful if you’re using users to segregate tunnels.

For some time, the admin API had the ability to set the list of domains that can be requested for HTTP/S traffic rules. When a requested domain is not in the set of domains associated with a user, then the traffic rule will be rejected and won’t be served.

This allows admins to create restrictions and controls on which tunnels can serve what domains for HTTP/S requests.

The user interface in the tunnel details page has been updated so the list of domains associated to the user is visible. Domain associations can be removed using the Remove button in the table.

New domains can be added to a users valid set by clicking on the Add Domain button and inputting a valid domain, including sub-domains.

When tunnels connect to the Spokes server, any request HTTP rules will be evaluated against this list of domains when the users associated with the tunnel have the Check Domain property set to true.

In this release, other new properties called Max Ports and Check Ports were added to users and can be used to control per-user the number of ports a user can allocate. Check Ports is a boolean flag that indicates the user has custom limitations set to them.

These are evaluated when incoming requests to allocate new ports are received and when TCP port forwarding traffic rules are received.

Other properties we’re added to the user as well. The Bandwidth property is used to store what the intended maximum bandwidth for a user across all their tunnels, can be. Max Bandwidth is a flag that indicates a user has reached their bandwidth max. Once this flag is set, no more traffic rules will be accepted until it is reset to false.

These features are used heavily in managing user bandwidth quotas for Packetriot.com. Still, they can be used for any other application where bandwidth or ports need to be limited, and domain associations are used to segment tunnels and which traffic they can possibly request and receive.

Backup & Archive Script

We’ve included a new script available in /usr/share/spokes called archive.sh. This script will archive the licenses, database(s), configuration (including the server certificates) and additional custom certificates stored in /var/lib/spokes/certs-extra.

The connection metrics database spokes-metrics.db is not stored since it can be large depending on the monthly connections received.

The archive script is a template that can be customized. It utilizes the minio client to connect and push the tarball containing the archive to S3-compatible services. We have also included a sample cron task that can be used to customize scheduled backups.

Note, if any customizations are made to the archive.sh script please store them in another location and not in the /usr/share/spokes directory. They may get overwritten in future updates or prevent updates since a package conflict exists.

New House-cleaning Background Tasks

There is a known issue about the slow growth of disk consumption due to unmanaged web sessions directory in /var/lib/spokes/sessions.

In this release, we updated the spokes clean command to make stale-keys a sub-command and added sessions as a new sub-command to clean up the web sessions directory.

We’ve added a nightly cron task at 00:15 to run the spokes clean sessions command to maintain the sessions directory so it doesn’t grow and eventually fill the host’s disk.

More Webhook Events

In this release, we added new webhook events for modifications of the tunnel and connection metrics.

Right now, the only edit for the tunnel that triggers a webhook event is changing the tunnel name. The tunnel object, with the new name, is published for this event.

The webhook event for connection events enables publishing connections and sessions and tracking incoming IPs and which services they request. The amount of data transferred per connection session.

Since connection metrics could generate a large number of objects and also be published extremely frequently, we recommend that you create a separate webhook for these events and include a delay for at least a few seconds.

You can update existing webhooks by clicking the edit button and checking off the newly added events.

Bug Fixes

We’ve refactored some areas of Spokes to ensure more code safety, hopefully resolving a still very difficult to-duplicate bug that we suspect is related to a race condition.

We also fixed a bug in the software releases section of our UI that would error silently when a release for an architecture it does not understand is received. This was the case for older Spokes instances that pulled down new releases for the newly added MIPS 32 and 64-bit targets for the client.

Thanks!

We appreciate all the inputs and suggestions you’ve sent us! The Spokes v1.4.x releases will iterate on these new features and make improvements and refinements. Let us know if we can focus on any new features or improvements we can add to Spokes.

Cheers!