Published at January 14, 2023 · 8 min readShare on:
We’re pleased to announce Spokes v1.4.0! This release introduces new features, improvements, and fixes to user-reported issues.
We removed the need to include the
database property in the configuration. Since we use SQLite for our database and use the
dataPath property is used to identify where all application data is stored, we just use that path as the root path for the database.
Older configurations that still use the
database property are unaffected. This small change to defaults in the configuration is meant to simplify configurations.
Extra Certificates & TLS Termination
TLS termination at Spokes is a new and useful feature since it allows users to deploy a single wildcard certificate that can be used across many tunnels without distributing the certificate or private key.
It’s more efficient than using individual certificates, and in deployments where Let’s Encrypt is utilized heavily and rate-limits reached quickly, this provides a solution.
Spokes use a new location in the application directory
/var/lib/spokes/certs-extra to store these additional certificates. Individual site certificates or wildcard certificates can be stored here. Self-signed certificates can be utilized for serving and terminating TLS as well.
When an incoming HTTPS request is received by Spokes, it will determine if it can perform a TLS handshake with that client. If so, it will terminate TLS and relay the traffic to a corresponding tunnel. The request will be modified to include additional HTTP headers to help the upstream understand the client’s IP address and that the original request was an HTTPS request.
Spokes uses standard HTTP headers, here’s an example of them:
X-Forwarded-For: <ip address> X-Forwarded-Host: subdomain.domain.com X-Forwarded-Proto: https
To use this features, admins will need to deploy TLS certificates and their corresponding keys in the
/var/lib/spokes/certs-extra directory using the following conventions:
- Each directory should represent a single site or a wildcard certificate
- Certificates and the private key are required and need to be PEM encoded
- Certificates must use the extension
- Private keys must use the extension
TLS traffic is proxied to plain-text rule over TLS-tunnel to clients.
Here is an example
tree output of the the
certs-extra/ ├── site-one.domain.com │ ├── site-one.domain.com.crt │ └── site-one.domain.com.key ├── wildcard_.domain-two.com │ ├── wildcard_.domain-two.com.crt │ └── wildcard_.domain-two.com.key
It doesn’t matter what the name of the sub-directories under
certs-extra are as long as there is a certificate and private key that are PEM encoded and use the right extensions.
Extra certificates used for TLS termination at Spokes can be copied into
/var/lib/spokes/certs-extra and picked up automatically by Spokes after an hour (at most).
Very important to note is that by default, some partial matching will be performed if the certificates share the same root domain as an incoming request but do not match perfectly. In the future, this behavior will be an option.
Upcoming releases will include additions to the admin API endpoints for listing extra certificates, adding, updating, and removing them. We will also upgrade the admin UI dashboard to support those functions too.
Tunnel Details UI Updates in Dashboard
We’ve made minor changes to our tunnels details page in the admin dashboard to improve the user experience and add new features.
We’ve renamed the
Shutdown button to
Terminate since that is what describes this activity’s behavior. Initially, we wanted to use the same terminology to manage virtual machines, such as
Shutdown. However, we’ve had users confuse
Shutdown with just stopping the tunnel. This has created many accidental tunnel deletions (we’re sorry). We hope the actions are more clear now.
The renamed label on the
Terminate button clarifies what will happen when you click it.
A new button added to the tunnel page is the
Stop Session button. This button will terminate the tunneling session and cause the tunnel to disconnect and auto-reconnect to the Spokes server. This might be helpful to diagnose the tunnel or reset the state of the tunnel without killing the remote process or having to log into the host running the tunnel to restart it.
User Management and Restrictions
In this release, we added the admin API and UI dashboard ability to set or update per-user traffic restrictions. This is useful if you’re using users to segregate tunnels.
For some time, the admin API had the ability to set the list of domains that can be requested for HTTP/S traffic rules. When a requested domain is not in the set of domains associated with a user, then the traffic rule will be rejected and won’t be served.
This allows admins to create restrictions and controls on which tunnels can serve what domains for HTTP/S requests.
The user interface in the tunnel details page has been updated so the list of domains associated to the user is visible. Domain associations can be removed using the
Remove button in the table.
New domains can be added to a users valid set by clicking on the
Add Domain button and inputting a valid domain, including sub-domains.
When tunnels connect to the Spokes server, any request HTTP rules will be evaluated against this list of domains when the users associated with the tunnel have the
Check Domain property set to true.
In this release, other new properties called Max Ports and Check Ports were added to users and can be used to control per-user the number of ports a user can allocate.
Check Ports is a boolean flag that indicates the user has custom limitations set to them.
These are evaluated when incoming requests to allocate new ports are received and when TCP port forwarding traffic rules are received.
Other properties we’re added to the user as well. The
Bandwidth property is used to store what the intended maximum bandwidth for a user across all their tunnels, can be.
Max Bandwidth is a flag that indicates a user has reached their bandwidth max. Once this flag is set, no more traffic rules will be accepted until it is reset to
These features are used heavily in managing user bandwidth quotas for Packetriot.com. Still, they can be used for any other application where bandwidth or ports need to be limited, and domain associations are used to segment tunnels and which traffic they can possibly request and receive.
Backup & Archive Script
We’ve included a new script available in
archive.sh. This script will archive the licenses, database(s), configuration (including the server certificates) and additional custom certificates stored in
The connection metrics database
spokes-metrics.db is not stored since it can be large depending on the monthly connections received.
The archive script is a template that can be customized. It utilizes the minio client to connect and push the tarball containing the archive to S3-compatible services. We have also included a sample
cron task that can be used to customize scheduled backups.
Note, if any customizations are made to the
archive.sh script please store them in another location and not in the
/usr/share/spokes directory. They may get overwritten in future updates or prevent updates since a package conflict exists.
New House-cleaning Background Tasks
There is a known issue about the slow growth of disk consumption due to unmanaged web sessions directory in
In this release, we updated the
spokes clean command to make
stale-keys a sub-command and added
sessions as a new sub-command to clean up the web sessions directory.
We’ve added a nightly
cron task at
00:15 to run the
spokes clean sessions command to maintain the sessions directory so it doesn’t grow and eventually fill the host’s disk.
More Webhook Events
In this release, we added new webhook events for modifications of the tunnel and connection metrics.
Right now, the only edit for the tunnel that triggers a webhook event is changing the tunnel name. The tunnel object, with the new name, is published for this event.
The webhook event for connection events enables publishing connections and sessions and tracking incoming IPs and which services they request. The amount of data transferred per connection session.
Since connection metrics could generate a large number of objects and also be published extremely frequently, we recommend that you create a separate webhook for these events and include a delay for at least a few seconds.
You can update existing webhooks by clicking the edit button and checking off the newly added events.
We’ve refactored some areas of Spokes to ensure more code safety, hopefully resolving a still very difficult to-duplicate bug that we suspect is related to a race condition.
We also fixed a bug in the software releases section of our UI that would error silently when a release for an architecture it does not understand is received. This was the case for older Spokes instances that pulled down new releases for the newly added MIPS 32 and 64-bit targets for the client.
We appreciate all the inputs and suggestions you’ve sent us! The Spokes v1.4.x releases will iterate on these new features and make improvements and refinements. Let us know if we can focus on any new features or improvements we can add to Spokes.