Choosing Your Services and Database

The goal of creating a self-hostable application is, well, allowing it to be self-hosted. That means also allowing others to self host your app. It has to be straight forward to do and maintain. As mentioned in the previous article, Importance of Repeatable Environments, we use Docker for everything and recommend you do the same. However, there are still some considerations when choosing the database and what services you include.

Basic Considerations

The power of Docker is amazing. You can add an extremely powerful full text, Meilisearch container to your project with minimal set up time. Configure your Docker compose file and you are off to the races.

However, that Meilisearch container requires a lot more resources, disk space, and maintenance. Even if you are licensing your self-hostable app to a business, it might be overkill for what they are using. They might only have 10-15 users and want the quickest solution to get up and running.

But that's one consideration. What if you want to re-use the same code for a cloud hosted version that does have a million users? These type of architecture and user decisions really have to come into play when designing your app. So what do you do? Well, there are options.

Service Options (Queues, Search, etc.)

The consideration for full text searching is similar for other processes as well. When we approached designing Bugflow we ran into this issue with our queues. We wanted to keep our self-hosted version as small as possible so it's easier to distribute, maintain and deploy on their own hardware. Why? Because the use case for our customers who want to self-host won't run into the same usage scenarios as we do (if they do, we offer an enterprise self-hosted version).

Architecture

For reference, before moving forward, we use Laravel to build our apps, and they make these next steps a breeze. All connection information for services is handled through the ENV file. I'm sure other frameworks have this as well, but the beauty comes from the abstraction layer above. If we dispatch a queue job, like we do for every feedback request, Laravel reads the configuration connection and dispatches the job where it needs to go. If you wanted to use SQS, Horizon (Laravel's powerful queue runner), or just a database, the job gets routed where it needs to go.

If you aren't using a framework that supports this architecture, I'd recommend developing something similar for your app. I'd recommend a data transfer object, a contract, and some env configuration that allows you to easily swap out what service you need depending on infrastructure and usage requirements.

How We Solved This

With Bugflow, on our cloud hosted version, we need to process multiple simultaneous queue jobs to ensure everything gets processed as quickly as possible. For self-hosted instances, we just use the database. This is all managed from our environment variable. That means we aren't bringing up an unnecessary server for a use-case that our customers don't have. It also means that users paying for our SaaS get the benefit of speedy feedback on our Horizon Queue.

You can also solve this similarly for full text searching depending on the database you choose (see next section).

Database Options

With Docker, the database is, guess what, another service and container! This means that when you bring up your app, you might have to bring up another service as well. However, the database is slightly different and depending on your use case, you could possibly remove the container entirely.

Let's say you want to get an application as small as possible to make it as simple as possible to distribute. You need full text search, queuing, and a database. Forget cloud for now, you can do whatever you want since you manage the infrastructure, let's focus on self-hosting.

The first thing I'd do is move all queued tasks to a "jobs" table on the database. This will eliminate the need for a queue runner and another overkill container. Next, I'd choose PostgreSQL or another database that supports full text indexing. You can then eliminate the huge overhead of another full text search engine running in your app. This is just a consideration! If your app heavily relies on natural language processing or super advanced indexing, this might not be worth it. However, you'd be surprised how far you can get!

If you really want to cut down and won't have an app that requires an insanely powerful database, you can even use SQLite. This removes the need entirely for even a database container! You could literally get down to deploying just your app code to self-hostable instances. We actually do this with Bugflow as well. On self hosted instances, SQLite runs our database and our queues. It makes our deployment extremely light weight and easy to run anywhere.

Building for Cloud and Self-Hosted

Now you've heard me talk about how we deploy to self-hosted environments and to cloud using the same code base. How does this work? Well, if you use Docker, it's a lot easier than you may think!

The most important pre-requisite to make your life easier is to allow your databases and services to be configurable and not hard coded. I mentioned this earlier, but it's even more important if you want to re-use your code and scale in the cloud while also keeping your footprint small in a self-hosted environment.

So how we do it with Bugflow and Docker is straight forward. All the configuration for our services and databases is held in the environment variables that are required for the app to run. The actual server configuration is set up in our docker-compose.yml file. This means, when we set our queues to run on Laravel Horizon, we can configure the service to run on Horizon container in production. The Horizon container comes on line when we bring up our app.

In the self-hosted version, we just have a database configuration file. We set our queues to run in the database. The combination of what services get deployed + how we access them are all managed through the environment variables and what gets brought up through Docker!

Need A Hand?

If you have questions about how to architect your application to make it easy to self-host or want us to do it for you, feel free to reach out. We've done the same approach for multiple clients that we've worked with and are well-versed in the process. Hop on Discord and we'd love to lend a hand!