Have you ever read or heard about news where organisations got hacked due to hardcoded data in the code? Following are a few examples -
Other more common problems that developers and organizations frequently face are -
Applications become unmanageable after a point in time. They are famously called legacy code, and then plans are created and executed to move away from that legacy code.
Code written quickly (quite common in startups) is unstructured and needs refactoring to perform better at scale.
Differences between various environments, like Dev, UAT, Production, etc., cause the business logic to break.
CI/CD works at different speeds in different environments.
The application layer breaks when the database or network configuration changes.
But what if I told you that most such problems are avoidable? All you have to do is strictly adhere to the Twelve-factor app methodology in the modern context.
What is the Twelve-Factor app methodology?
The twelve-factor app is a set of principles and practices to design and build scalable application architectures. The co-founder of Heroku first published it in 2011. After observing hundreds of thousands of apps via their work on the Heroku platform, he created these principles to raise awareness about some systemic problems in modern application development. The twelve-factor methodology can be applied to apps written in any programming language and which use any combination of backing services (database, queues, memory cache, etc.). The original methodology can be found on this dedicated website - https://12factor.net/
Are these factors still relevant?
I have found people debating the relevance of Twelve-Factor apps in the modern world. But let me tell you where the disconnect is. The principles written on the original website are specific to Heroku's PaaS platform. Still, they helped a generation of application architects like me to develop their mental models for approaching this problem of building applications for scale. You should go through this blog post even if you know the methodology. It's up-to-date with the most recent tools, technologies and techniques to implement the twelve factors, and I intend to keep updating this post whenever required.
Factor number one - Codebase
This "Codebase" principle states that -
Each application must have its own single codebase.
This codebase should be managed using a version control system like Git (or others)
If there are multiple codebases, then it's not an application. It's a distributed system.
Every application version, like - development, staging, production, and other versions, should be in the same codebase.
Multiple apps sharing the same codebase (repository) violate this principle. If you want to share a particular piece of code across different apps, then you should use ** factor number two - Dependencies (Next one)**
How are people implementing this factor in modern apps?
If you are a developer reading this in 2023 or later, you must know how this factor is implemented. It was not as obvious when this principle was first written. But today, managing multiple services modularly, pushing code in a Git repository, creating multiple branches, maintaining clean branches, merging them, reverting changes to a particular commit, etc., has become an everyday activity. The biggest takeaway that people still miss is the codebase should be the same across different environments. Infact, a lot of the twelve factors focus on this. You will see why and how later in this blog.
Factor number two - Dependencies
The principle of dependencies says -
Only the code uniquely required for your application should be stored in the source control.
All the dependencies and external artefacts should be declared separately using a manifest file (like Gradle for Java) or a package manager file (like package.json in JavaScript projects, go mod file in golang, Rubygems for Ruby, etc.)
These dependencies should not be in your main source code repository at any cost. 4.They should be loaded at runtime using some "install command" while running this app in development, testing, or production.
You must have already guessed it. Most modern programming languages and frameworks come with package management tools. They also come with a set of best practices while using dependency management. Some of the most famous examples are -
Javascript - npm, yarn, pnpm
The go-module ecosystem of go programming language
Python - Pip
Ruby - RubyGems
PHP - composer
Rust - Cargo
How are people implementing this factor in modern apps?
Dependency management used to be a big problem ten years ago. It's not now. The open-source ecosystem has matured to a level where you can find a package (or library - use the terminology suitable for your language) for everything you want to develop. You can create your own if you do not find the package you want. If your package has a use case of a particular functionality missing, you can code that and raise a pull request. Some organizations also maintain private packages to achieve this functionality. If you are a developer working with a relatively new language or framework and come across a use case where a package is unavailable, please take the time out and program that package. Such open-source projects and contributions will significantly boost your knowledge, help you build an excellent network, and help you get shortlisted for good jobs.
Factor number three - Config (or configuration)
This config principle states that -
You should not hardcode any configuration in your codebase. These configurations can be database credentials, third-party service credentials, networking-related credentials (like IP address, ports, hostnames), etc. Anything considered highly sensitive or frequently changing information should not be part of your code repository.
Such information should be injected into the runtime environment using "Environment variables" or other ways (discussed later), setting them as independent configuration files / storing them in secret management tools like AWS or Google Cloud Secrets Manager, Hashicorp vault, etc.
Configurations and code should be strictly separate from each other.
How are people implementing this factor in modern apps?
For simple environment variables, while working in local or development environments, people use a ".env" file supported by most modern programming languages.
People pass the environment variables in their docker files for non-sensitive configurations like application port, host, etc. Usually, these files are not pushed with your application code. They are added to files like .gitigore file.
While running these containerized environments in container orchestration tools like Kubernetes, people use the configuration options offered natively in Kubernetes, i.e. config maps and secrets.
For storing highly sensitive information like passwords, access keys, SSH keys, encryption keys, etc., people use secret management tools like AWS Secret Manager, Google cloud secret manager, Hashicorp Vault, Azure key vaults, etc.
Configuration management is a big industry on its own. Hundreds of big companies are trying to solve this problem for various general and specific use cases. My recommendation for choosing your solution is simple. Take the following factors into account, in order of priority -
Priority 1 - Evaluate your use case. Evaluate if it's a simple use case or if you need a very specific tool. If your use case is specific, then you compare the offerings and shortlist the ones that solve your problem best.
Priority 2 - The cost of the solution should be the second priority. It can increase fast as you scale your systems.
Priority 3 - Ease of use with your application layer - If the solution fits the above two criteria but is very difficult to integrate with your application layer, it will become useless/inconvenient fast. So, ensure that your tool fits your application and infrastructure well.
Factor number four - Backing services
The "Backing services" principle advocates that -
Any service an application consumes over the network as part of its regular operation, like databases, message brokers, external third-party API, etc., is called a backing service.
You should treat these external components like an attached resource or a different part of your system that should be changeable at any point without making any changes to the application code. For example, if you are developing an application that uses any database and your database is currently supported by your local environment only (probably installed in your system and accessible over localhost). Your code should be written so that this local database endpoint (and credentials) can be replaced with a hosted database service or solution. You should be able to do that by simply changing the credentials and endpoints in your configuration. Treating these services as backing services enables developers to write highly portable code that can run anywhere without modifying the application code.
How are people implementing this factor in modern apps?
As you must have guessed, people use their configuration management (the previous principle) setup to achieve this. Typically, connecting to these external services requires one or more of the following things -
Hostname or endpoint
Port
Passwords/keys etc
Configuration settings (like database pool size)
Since all of them are configurations, you can use one of the possible configuration management setups discussed in the previous module. I have noticed that this is the factor that is ignored the most by developers. The reason is simple. In online courses or programming boot camps, We are taught to run these backing services on our local machine and integrate them directly into the code. Most of the SDKs and package tutorials also follow a similar approach. But I recommend not to do that. Take this factor seriously, and your code will be truly portable. You can move it around quickly without any challenges and with predictable behaviour in different environments.
Factor number five - Build, release, run.
This principle advocates that -
The deployment of your application must be properly separated into three non-overlapping and non-dependent phases called - Build, Release and Run.
The build stage should be about transforming the raw application code into an executable bundle or other such artefacts (Explained later)
The release stage should be about getting the package from the build stage, combining it with the configurations of the environment (development, staging, production, etc.) you are trying to deploy in, and making the application ready for running.
The run stage should be about running your application in the environment of your choice.
This process enables a clear separation of concerns and should ideally be automated.
How are people implementing this factor in modern apps?
People use CI/CD tools to automate the build and deployment process. CI/CD is a vast topic. Many open-source and managed tools are available for setting up CI/CD. If you are a beginner and have no idea about how to start with CI/CD, I would recommend starting with mature tools like -
GitHub actions - https://github.com/features/actions
Gitlab - https://about.gitlab.com/
Bitbucket pipelines - https://bitbucket.org/product/features/pipelines
They will help you start with CI/CD quickly and have pricing models suitable for beginners. From there, you will get a fair idea about how to start with this and get enough resources and support to learn CI/CD quickly.
Factor number six - Processes
This principle advocates that -
Applications following the twelve-factor app principles should be run as a collection of stateless processes.
It means that no single process keeps track of the state of another process, and no process keeps track of information such as session or workflow status.
When the process is stateless, application instances can be added and removed to address the increase or decrease in load (traffic) at any given time.
It enables you to scale your application horizontally without worrying about side effects.
How are people implementing this factor in modern apps?
Horizontal scaling is widespread nowadays, thanks to auto-scaling functionalities offered by cloud providers and container orchestration systems. Today, Most applications are programmed to follow "stateless principles" to keep things simple. If you still want to maintain the state of your application, you can use cache databases like Redis or Managed data stores on the cloud. They help you write and fetch application state information at a very high speed without worrying about breaking your stateless processes principle. This principle is also a prerequisite for other Twelve-Factor guidelines, such as concurrency and disposability The horizontal scaling approach offers many operational benefits, making scaling and recovering from failures easy.
Factor number seven - Port Binding
This principle states that -
A service or application should be identifiable to the network by port number, not a domain name. It should be self-contained and standalone and export the functionality it's supposed to perform (like an HTTP REST API) as a service by binding to a port and listening to requests on the port.
The reason is that the domain names and the associated IP addresses can be changed on the fly for various reasons. So, using them as a point of reference is unreliable.
In your local development environment, you must have worked with endpoints like
These are port bindings, and you must listen to these particular ports in your application. This application can be anything. It can be a React or Angular app, a REST API application, a GRPC application, etc. It's not limited to just services written by developers. It's also true for other backing services like databases, message brokers, etc.
How are people using this factor in modern apps?
As a developer, you must have noticed some patterns of port usage that are almost conventional now. For example -
Using port 80 for HTTP web servers
Using port 443 for HTTPS
Using port 22 for SSH
Using port 3306 for MySQL
Using port 27017 for MongoDB
The essential idea behind the principle of port binding is that the uniform use of a port number is the best way to expose any process to the network. In production use cases, we hide these port mapping behind a reverse proxy such as Nginx, Apache web server, Tomcat, HAProxy, cloud load balancers, etc. However, having a port mapping to identify these services individually helps configure these networking mapping from reverse proxies to actual services relatively easily. It also enables these services to become backing services for other apps.
Factor number eight - Concurrency
This principle states that -
You should organize the processes according to their purpose and then separate those processes so that they can be scaled up and down according to the need.
You should run the application as multiple instances instead of running one large system.
Each of these processes should be able to start, terminate and replicate itself independently and at any time. 4.You can still opt-in for threads to improve the concurrent handling of the requests.
How are people using this factor in modern apps?
One word: containerization. Before containers, running processes like these meant copying the entire Virtual Machine Image to another VM (server) and running the application in that. It worked, but it was not optimal. With containers, you don't have to worry about this. Scaling up and down is fast and can be configured using various conditions. The typical ways to achieve this are -
Scaling out containers and letting the load balancers (or service mesh technology) figure out the traffic distribution.
For applications running on Kubernetes or similar orchestration tools - it means creating multiple pods manually or automatically using the built-in horizontal scaling.
The twelve-factor methodology promotes horizontal scaling over vertical scaling.
Factor number nine - Disposability
This principle asserts that -
The processes running in a distributed manner with multiple instances are disposable and can be started or stopped at a moment's notice.
The applications should start and stop gracefully.
These stops and starts should not affect the application's functionality and performance.
How are people using this factor in modern apps?
If you follow the principle of stateless applications, this becomes easy. In production environments, people do this via the combination of the following two steps -
Step 1 - Gracefully shut down all the connections like databases, network resources, or any other backing service as soon as the application receives the command to terminate. This termination command could be application generated or may be due to heavy load (High CPU usage, High memory usage, High throughput of data transfer). But irrespective of the use case, to the best of its availability, the application code appropriately disposes of all these resources and should also log why this shutdown occurred. Almost all modern programming languages have an option to handle this termination state.
Step 2 - At the infrastructure level, allow the application instance to perform the abovementioned functionality after receiving the shutdown command. i.e. your infrastructure setup mustn't kill the container immediately after the application has been terminated. It should provide extra time to clean up resources (as mentioned in Step 1) and terminate the container. Most of the modern cloud-native deployment strategies and tools allow for this. For example, in Kubernetes, you can set properties like Termination Grace period, etc., to enable the above functionality to take place at the application level. Different deployment environments give their feature to support graceful termination.
Factor number ten - Dev/Prod parity
This principle advocates that -
You must keep your development and production environments as similar as possible.
It will ensure that the risk of bugs or performance issues in a specific environment is significantly less.
It also helps in the CI/CD process. (Factor number five - Build, release and run)
Developers writing the code should be a part of deploying and monitoring the application in production. It ensures that the personnel gap is small and they have a fair idea about problems due to different environments.
How are people implementing this factor in modern apps?
The fourth point, i.e. making developers responsible for Ops, like development and monitoring, is not a common practice. Different companies have different opinions on this, but most of the companies agree that the developers should at least get the data regarding the performance and monitoring of the application in production to have a fair idea about how their code is performing. Having a separate SRE or DevOps team is a luxury that most startups cannot afford; hence, both development and deployment are done mainly by the same person. As far as the remaining rules are concerned, companies take the following steps to ensure that this parity is least -
Take CI/CD seriously - As we learned in factor number five - Build, release, run, a promising CI/CD pipeline can take away a lot of this manual work from developers and automate it. That's what most companies try to do today. For most people, developers' time is costly, and they want developers to be able to handle environment setup and build the application code. So, they either use an open-source or managed CI/CD solution to solve this problem.
Have a "Dev environment" setup for backing services - For all the backing services like databases, Queues, etc., companies tend to maintain a dev copy to ensure that developers don't have to spin up these backing services in their local system. These Dev backing services also have similar datasets with sizes mimicking those of production to ensure that code performance issues are realized during development only.
Many organizations (especially startups) consider development environments an additional cost and only bothers a little about it. They encourage engineering managers or leaders to save as much as possible on technology costs. Usually, the "Dev environment setup" is the part that witnesses cutdowns because of these "cost-saving measures". It's okay if you build a minimum viable production or an experimental product used by a few users. But, if you are a company with a loyal customer or user base and want to retain them, development environments and backing services are a tiny price. They help you identify the problems in your applications/infrastructure/backing-services setup and help you get ahead of the issue before it becomes a problem for your users. So, stop bragging about how you averted a production disaster in just X minutes and think about how I can achieve a production setup with zero downtime. And the first step in that direction is the factor discussed above.
Factor number eleven - Logs
This principle advocates that -
Logs should be written to standard output and treated as an event stream.
The application shouldn't care about where the logs are ultimately stored.
The application shouldn't be responsible for storing the logs in memory or any secondary storage (like files or databases) because this will impact the application's CPU and memory usage and ultimately hamper the functionality the application was initially supposed to perform. Also, such an application will require the logging to be programmed carefully with tight integration into the logging system. Such an application cannot be considered stateless. This is why decoupling the log tracking and storage system from your application's main business logic is essential. Also, storing logs in persistent storage separate from your application's runtime is highly recommended to ensure that the logs are not lost when the application crashes or the infrastructure goes down.
How are people implementing this factor in modern apps?
Logging is a massive industry today. Several open-source and managed solutions are available for logging data from various types of applications. Each cloud provider also has its built-in solution for log viewing and analysis. The solutions primarily try to differentiate themselves based on -
Features available
Pricing
Maturity of SDKs or APIs available (for special programming languages)
Analytics ability
UI/UX and Visualization options available
Developer experience
Storage and query pricing
If you are a developer or ops person, my guess is you already have a preferred logging tool or stack. And you will be surprised to know that I don't have one. I like to choose a tool according to the use case, possible scale, budget, and ease of use. Indeed, it comes at the cost of doing a little POC occasionally, but it's usually worth it.
Factor number twelve - Admin processes
According to this principle -
One-off processes like database migration, environment preparation, and periodic cleanups and updates are integral to application development and deployment. These tasks are not required frequently, so developers write them as simple script files and keep them as separate files in their system.
This factor advocates that such scripts should be a part of the codebase and be treated as a twelve-factor app. They should be maintained in a version control system and follow all the remaining factors of a twelve-factor app.
It also advocates REPL shell access to production to inspect script performance against the live database - But this practice is now archaic and is not followed in the industry due to security reasons.
How are people implementing this factor in modern apps?
It is one of the twelve factors that became obsolete with time. Instead of including scripts in the main application code, the way this is usually implemented is as follows -
Create a separate service containing the script code.
Treat this service like a separate twelve-factor app and follow all the eleven factors discussed in earlier modules.
Run this service like a scheduled task, cron job, or manually triggered job.
Development and deployment are done using the usual CI/CD process.
Admin processes are crucial in any environment. Maintaining that locally is a considerable security risk. I recommend never doing that. As mentioned, you should treat these processes or scripts as a different service and follow all the practices (including security) while writing these scripts. It will undoubtedly increase the effort required to write, run and maintain such scripts, but it will come with the most significant added benefit you can get as an Ops/developer - "Peace of mind".
Twelve factors are a preemptive measure
Think of the above twelve factors as a preemptive measure to ensure that your applications are -
Easy to scale
Easily Manageable and portable(in containerized environments)
Secure
Easy to collaborate.
There are more such techniques and principles that can help you as well. But they are topics for another blog post.
Till then, share this article with your friends and colleagues. If you have any questions, feel free to mention them in the comments. You can also ping me directly on Twitter, Linkedin or Instagram