Building a digital product in AWS
Part 1 - essential concepts
• Jonas Svalin
So, you've decided to build a digital product. Maybe it's something fairly simple, such as a website for recipes. Maybe it's something highly complex, such as a bank. Depending on where your product falls on this scale, there are two primary options for building your digital offering:
- Using an off-the-shelf solution which allows you to compose simple components
without requiring deep technical expertise
(WordPress, SquareSpace, Hubspot, BigCommerce etc)
Pros: Fast, cheap, simple - Building it yourself from scratch
Pros: Fully customisable, more ability to scale your offering, sometimes the only plausible path if your product is sufficiently complex
Naturally there's a whole slew of products that fall somewhere in between which, to varying degrees, can take on the responsibility of certain aspects of your product. Examples of this include using Stripe for managing your e-commerce payments, Onfido for doing identity checks, or outsourcing your bank ledger to Mambu. For the purpose of this article series, we consider using such products as still falling under option 2, as they typically mean that you're still creating and hosting proprietary software which composes and integrates these capabilities to create a complete digital offering.
The goal of this article series is to help you on your journey of building a digital product from scratch. At Atomic, we've spent the last decade working with various partners to realise their digital product visions. In the process we've established a set of reusable open-source components which allow you to create your product faster, more easily, and more cheaply. We will do our best to explain the essential concepts, as well as guide you in using the components we've created under the InfraBlocks banner. If you've been working as a software engineer (or in a related role) for a meaningful amount of time these terms may be completely obvious to you already, in which case you can hop directly to Part 2 of the series.
Amazon Web Services
If you write proprietary software, such as Java microservices or a JavaScript web app, you need a central place where you conduct the activities associated with creating, running and hosting your software. This includes things such as building and hosting docker images, running software containers, managing hosted databases, creating DNS entries, setting up load balancers and much more. Amazon Web Services (AWS) is a popular cloud provider which allows you to do all of these. AWS is the provider we will be using in this article series as well as the provider that the majority of the components we've built are targeted towards.
Terraform
Running software infrastructure with a cloud provider such as AWS is not a simple activity. AWS provides a graphical interface for conducting all activities, but in most cases this is not the preferable approach. Manually creating and updating software infrastructure is highly error prone and difficult to make reproducible. For this reason, most modern organisations will rely on infrastructure-as-code instead, a concept in which you template your infrastructure in dedicated infrastructure files and check them into a version control system such as Git alongside all your other software.
Terraform is arguably the most popular infrastructure-as-code platform out there today, and is the foundation of all the components we've built. Terraform allows you to build your infrastructure using HashiCorp's proprietary configuration language, HCL. Terraform automatically transforms your infrastructure files into the actual physical components that you require, without needing any manual provisioning work. This has a number of critical benefits. Amongst these are that it's simple to make changes, it's easy to keep track of changes, and it's straight-forward to share common patterns through reusable components such as the ones we've created in InfraBlocks.
Deployment environments
If you've previously worked in a collaborative software development project you'
ve likely heard colleagues refer to different environments. When we talk about
environments in the context of software we mean the same/similar software
running in separate, isolated systems. The most important environment is
typically referred to as the production environment, which is the one your
end-users are actually connecting to. Making untested changes directly in
the production
environment is dangerous because the results can be unpredictable. For this
reason, most organisations will have at least one additional environment, often
referred to as the development
environment. This environment functions more or less exactly the same as the
production environment, but is completely isolated from it, such that you can
make changes (such as deploying a new version of a software component, updating
a database or testing a new integration) without risking degrading the
experience of your end-users. When the change is thoroughly tested and you have
a high degree of confidence that it works as expected, you can promote the
change to your production environment.
At some point many organisations will outgrow their initial 2 environments, and may add arbitrarily many environments to allow for more isolated testing and validation of specific functionality. Some even go so far as creating and tearing down new environments on an ad-hoc basis. Using Terraform makes it very simple to spin up multiple environments and promote changes between them, since Terraform defines configuration in a way that makes it precisely repeatable between environments. This is one of the many benefits of relying on infrastructure-as-code as the foundation for all your work.
Continuous integration / delivery / deployment
The family of Continuous X terms can sometimes be hard to keep track of.
Nonetheless, it's essential to understand the basics since we'll assume this is
how you deliver software. For the purpose of this series, the critical points
are that we assume some type of CI tool is used to build and deploy software
changes. A CI tool allows you to create build pipelines, in which you define
automated steps to perform the discrete activities required to release software.
This typically includes (but is not necessarily limited to):
- Build (building a deployable version of your software)
- Test (running an automated test suite on your software)
- Deploy to development (deploying your new software to the development environment)
- Deploy to production (deploying your new software to the production environment)
Most of the time the first three steps are performed automatically whenever any change is pushed to Git (or whichever version control system you use). The final step of promoting the change to production is often a manual step which is performed after sufficient manual testing has been done in the development environment. Naturally, we apply these principles for the components we build under InfraBlocks as well.
- Wikipedia → continuous integration
- Wikipedia → continuous delivery
- Wikipedia → continuous deployment
Rake
The InfraBlocks Terraform components are perfectly valid to use in any Terraform based environment, but InfraBlocks as a platform leverages Rake as the build utility of choice. Rake allows you to write command line tasks using standard Ruby syntax. We find this to generally be preferable over simple bash scripts because it is more composable and testable. We have created a number of Rake based tools as part of the InfraBlocks platform which simplify many standard infrastructure tasks, including interacting with Terraform. In upcoming parts of this series we will be leveraging these tools as we build an example system.
Conclusion
In this article we covered some basic concepts which the reader is required to grasp before we move on to actually building something concrete. In the upcoming parts of this series we will create a digital product together, using the open-source components of InfraBlocks.