The evolution of application software development over the last 15 years has given rise to new methodologies in the cloud computing space that were rarely seen when development was purely an in-house activity.
New ways to track versions and deploy software updates using automation and code repositories created the opportunity for developers to pass on the task of testing and deployment to team members who more often than not were responsible for the day to day operations of the network infrastructure supporting the applications being developed.
DevOps is the term used to describe engineers who fulfil the role of taking newly developed code, passing it through test environments and continuously deploying that code into production. Back in the day, new features and capabilities would be developed and packaged into new major releases and delivered to end users as a major upgrade all in one go. The advantage of DevOps driving a continuous integration / continuous deployment (CI/CD) Pipeline is that improvements, new features and fixes reach the end user as they are developed, delivering a much better user experience than having to wait several months or years for the next major release.
In this post we’ll take a look at the typical tasks and responsibilities you would need to take on as a DevOps engineer and the tools and skills you should learn in order to take on a DevOps role.
All applications that are in a production state are normally deployed via two or three segments.
Development - this is where the developers create the application and features and typically place code into a repository so that multiple developers can work on it at the same time and version control can be applied.
Testing - where the code is manually reviewed or subjected to automated testing procedures to ensure that it is working as expected and had not broken anything already in production.
Operations - This is where the application is deployed, dependent resources are provisioned like load balancers, firewalls and database instances. This either happens on servers either in-houe, in data centers or on cloud infrastructure provided by IaaS vendors like Amazon Web Services, Google Cloud Platform and Microsoft Azure.
DevOps is the link between Development and Operations. Developers will often retain the testing responsibility, or simply hand over that task by dropping code into a code repository like Github and letting DevOps take it from there.
Developers can use a variety of languages and frameworks to develop code and will typically store that code in a repository like Git. You as a DevOps engineer won’t be involved in the development process, but you will need to understand how the developers work, what workflows are in use such as specific Git Branches and how the application is configured so that you know what resources are required for the application to perform at it’s best.
In terms of testing, if the testing regime is to be handled by DevOps then you will need to know how to run and understand the tools used in automated testing.
One of the roles a DevOps engineer will undertake is preparing the infrastructure required to host the application and supporting resources. One of the most popular server and cloud machine image platforms is Linux, so as a DevOps, you should have a basic understanding of:
- Linux Basics
- Command Line Interfaces (CLI)
- Shell Commands
- Linux File System
- Basic Server Administration
- SSH
- Security Key Management
While enterprises will normally have security specialists or a security team to manage all the security around your organization’s infrastructure or physical hardware, as a DevOps engineer you will need to know some security basics like:
- Setting up firewalls
- Setting up proxy servers and access
- Setting up load balancers
- HTTP vs HTTPS
- Understand IP address setup
- Understand DNS resolution
- Understand how to open ports to enable users to safely access the application
With all of the above operation and security tasks, A DevOps engineer is expected to know the basics and not be an expert in ops or security as these roles are generally separate and the responsibility of other professionals like security engineers or network admins and systems administrators who specialise in each discipline.
You should know enough to be able to set up the servers or cloud infrastructure securely so the application can run, without taking on the responsibility of server management or security.
Containers and DevOps
These days using containers as the preferred software packaging model has gained a lot of momentum, so as a DevOps engineer you will need to be across the tools and methodology to deploy containerised software applications. In the past 8 years since the introduction of Docker as many as 50% of companies have reported using containers in some form.
So you will almost certainly need to understand software virtualisation methodology, containers and how to deploy containerized applications. Docker is still the most popular software packaging utility and something every DevOps engineer will need to learn at some point.
What do DevOps Engineers do?
By far the most significant proportion of a DevOps engineer’s day to day activities revolve around deploying new code, features and bug fixes released by the developers onto the server infrastructure so they are available to the end users.
The DevOps skill set centres on how to take software updates and deploy them continuously as they become available (CI/CD) using as much automation as possible.
Once updates are made available in the code repository, DevOps steps in to run the testing and package up the updated application into JAR or ZIP files (artifacts) using build tools and package managers like Apache Maven, ANT, Gradle, GNU Make or, so you will need a solid understanding of how these build tools work.
If the package is containerized using something like Docker, the next step would be to push the artifact to an artifact repository, so as a DevOps you should become familiar with the popular repos like Nexus or DockerHub.
So while all of this can be done manually you should know how to automate this process in sequential steps using an automation CI/CD deployment server and build automation tools like Jenkins. Being able to configure complete CI/CD pipelines is the core responsibility of a DevOps engineer.
DevOps and the Cloud
Most organisations now have part or all of their server infrastructure hosted in the cloud. The main Infrastructure as a service (IaaS) vendors based on market share are Amazon Web Services (AWS), Google Cloud Platform (GCP) and Microsoft Azure.
As a DevOps engineer you would be well advised to learn one or more of these cloud platforms. As companies move away from on premise server technology to cloud solutions from one or even multiple cloud vendors, you will need to know your way around the cloud resources and consoles and know how to leverage the many managed services like load balancing, backups, clustering and security and so forth that are a major benefit of moving from on-premise hardware tp cloud based solutions.
A lot of services and resources that you will need to provision can be easily created via the various platform consoles and CLI interfaces, even though they are typically platform specific and each have their own strengths and nuances. Obviously you would concentrate on the platform that is hosting the applications you will be supporting and you only need to know the services required to support the applications.
There are hundreds on services on AWS for instance and while become an expert on deploying machine learning or data warehousing would be a great skill to have, it isn’t necessary from a DevOps perspective if the applications your enterprise produces doesn’t need ML or Data Warehouses.
DevOps and Container Orchestration
If you are using containers to package and deploy applications, then you most likely are building Docker images. Once deployed these containers still need to be managed and maintained.
If your application only has a handful of containers then services like Docker Swarm or Docker Compose will be all you need to learn in order to manage them.
If your application has lots of containers or micro-services then you will need to utilise a container cluster orchestration service. One of the most popular to date is Kubernetes. As a DevOps engineer you will need to get to grips with using Kubernetes and learn how to both administer a Kubernetes cluster and deploy applications within it.
It is entirely possible for Kubernetes clusters to scale to thousands of containers across multiple server instances and so one of your tasks may be to monitor the run state of your containers and track the performance to discover potential problems before your application users do.
Getting familiar with monitoring tools that keep an eye on your application, clusters and underlying server infrastructure is something that your DevOps skill set will benefit from. Some popular options include Nagios, Prometheus, Datadog, Dynatrace and Solarwinds to name a few.
DevOps and Infrastructure and Code
In most application development environments there is a need for multiple sets of infrastructure. The main three that spring to mind are Development, Testing/Staging and Production. It will usually fall to the DevOps engineer to replicate environments.
Creating environments manually can be very time consuming and prone to error. It’s easy to miss a resource setting or vital component that will ultimately cause problems in testing or worse still production if it isn’t picked up during testing.
So creating all the necessary resources and infrastructure manually 3 times isn’t ideal, which is where another major DevOps task - Infrastructure as Code comes into play.
Automating the creation of infrastructure and setting configuration files to run and deploy apps are two skill sets expected of most DevOps engineers.
A popular infrastructure provisioning tool is Terraform, while configuration management tools like Ansible, Puppet and Chef are often used. As a DevOps, it would be wise to learn services like these to assist with the programmatic deployment of infrastructure and applications.
Once you have mastered Terraform and one of the config management tools, you will become more efficient when deploying, your configurations will be transparent for anyone reading the deployment code.
When you have automated deployment code in place, it makes it easy to replicate infrastructure into another region or availability zone, should you need to recover fast from an outage or if you need to replicate your environment somewhere else for testing or analysis.
DevOps Scripts and Programming
When you work closely with developers and system administrators, the need to write scripts or small applications to automate tasks around infrastructure and app deployment will inevitably crop up. Tasks like backups, system monitoring, network tasks or cron jobs to automate job scheduling and clean up are often delegated to DevOps.
So to do this, you’ll need to learn a scripting language or two. These might be operating system specific, like Bash on Linux or PowerShell on Windows which are essential skills any DevOps engineer should have.
Other platform independent scripting languages are also a good skill to master and there is always strong demand for anyone with Python, Ruby or GoLang knowledge that can be utilised in a DevOps scenario. Of the three, Python is probably the most commonly used with strong demand.
DevOps and Version Control
Just like developers need to keep a tight control over the versions of application software they write, the code and scripts you use to provision infrastructure and deploy apps using the tools discussed like Terraform, Ansible and even python scripts should have equally rigorous version control.
To do this it is a good idea to follow the same principles as the development team and utilise Git repositories like Github or GitLab to manage your IaaS code, so learning Git is definitely something a professional DevOps engineer should do.
What tools should I learn to be a DevOps engineer.
We’ve mentioned a lot of tools in this post and several distinct areas that a DevOps would be responsible for, however you shouldn’t feel like you need to be proficient in every single tool up front in order to be an effective DevOps engineer.
Learning one tool or service from each of the areas will get you off to a good start and once you are familiar with how the tools work and what they do, it will be easier to pick up alternative options as your career progresses. A good strategy when starting out would be to pick the most popular in each category.
This can be a matter of interpretation, but starting with the following skillset won’t hinder your ambitions of becoming a DevOps engineer.
| Operating System (Basics) | Linux |
| Cloud Platform | AWS |
| Scripting Language | Python |
| CI/CD Automation | Jenkins |
| Containers | Docker |
| Container Orchestration | Kubernetes |
| Monitoring | Prometheus |
| Infrastructure as Code | Terraform / Ansible |
| Version Control | Git |
So there you have it, a quick run through the basics of what it will take to enter the exciting world of DevOps. There are literally hundreds of free and paid courses available on DevOps specifically and also on all the tools and services mentioned in this post.
When you do start to build infrastructure and deploy applications on AWS, Azure or GCP you will soon discover how invaluable good network documentation is, especially network topology diagrams. You will also appreciate the core premise of DevOps is to automate early and often, which is why we believe no matter how far along you are on your DevOps journey, you will appreciate the diagrams and security documentation produced by hava.io
By connecting your AWS, Azure and/or GCP accounts to Hava, you will always have accurate up to date diagrams of your cloud infrastructure on hand to validate your DevOps automation has produced the desired results.
You can learn more about Hava for here:
