Making a Positive Difference in Students Lives
On the ASR App Dev team we use Docker extensively in our development process. Our standard project setup is to have a Dockerfile and docker-compose.yml file defining the base image that we are using for the application while in development, and then use that image for running all of our local tests. This has removed a huge amount of setup work that we used to need to do whenever we picked up a new project. In the past it wasn’t uncommon to spend the first week of a sprint on a new project just setting up the development environment, getting Ruby installed, fixing all of the weird conflicts that would come up with OpenSSL versions, etc. In addition, we’ve leveraged this docker configuration to use Drone for automated testing.
However, with the new M1 Macs this process has hit a snag. Some of us are now using the new M1s, which use the arm64 architecture, while others are still on previous models of Macs, using the x84 architecture. Many of our images were built to only work on the x86 architecture, and just don’t work on the new M1s. We can’t just switch over the images to only using the arm64 architecture, because that would cause the same problem we currently have for the folks on the x86 architecture. But we want both architectures to use the same Dockerfile and docker-compose.yml, otherwise we will get into a situation where we could have different setups for different people, which leads to divergent development envrionments - which leads to “but it works on my computer” problems.
Fortunately, Docker has a way to handle this - multi-platform images. We can build a single image for both the x86 and arm64 architectures.
To do this, we made the following changes:
To create a build environment, we needed to run the following command to create a buildx environment and mark it as the default one to use:
docker buildx create --use --name multiarch
Modifying our Dockerfile was mostly easy. If there was no change in how the two architectures were built, then nothing needed to be done. We had one thing which was different between the two: a set of Oracle InstantClient RPM files. To handle this, we added the following to our Dockerfile:
ARG TARGETARCH
ARG ORACLE_VERSION=19.10
COPY ./rpm/$ORACLE_VERSION/*.$TARGETARCH.rpm /home/oracle/
RUN /usr/bin/alien -i /home/oracle/oracle-instantclient$ORACLE_VERSION-basic-*.$TARGETARCH.rpm
RUN /usr/bin/alien -i /home/oracle/oracle-instantclient$ORACLE_VERSION-devel-*.$TARGETARCH.rpm
RUN /usr/bin/alien -i /home/oracle/oracle-instantclient$ORACLE_VERSION-sqlplus-*.$TARGETARCH.rpm
TARGETARCH is set to the architecture that you are building. For us, is either amd64 or arm64. We needed to do some shenanigans to get that to work with the oracle packages, because the oracle packages are actually named aarch64 and x86_64. To make that work for us, we just created simlinks from the Docker architecture names pointing to the original oracle files.
Finally, we needed to change our build command to actually build the new new architecture images. The command we used is:
docker buildx build --platform linux/amd64,linux/arm64 --push -t asr-docker-local.artifactory.umn.edu/image_name:tag .
This kicks off two different build processes that you can watch on your terminal, as it builds both the amd64 and arm64 images, then pushes them both up to our artifactory repository. This takes a while, but when we are done we have a single image that developers on either platform can use.
I hope you find this useful!
At ASR Custom Solutions, we support a number of applications and systems. These run in multiple environments and hosts, talking to other services that also exist in different environments. At any point, there’s dozens of passwords, secrets, connection strings and other configurations we have to manage.
Our team stores these in LastPass. When we need to access them though, we had to deal with hunting within LastPass for the specific note we needed. For a large, microservicey-system like our Student Degree Progress Service, this could mean dozens of hostnames, usernames and passwords. Onboarding a new developer to the project was distinctly painful here.
So, we made a Ruby gem to automate all this for us; Lastpassify.
Lastpassify is a commandline tool packaged as a Ruby gem that takes in an YAML ERB template file and outputs a populated YAML file. We primarily use it for our database.yml files. Now, what might’ve taken 20 minutes or more to fully populate a large config file by hand is done in milliseconds.
Besides efficiency improvements, using the gem has given security benefits as well. For example, having trust in the tool means that we’re able to delete our credentials and repos when we are not working on the project. Not having the credentials on the developer machine adds to the security protecting the data behind our applications.
Lastpassify requires lastpass-cli and Ruby v2+ to be installed.
LastPassify expects an input of one YAML file to be processed and outputs one YAML file. The input file can be passed in as an argument at the commandline like so:
$ bundle exec lastpassify my_input_file.yml
The output file and path can also be specified:
$ bundle exec lastpassify my_input_file.yml config/my_output_file.yml
LastPassify has default values that silently get passed if no input or output file is specified. The default input file LastPassify will look for is config/database.example.yml. The default output will also live in the config directory, with a filename of database.yml.
An example database.example.yml file might look like this:
---
# Shared
global_defaults: &global_defaults
adapter: oracle_enhanced
# Development environment
development: &development
<<: *global_defaults
host: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='Hostname') %>
database: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='Database') %>
username: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='username') %>
password: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='password') %>
secret_key: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify_secret_key', field='Secret Key') %>
staging:
<<: *development
Finally, Lastpassify strips out any YAML keys with production, staging or qat in their names. This is a security measure to ensure no prod or staging environment credentials sit on your local development machine unnecessarily.
This can be overridden by passing in a -s (staging) or -p (prod) flag to Lastpassify, e.g.: $ bundle exec lastpassify -p
We hope you find it useful and appreciate any pull requests!
At ASR, we have a large number of SQL queries, functions and stored procedures for retrieving and manipulating data. One value we stand by for the software we maintain is test-driven development. Generally speaking, I feel we do a pretty good job of this. However, there is one area that had glaring holes: our large corpus of SQL objects (functions and stored procedures).
Enter utPLSQL, a unit-testing framework for Oracle PL/SQL. If you’ve ever worked with RSpec, it will look very familiar.
This will be a two part series. In this post, we’ll go over a sample SQL function that we use at ASR. In the next, we’ll go over how to use utPLSQL to write unit tests for it and nice features of utPLSQL you might find handy like reporting.
At the U, we have three terms per academic year; Spring, Summer and Fall. We store these in our databases in a scheme known as “STRM”. This consists of a four-digit number, the first three of which hold the year value and the last digit representing the semester. The year value is calculated by adding 1900 to the first three characters. Semesters are as follows:
| No. | Semester |
|---|---|
| 3 | Spring |
| 5 | Summer |
| 9 | Fall |
So for example, 1179 means Fall 2017.
I found that our team was re-writing logic to make this conversion from the STRM value to a human readable value in many queries and procedures. Therefore, I decided to create a function to do this conversion for us that could be used by other SQL objects. Enter wfg_f_convert_strm:
CREATE OR REPLACE FUNCTION wfg_f_convert_strm
(
p_strm IN VARCHAR2
)
RETURN VARCHAR2
IS
BEGIN
RETURN
(CASE SUBSTR(p_strm, 4, 1)
WHEN '3' THEN 'Spring'
WHEN '5' THEN 'Summer'
WHEN '9' THEN 'Fall'
END)
|| ' '
|| (SUBSTR(p_strm, 1, 3) + 1900);
END wfg_f_convert_strm;
The goal of the function is modest. Its signature expects one STRM parameter and returns one value. But what if we wanted to write tests for this? Stay tuned for the next post.
A little preamble before we start. For this post I’ll be using example code from the ssl branch of the Kafka demo playbooks repo. These set of playbooks are different from the ones used in the last post, as we’ve made many improvements since writing that post. However, the playbooks should still be thought of as illustrative, not authoritative. I.e., if you try to run them as-is, they probably won’t work.
Also, we’re pretty new to some of the SSL tooling described below. I’m sure we’re taking some unnecessary steps. We will probably refine our work as we learn more.
Also also, Confluent now has their own Ansible playbooks that do SSL. You should take a look at those.
Ok, preamble done.
Read MoreEver since reading Desgining Data Intensive Applications I’ve looked for any excuse to use Apache Kafka. I experimented with the Kafka quickstart and ran some small-scale proof-of-concept projects. When a real project came along that was a perfect fit for using Kafka, I jumped at the chance.
Which is when I learned that there’s a large difference between tinkering with Kafka on my laptop and deploying it to a bunch of real servers. And when I went looking for guides to moving from Quickstart Kafka to Real Kafka, I couldn’t find much. So I wrote my own! The following post covers my use of Confluent and Ansible to configure and automate a multi-host deployment of:
If you’re interested in learning about the Whys or the Whens of using Kafka there are other blog posts (or books, such as Designing Data Intensive Applications) that cover those topics in detail. If you’re interested in a quickstart, then I recommend trying out Confluent’s quckstart. But if you’re looking for how to move beyond experimentation in to a real running Kafka system, read on.
Read MoreRecently I’ve been telling people on my team that they are the “Technical Lead” on a project. Unfortunately, I did not have a clear definition of the role in my mind. And my hazy definition of “I dunno, just be responsible for this work I guess?” didn’t help the people who ended up working as Technical Leads.
Thankfully many people smarter than me have done an excellent job of defining what a Technical Lead is. So I’m going to leverage their work to help clarify the role within my team. Maybe it’ll help your team as well! This will be the first in a series of posts about resources for being a successful Tech Lead.
Eryn’s background is with consultancy work and I think there are strong parallels between consultancies and ASR Custom Solutions which lead us both to need Tech Leads to do similar work for similar reasons. Let’s dig into those parallels.
First, consultants work on a wide variety of projects for short periods of time. This is similar to ASR Custom Solutions where we divide our time amongst a large portfolio of applications. Compare this to a team within a corporation that is responsible for a single service or application. On those teams people are likely to know their single application well; on my team we’re often coming to a code base with little-to-no experience.
The second parallel is that consultants do not own the code that they produce. When the project ends the application becomes the customer’s responsibility. ASR Custom Solutions does not hand ownership of code over to customers but we highly value “Shared Ownership” of code, meaning that no team member owns a project or library. Any of us could be working on any repository at any time. Compare this to organizations which assign application ownership to to a person or job title. On those teams the application or component has a clear ‘owner’; on my team ownership and responsibility is evenly shared.
These two ‘consultancy-like’ traits mean that my team members are
This week you might be working on a tool that you helped develop 5 years ago; next week you might be pairing with someone on a greenfield project that you’ve never seen before. Regardless of what you’re working on, your opinion of what change we should make is still valuable.
This is fun way to work but it comes with challenges! In particular, people on my team have struggled with answering these questions:
Everyone could ask the Team Lead (me) these questions, but then I would become a massive bottleneck and nothing would get done. Enter the Tech Lead! Eryn describes the Tech Lead role as, “the owner of the technical vision for a project and the technical leader of the project team.”
Which means that the Tech Lead is responsible for answering the three questions my team mates need answered. And since they can ask the Tech Lead and not me, they’ll get their answer much faster. Everyone wins.
Additionally, Eryn emphasizes that the role is temporary. The person is only performing the role for the length of the project. This is for the best as tech leads have a lot to do. Eryn lays out the following responsibilities of Tech Leads:
All of these map quite nicely to solving the problems that I saw my team mates having. Eryn’s clear description of a Tech Lead’s responsibilities has helped me understand how I want them to work within ASR Custom Solutions.
In future posts I’ll be talking about other useful talks about Tech Lead work and how Tech Leads work within ASR Custom Solutions.
We are pleased to announce a new JSON API, http://terms.umn.edu. As with our other APIs this is a read-only web service that provides easy access to public data.
In this case the public data in question is, “What term is it today?” You might think this is an easy question to answer. But no! Read on to find out the surprising complexity.
Read MoreI recorded a quick (3 minute) video for a lightning talk introducing hub, a GitHub CLI that I use all the time.
The video is silent, but below is the transcript of what you hear if you heard me give this talk!
Read More