Academic Support Resources Custom Solutions

Making a Positive Difference in Students Lives

Blog

It's a Blog

Lastpassify

A Ruby gem to automate YAML configuration files with data from LastPass

At ASR Custom Solutions, we support a number of applications and systems. These run in multiple environments and hosts, talking to other services that also exist in different environments. At any point, there’s dozens of passwords, secrets, connection strings and other configurations we have to manage.

Our team stores these in LastPass. When we need to access them though, we had to deal with hunting within LastPass for the specific note we needed. For a large, microservicey-system like our Student Degree Progress Service, this could mean dozens of hostnames, usernames and passwords. Onboarding a new developer to the project was distinctly painful here.

So, we made a Ruby gem to automate all this for us; Lastpassify.

Lastpassify is a commandline tool packaged as a Ruby gem that takes in an YAML ERB template file and outputs a populated YAML file. We primarily use it for our database.yml files. Now, what might’ve taken 20 minutes or more to fully populate a large config file by hand is done in milliseconds.

Besides efficiency improvements, using the gem has given security benefits as well. For example, having trust in the tool means that we’re able to delete our credentials and repos when we are not working on the project. Not having the credentials on the developer machine adds to the security protecting the data behind our applications.

Usage

Lastpassify requires lastpass-cli and Ruby v2+ to be installed.

LastPassify expects an input of one YAML file to be processed and outputs one YAML file. The input file can be passed in as an argument at the commandline like so:

$ bundle exec lastpassify my_input_file.yml

The output file and path can also be specified:

$ bundle exec lastpassify my_input_file.yml config/my_output_file.yml

LastPassify has default values that silently get passed if no input or output file is specified. The default input file LastPassify will look for is config/database.example.yml. The default output will also live in the config directory, with a filename of database.yml.

An example database.example.yml file might look like this:

---
# Shared
global_defaults: &global_defaults
  adapter: oracle_enhanced

# Development environment
development: &development
  <<: *global_defaults
  host: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='Hostname') %>
  database: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='Database') %>
  username: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='username') %>
  password: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify', field='password') %>
  secret_key: <%= lookup('lastpass', 'Shared-Artifactory/lastpassify_secret_key', field='Secret Key') %>

staging:
  <<: *development

Finally, Lastpassify strips out any YAML keys with production, staging or qat in their names. This is a security measure to ensure no prod or staging environment credentials sit on your local development machine unnecessarily.

This can be overridden by passing in a -s (staging) or -p (prod) flag to Lastpassify, e.g.: $ bundle exec lastpassify -p

We hope you find it useful and appreciate any pull requests!


Testing Oracle SQL objects with utPLSQL

Part 1

At ASR, we have a large number of SQL queries, functions and stored procedures for retrieving and manipulating data. One value we stand by for the software we maintain is test-driven development. Generally speaking, I feel we do a pretty good job of this. However, there is one area that had glaring holes: our large corpus of SQL objects (functions and stored procedures).

Enter utPLSQL, a unit-testing framework for Oracle PL/SQL. If you’ve ever worked with RSpec, it will look very familiar.

This will be a two part series. In this post, we’ll go over a sample SQL function that we use at ASR. In the next, we’ll go over how to use utPLSQL to write unit tests for it and nice features of utPLSQL you might find handy like reporting.

wfg_f_convert_strm.sql

At the U, we have three terms per academic year; Spring, Summer and Fall. We store these in our databases in a scheme known as “STRM”. This consists of a four-digit number, the first three of which hold the year value and the last digit representing the semester. The year value is calculated by adding 1900 to the first three characters. Semesters are as follows:

No. Semester
3 Spring
5 Summer
9 Fall

So for example, 1179 means Fall 2017.

I found that our team was re-writing logic to make this conversion from the STRM value to a human readable value in many queries and procedures. Therefore, I decided to create a function to do this conversion for us that could be used by other SQL objects. Enter wfg_f_convert_strm:

CREATE OR REPLACE FUNCTION wfg_f_convert_strm
(
  p_strm IN VARCHAR2
)
  RETURN VARCHAR2
IS
BEGIN
  RETURN
      (CASE SUBSTR(p_strm, 4, 1)
        WHEN '3' THEN 'Spring'
        WHEN '5' THEN 'Summer'
        WHEN '9' THEN 'Fall'
        END)
      || ' '
      || (SUBSTR(p_strm, 1, 3) + 1900);
END wfg_f_convert_strm;

The goal of the function is modest. Its signature expects one STRM parameter and returns one value. But what if we wanted to write tests for this? Stay tuned for the next post.


Encrypting Apache Kafka Traffic

SSL for Kafka servers and clients

A little preamble before we start. For this post I’ll be using example code from the ssl branch of the Kafka demo playbooks repo. These set of playbooks are different from the ones used in the last post, as we’ve made many improvements since writing that post. However, the playbooks should still be thought of as illustrative, not authoritative. I.e., if you try to run them as-is, they probably won’t work.

Also, we’re pretty new to some of the SSL tooling described below. I’m sure we’re taking some unnecessary steps. We will probably refine our work as we learn more.

Also also, Confluent now has their own Ansible playbooks that do SSL. You should take a look at those.

Ok, preamble done.

Read More

Apache Kafka

From Quickstart to Reality with Ansible and Confluent

Ever since reading Desgining Data Intensive Applications I’ve looked for any excuse to use Apache Kafka. I experimented with the Kafka quickstart and ran some small-scale proof-of-concept projects. When a real project came along that was a perfect fit for using Kafka, I jumped at the chance.

Which is when I learned that there’s a large difference between tinkering with Kafka on my laptop and deploying it to a bunch of real servers. And when I went looking for guides to moving from Quickstart Kafka to Real Kafka, I couldn’t find much. So I wrote my own! The following post covers my use of Confluent and Ansible to configure and automate a multi-host deployment of:

If you’re interested in learning about the Whys or the Whens of using Kafka there are other blog posts (or books, such as Designing Data Intensive Applications) that cover those topics in detail. If you’re interested in a quickstart, then I recommend trying out Confluent’s quckstart. But if you’re looking for how to move beyond experimentation in to a real running Kafka system, read on.

Read More

Tech Lead with Eryn O'Neil

Recently I’ve been telling people on my team that they are the “Technical Lead” on a project. Unfortunately, I did not have a clear definition of the role in my mind. And my hazy definition of “I dunno, just be responsible for this work I guess?” didn’t help the people who ended up working as Technical Leads.

Thankfully many people smarter than me have done an excellent job of defining what a Technical Lead is. So I’m going to leverage their work to help clarify the role within my team. Maybe it’ll help your team as well! This will be the first in a series of posts about resources for being a successful Tech Lead.

Eryn O’Neil - You’re the Tech Lead! Now What?

Eryn’s background is with consultancy work and I think there are strong parallels between consultancies and ASR Custom Solutions which lead us both to need Tech Leads to do similar work for similar reasons. Let’s dig into those parallels.

First, consultants work on a wide variety of projects for short periods of time. This is similar to ASR Custom Solutions where we divide our time amongst a large portfolio of applications. Compare this to a team within a corporation that is responsible for a single service or application. On those teams people are likely to know their single application well; on my team we’re often coming to a code base with little-to-no experience.

The second parallel is that consultants do not own the code that they produce. When the project ends the application becomes the customer’s responsibility. ASR Custom Solutions does not hand ownership of code over to customers but we highly value “Shared Ownership” of code, meaning that no team member owns a project or library. Any of us could be working on any repository at any time. Compare this to organizations which assign application ownership to to a person or job title. On those teams the application or component has a clear ‘owner’; on my team ownership and responsibility is evenly shared.

These two ‘consultancy-like’ traits mean that my team members are

  1. Changing projects frequently
  2. Learning unfamiliar code
  3. Trusted with making good decisions

This week you might be working on a tool that you helped develop 5 years ago; next week you might be pairing with someone on a greenfield project that you’ve never seen before. Regardless of what you’re working on, your opinion of what change we should make is still valuable.

This is fun way to work but it comes with challenges! In particular, people on my team have struggled with answering these questions:

  1. What should we work on next?
  2. How do we solve this problem?
  3. Is this solution a good idea?

Everyone could ask the Team Lead (me) these questions, but then I would become a massive bottleneck and nothing would get done. Enter the Tech Lead! Eryn describes the Tech Lead role as, “the owner of the technical vision for a project and the technical leader of the project team.

Which means that the Tech Lead is responsible for answering the three questions my team mates need answered. And since they can ask the Tech Lead and not me, they’ll get their answer much faster. Everyone wins.

Additionally, Eryn emphasizes that the role is temporary. The person is only performing the role for the length of the project. This is for the best as tech leads have a lot to do. Eryn lays out the following responsibilities of Tech Leads:

All of these map quite nicely to solving the problems that I saw my team mates having. Eryn’s clear description of a Tech Lead’s responsibilities has helped me understand how I want them to work within ASR Custom Solutions.

In future posts I’ll be talking about other useful talks about Tech Lead work and how Tech Leads work within ASR Custom Solutions.