Configuration management in Golang. Part 1: Environment Variables.

Any big enough web project has a lot of things which can be configured here and there. What is the current application environment? What port should server be listening for incoming connections? What is our publicly available hostname we should use for links in generated emails? Should SMS messages be sent or just logged to save some money on testing stages? What about reporting data to New Relic, should it be working? All of those are settings.

Another part of configurable parameters are secrets. Starting from session secret which is used to generate secure JWT token and going on to different access tokens, secret keys and login / password pairs for external integrations.

The question is how can we manage all of that easily taking into account that amount of configuration options may grow with time?

Well, there is an approach called “The 12 factor” originally endorsed by Heroku platform. It says that you should store all of your configuration in application environment. There is no farther clarification in this document what does “application environment” mean but in practice it’s usually system level environment variables. In the brave new world of Docker images those variables are usually provided by orchestrating system, such as Kubernetes or Nomad, during deployment process and consumed by application on start.

Sounds good enough, so let’s start from implementing that.

First of all we will need to decide how we are going to represent our configuration values inside of our code. There is quite popular library called Viper which stores settings as key-value pairs with some type casting methods on top of that. By following this way you are obviously loosing quite a good thing — type checks. Not that fun… Can we actually use regular strictly typed structure instead? Yes, we can! Let’s just assume that our application configuration is a big-big structure with many fields and see how it works? We will start from one simple thing: application environment name. It’s a string such as production , test , staging or something like this. This setting determines how application instance behaves in many different ways. Quite important thing, huh? Let’s read it first.

Of course, you can find many techniques to get values from operating system environment variables into a structure. In the simplest case just use os.Getenv from the standard library and do everything yourself on top of that. Why not? It’s a lot of code though. For the sake of simplicity we will take a shortcut and use a library: https://github.com/sethvargo/go-envconfig.

Here is a starting point code:

Let’s launch our little test and see what it prints out:

➜  envconfig MYAPPENVIRONMENT=production go run main.go
2021/04/13 23:44:08 production

We managed to specify application prefix to differentiate “our” variables from everything else is the system. Nice, but I would prefer to have some underscore based separation of the name, like MYAPP_. Quite easy to add I guess. Another thing I really don’t like about our code is a struct tag written in caps. It screams! Can we make it lower? In the end of the day environment variable name is not case sensitive by default, so it makes a lot of sense to ignore letters case in our struct tags as well.

Let’s check it out:

Nope. Stopped working:

➜  envconfig MYAPPENVIRONMENT=production go run main.go
2021/04/13 23:49:11

Frustrating… But luckily for us go-envconfig has pretty much straightforward way to extend how variables are being searched in the system by implementing simple Lookuper interface defined here:

Now we can teach our code how to deal with variable names in any reasonable case:

Nicely working as expected:

➜  envconfig MYAPP_ENVIRONMENT=production go run main.go
2021/04/14 00:00:19 production

Now let’s see what about namespaces? Sometimes we would like to have some kind of a section (let’s say newrelic) which has a key inside (we will use token ). Can we do that?

Sadly when we run our new version of the application nothing is printed in the second row:

➜  envconfig MYAPP_ENVIRONMENT=production MYAPP_NEWRELIC_TOKEN=8a74d4a5 go run main.go
2021/04/14 10:42:57 Environment:  production
2021/04/14 10:42:57 New Relic Token:

Looks like our little library doesn’t support nested struct tags out of the box. In order to make it work we have to specify the full key manually:

This a bit annoying but we can live with it for now. Good point of improvement though. Now it works:

➜  envconfig MYAPP_ENVIRONMENT=production MYAPP_NEWRELIC_TOKEN=8a74d4a5 go run main.go
2021/04/14 10:45:37 Environment:  production
2021/04/14 10:45:37 New Relic Token:  8a74d4a5

With a little help we achieved pretty good result, didn’t we? Our configuration is scanned from the environment to the struct in a few lines of code and now can be passed down to the application logic in form of a pointer. But with all of a goodness there is a dark side of using environment variables only.

First of all, it’s hard to manage. Regular web application has dozens of settings. It means you need to provide each and every of them in form of an environment variable when you are starting your container up. For production / staging that can be viable. Automated configuration system will store those settings somewhere and provide them when CI / CD makes a deploy. What really suffers is a development experience. Adding tons of variables to the script when you would like to test an app locally is a hell of a pain.

Storing default local settings in some kind of .env file partially solves the problem, but in many cases it’s quite insecure: for example, you can’t keep some development secrets there. It means that such kind of a settings will be passed between members of a team in Slack or by email and stored locally. We also have to keep this .env up to date which is not always happening in the real life.

So, in the next post of this series let’s try to find another way of configuration management in addition to environment variables which will help us to improve DX and security. There should be something better anyway!