Recreating Google Photos Part I

Preface

Originally, this project started as a final assignment for my Server-Side development class. We were expected to create a system that encompassed all the server-side techniques/technologies that we learned throughout the quarter.

After brainstorming with my team of 3 others, we decided that we wanted to recreate Google Photos, essentially acting as a data-store for images, all the while enabling smart tagging that would allow for users to search for photos using generic tags. We also planned to add some bells & whistles, allowing for users to share albums with others, server-side image filtering (sepia, black&white, etc), and some generic social media functionality (sharing, commenting, etc).

Simple enough... right? From our initial glance, it seemed like it was only 2 microservices and a gateway that we could bootstrap from a previous assignment. Plus, we had two whole weeks!

In typical college fashion, we thought it was a good idea to wait a week before starting to work. It was upon our second meeting where we began to realize the scope and complexity that the project entailed, and we knew we had to get to work ASAP.

Architecture

Our project breaks down into the following components:

1. Gateway

2. Image Microservice

3. Tagging Microservice

We decided to build our system using the Microservice architectural pattern (for a refresher on Microservices, visit here). Each of our microservices were anchored together by a Gateway that would act as our authentication & session-state manager, as well as a reverse-proxy that forwards requests to the proper host.

In the first part of this series of articles related to this distributed system, I will discuss the architecture and features built into our Api Gateway. I will also dive deep into the techniques, technologies, and methodologies used to implement the module, as well as show some basic code I used for implementation.

In the second part of this series, I will dive deep into each microservice, analyzing the architecture that we chose, the reasoning behind those choices, and the methodology & technology used to build the modules.

Now, let's get started!

The Api Gateway

Our gateway was built out using Golang, a newer language that offers great benefits for writing server-side code due to its great concurrency support, simplicity, and quick build-time.

Authentication

In this step, we provided simple sign-up/sign-in with basic auth and https encryption, hashing passwords and emails using golang’s md5 and bcrypt package, as well as generating and validating digital certificates. We also built out a custom index-trie that allowed for indexing specific values and quick prefix-search operations (typing '@m' and finding all users with first name with a prefix of ‘m’). We then hosted our user-store on Mongo using golang’s mgo package.

Up next was handling Sessions.

Sessions

In this step, we generated our own session-state and utilized Redis as a cache to store our state objects. Redis was a great fit here, as its Expire functionality allowed for session-timeouts after n seconds by evicting the session struct from the cache. As for the inner workings, there was nothing too special here — we generated our own sessionID with the following format:

For authorization, we verified that each request header contained a Bearer Token and then validated that our redis-cache contained the corresponding state.

Now that authentication and sessions were setup, it was time to setup our gateway as a reverse-proxy.

Reverse-Proxy

First off, what is a Reverse-Proxy?

Here, our gopher, representing the reverse-proxy, fires requests to the appropriate service-instance, based on the context of the request from various clients.

A reverse proxy is simply a module that sits in-between a request and a server. The job of a reverse proxy is to forward requests from a client to an appropriate server that can handle the incoming request. In the context of microservices, the reverse-proxy is in charge of forwarding a request to not only the correct microservice, but also a service instance of that microservice. This means if your cluster for Microservice X contains the following hosts,

[196.0.0.1, 196.0.0.2, 196.0.0.3]

then your reverse-proxy should forward the request to either of the host-addresses listed above. (I will discuss later how to optimize this choice of which host to forward the request to via load-balancing)

If this were a static system, we could hard-code these host ip-addresses, like that of above, in some environment variable, and then allow the reverse-proxy to read that list. This, however, is a naive solution, as most modern cloud-based distributed-applications are elastic by nature, allowing for the addition and removal of hosts based on current load.

This leads to a really big problem. How does the reverse-proxy know where to forward the incoming request, if the host addresses are not hard-coded? One solution to this dilemma is Service Discovery.

Dynamic Service Discovery

What is Dynamic Service Discovery?

Dynamic Service Discovery (DSD for this article) is the process of dynamically adding/removing service-hosts to/from the pool of possible hosts for the reverse-proxy to forward requests to.

This solves our question of knowing where to forward incoming requests to! With DSD, having to manually add host-addresses to a spec file is avoided, as whenever a new service-instance is spun up, the system will automatically detect and add that instance to the pool!

Service Registry At the core of any service discovery solution, there is a service registry that contains the information needed by the reverse-proxy. This means that each microservice must register itself with the service-registry upon creation. Furthermore, to ensure the hosts within the service registry are healthy, each microservice must publish a heartbeat every n seconds.

In my implementation, I used Redis as our service registry, and I will explain why in the next section.

Heartbeats

A heartbeat is nothing more than an object that contains some information about the current host. Our heartbeat struct looks like this:

Some existing DSD platforms include: Consul, Zookeeper, and more.

My implementation of Dynamic Service Discovery

My implementation is broken into 2 major parts:

1) Redis as a Service Registry

2) Service Discovery Middleware

Redis as a Service Registry

We decided to go with Redis as our service registry because of it's Pub-Sub functionality. Pub-Sub, or the publish and subscribe model, allows for services to subscribe to a specific channel of a Redis Client instance, as long as your microservices and gateway share the same private network. Since our microservices shared the same private network (Digital Ocean Private Network in our case), we were able to publish these heartbeats from our microservices to a specified channel. Then, on the gateway side, I subscribed to the specific channels on the Redis Client and spawned a goroutine to listen for heartbeat packets sent from the microservices.

As mentioned above, this type of model requires a contract between each microservice and gateway. Each microservice must publish a heartbeat every n seconds in order to be considered healthy, or else my "janitor" goroutine will remove the host from the service registry.

If you're interested, here is the code on the Gateway side that listens for heartbeat messages. Code for publishing heartbeats will be shown when I analyze each microservice.

Note: If you're unfamiliar with Go, the syntax can seem a bit wonky. If you don't understand the code, don't worry, as understanding the architecture and high-level concepts is more important here! I'll also eventually attach the repository if you want to see the full implementation.

Service Discovery Middleware

First off, what is middleware?

In a standard http(s) web server, the lifecycle of a request is as follows:

With middleware, however, another layer is added between the server and handlers, like so:

This allows for not only injecting logic before a request is routed to a handler(s), but post-request processing as well.

Visit here for examples and in-depth analysis of middleware patterns in golang.

Now that we know what middleware is, let's get back to Dynamic Service Discovery.

Since our gateway is actively listening for heartbeat messages via Redis Pub-Sub, I choose to store a map of Heartbeats in memory, where the key is the name of the microservice, and the value is a slice of Heartbeat objects.

I then, in my DSD middleware, inject logic to extract the correct set of hosts based on the request context -- prior to the request being routed to a handler. My middleware then attaches a delimited list of host-addresses as a request-header. This list is then parsed in my handler function, which is then sent to the load-balancer, which then dictates which host to reverse-proxy the request to.

For security reasons, after the request has processed, I then remove the metadata from the header in my middleware to avoid users from knowing sensitive host information.

Summary

Make it here in one piece? If not, don't worry, this stuff can get confusing, especially when it's 3AM with deadlines fast approaching and you don't have the centerpiece of your project working.

Note: Yerba Mate did not sponsor this post.

Note 2: I do not advise one to drink this much caffeine in a short timespan, nor do I advise procrastination!

Now let's summarize

Just in case you forgot or got too confused, this post covered the high-level concepts I used to build the Api Gateway for our embedded system!

We started it all off with authentication. For this part, we went over the generic structure of the module, as well as elaborate about our indexing trie for quick prefix-search operations.

Up next was session management. In this part, we covered our hashing technique for generating session-ids, using Redis as a cache for storing session objects for individual users, and then quickly covered how we handle authorization via Bearer Tokens.

We then dived into the more core features of an Api Gateway, starting with the Reverse Proxy. With the gopher diagram, we learned how a reverse-proxy works, as it routes requests to the correct service-instance. In terms of implementation, we used go's native httputil package to actually route requests to the appropriate host.

Finally, we got into dynamic service discovery. Here, we learned the overarching architecture that powers any service-discovery solution. I elaborated on the contract between microservice and gateway, where each microservice publishes its heartbeat packet to a service-registry. I then went into implementation details, discussing how we used Redis' Pub-Sub functionality to connect the gateway and microservices. The gateway would then listen for heartbeats on specified channels. Having access to all the healthy hosts, I then inject logic to extract the pool of possible hosts using middleware. From there, the hosts are sent to the load-balancer, which finally chooses which host for the reverse-proxy to forward the request to.