How to solve the basic problem when trying to build serverless react on lambda
The basic problem that we were trying to solve when building serverless react on lambda was that our website sends a lot of email and the way that our website works are. We have a very spiky load on our emails so the way. praise is a system that helps companies manage performance, reviews, and typically.
We want to send out reminders for everyone in the company to say “Hey you’ve got you know 48 hours left to submit your review, hey you’ve got 24 hours left, you’ve got 4 hours left go to the website and actually do your review please”.
So what that means is every so often we go from 0 email sent to a couple hundred or thousand emails sent as quickly as possible. Because these are notifications and they are time-sensitive. Well, we don’t actually do the email sending ourselves.
We do need to render the HTML that goes into each of these emails so that the people receiving the email sees their name and the information related to their review. And it got to the point that our back-end servers were getting bogged down by.
So much HTML and text that they needed to render for these emails. What was a good solution to having a spiky workload we decided to check out AWS lambda and see how we could harness the power of lambda to scale up and down very very seamlessly.
What do we actually run on lambda?
The next question was what do we actually run on lambda and the front end team at our software development company decided why don’t we check out react? Why don’t we try running react on the server side in order to render the HTML for these emails?
So the nice thing about that was it’s a familiar technology, we’re already using it in praise it’s a familiar use case. Because basically rendering HTML is what front development frameworks are all about.
This is like the hips turist of hipster technologies, so we figured why not see just how far we can push the envelope. so we came up with this project that we’re calling react mail merge.
It is open-source. You can go on github right now, check out the project clone, play around with it and so on. Some people might be thinking, so what’s with the name well any of you who were around in the battle days before email got to be super popular.
There was a process in Microsoft Word called a mail merge, where you’d link up your address book with a form template in Microsoft Word that had forms for like a first name and address and so on.
And it linked these two together and created for you a couple hundred different copies of a letter that’s almost the same but has information from the address book for each person. So it’s sort of like what we’re trying to do.
Here this project is not actually delivering emails, it’s just putting together the HTML to do so, plus it’s way retro and we’ve decided that everything old is apparently new again. Because tech keeps on reinventing the wheel over and over and over but that’s inside conversation anyway.
How does this project actually work?
So how does this project actually work? It’s, first of all, it’s more of a set of guidelines than a reusable framework, we don’t really have a project that you can plug your own code into. But the way that the code is we’ve written it so far works is we have one lambda function per email template.
Now this means that your email templates can be as flexible or as inflexible as you want them to be. For example we have an email template for inviting users to our platform, we have an email template for letting them know that they’ve joined the group and so on.
But we also have one email template for all the different reminders that we send whether it’s a four-hour reminder or a 24 hour or 48 hours or one-week reminder or whatever. We all use the same template for that, so you can be as flexible as you want.
We have an API exposed so that you can invoke this temp with structured data mail-merge does data validation to make sure that the data that’s being passed is exactly what’s needed to render the template. And you’re not missing anything important like oh the name of the person who we’re sending the email to, then it renders the email templates both HTML and text.
You can either pass it a single set of data for a single person or you can pass an array of data for every single user that needs to receive one of these emails. Then it takes the results of that rendering and passes it along to your mail delivery service.
We happen to use a company called postmark but you could customize this to use whatever mail delivery service you want and it gets any errors that might occur from the process of sending the emails from the mail delivery service and logs them and returns them to our back-end that’s calling mail merge as well.
What are the benefits of Serverless react on lambda projects?
There were a couple of benefits that we wanted to get out of this project like being able to deal with spiking workloads. But there were also a couple of unexpected benefits that we found along the way.
And briefly those are
- Using storybook for mail preview
- Finding out that fixing the email copy using this project was surprisingly simple using
- JSON schema as documentation for our back-end developers having robust
- Localization and internationalization support
- And also being able to do continuous deployment via our CI system.
Storybook for mail preview
What all those means first let’s start with a storybook I don’t know if any of you are familiar with this really handy project called storybook.
But basically what it allows you to do is to take a component that you’ve written or a section of your code and display it in a way that’s very friendly to see and easy to modify.
So in this case we’ve set up a storybook with a bunch of our different email templates and we’ve added in some basic information that you can use to render the information as well.
We’re using this plugin called storybook knobs, so you can put in the base URL that you want the links to go to the username that you want to show up on in that email.
And a couple of other pieces of information as well it’s really easy to change, it’s really easy to instantly type in a different name type, in a different group name and so on. See how your email is going to look with that new information.
Now as an aside email HTML, HTML email is awful and just because something displays properly in the latest version of Google Chrome does not mean that it’s going to display properly in an ancient version of Microsoft Outlook.
So you have to do more validation than just checking in the storybook, but this is a good start.
Fixing email copy is easy
Another unexpected benefit that we found was easy copy edits, you’re all familiar with react, you’re all familiar with the JSF, with the JSX syntax.
Essentially, it looks more or less like HTML and we found that people at our reactjs development company who were not developers but already knew HTML were able to understand what was going on.
In our email templates and to very easily make copy changes, so if we found that a word was misspelled or there was some phrasing that was a little bit awkward or something like that, the product managers and product owners didn’t have to wait for the developers to get started on this.
They could actually make their own pull requests to the repository and this happened multiple times which was really cool. Because it meant that the developers could focus on the more difficult deeper problems.
And the people who wanted to focus more on exactly how the emails were written and how they were displayed could focus on that very easily as well.
JSON schema as documentation
We also use JSON schema to validate the data coming in to our lambda function. So this serves two purposes one it makes sure that the templates that we’re using receive the data that they need in order to render properly.
For example if you have a reminder email that is supposed to remind someone about an upcoming event, but the data that gets sent to react mail emerges it doesn’t have any information about that event.
You’re not going to be able to generate an email that’s actually meaningful to send off to the user, so we use schema validation to make sure that whatever the backend is passing is actually meaningful and has the correct structure.
But in addition, we found an extra unexpected benefit here which is that the people who are writing the backend of our software and the people who need to actually integrate with AWS lambda we’re able to use these schema files as a form of living documentation.
So they could simply read the JSON schema file and see exactly which information the lambda function was requiring and the format that it was requiring it in JSON schema is a pretty readable standard.
And you can learn more about it on the website we were using a library called a JV to handle the validation, It also does really nice error messages which is important.
Localization and Internationalization support
We also do localization in our emails because we need to be able to display information such as dates and times. In a way that is robust and works for lots of different users and if you don’t think that this is important then you’ve probably never had to deal with the user time zones.
Before we have users all over the world and all of them expect to see notifications in their native locale, it doesn’t help someone who’s living in Europe to see the date. That the date and time, that an event is closing if that time is in u.s. Time zones or vice versa.
Everyone wants to see information in their own local time zone, so that’s why it’s important to be able to have information about your time zone. Fortunately, there’s a really handy library called Format.Js which handles localization.
React.Js really smoothly, I was shocked to find that it was by Yahoo. I didn’t know that Yahoo was still doing cool stuff these days, but apparently they are and lastly we’re doing continuous deployment as well.
Because lambda is very easy to deploy new versions of functions on – and very easy to test as well. By having lots of different functions destroyed deployed in staging services as well.
Continuous deployment via our CI system
We’re using a system called apex to manage how we upload and deploy all of our functions. I’ll be honest, I don’t know all that much about how apex works.
I’m a developer. I’m not so much focused on the DevOps side. But it’s something that was very easy for our DevOps people to set up.
We have our system running on CI specifically we use “CircleCI”. And every time a commit is merged to master it just redeploys all of our lambda functions and it works beautifully.
You can also run automated tests to make sure that the code that you’re writing is safe and it is not going to break things and we have very high test coverage actually on our project. We have 100 test coverage at the moment.
Now it’s not all roses, we did run into a couple of unexpected problems and I’ll go over them briefly.
- Backwards-incompatible changes are painful
- Discovered data problems (But fixed them!)
- Still need to gather performance metrics
- Unexpected timeouts on Lambda
Backwards incompatible changes are painful.!
First of all we discovered that backwards incompatible changes are painful. As I just mentioned we do continuous deployment for all of our lambda functions. So what that means is as soon as something is merged to master it out on production, there’s not really any delay.
If you want to change the schema that we mentioned in the JSON schema that I talked about earlier, let’s say we want to change the name of one of the parameters.
Well we did that a couple of weeks after launch and then shortly discovered that all of our emails were failing and that happened. Because we hadn’t yet made the change to the backend.
That was calling our lambda functions, so it was passing information in the old format whereas lambda was immediately expecting information in the new format.
So it was just throwing validation errors, so we reverted that change very quickly and we’ve learned that now all changes need to be backwards compatible at least until the back end is modified to fit with the new structure of the data as well.
Discovered data problems (But fixed them!)
We also unexpectedly discovered problems in our database, problems with the data that we held in increase. So for example every single one of our emails has a big highlighted call-to-action button.
It might be something like login and accept your invitation or go and submit your review to this person. We had our email setup, so that if you pass an optional invitation token parameter for each user that was supposed to indicate that the user hasn’t actually claimed their account on M phrase yet.
Therefore the call-to-action should say register and submit your review otherwise, it should just say submit your review. Well we had a couple of customers contact us and ask us why they were being asked to register for their emperor’s account that they’d had for a couple months.
Now it turned out that the code in our back-end wasn’t properly deleting the invitation token and the back end was just you know sending that information along to lambda which meant that our emails were doing exactly what they were coded to do.
They were sending along the information that said, you need to register. So having this robust data validation helped us discover problems that we didn’t think we had in a very unexpected place and we were able to solve them pretty quickly.
Still need to gather performance metrics
Another problem, this is not so much a problem just sort of a lack of time issue we haven’t actually gathered performance metrics now.
I said right at the beginning of the talks that one of the reasons why we wanted to do this is because we had a very spiky very heavy workload. I’ll be honest I don’t actually know what our workload was like and I don’t actually know how much it’s improved.
I presume that it has but this is one of the things that we’re looking for help on this we know that we want to move in this direction.
We know that we want to gather performance metrics and make sure that our service, our servers, and our services are running smoothly it’s just not something that we’ve gotten to yet.
Unexpected timeouts on Lambda
There’s another problem that we’ve been having where we have unexpected timeouts on an AWS Lambda. Most of our functions go perfectly, smoothly and all of our emails are getting sent out.
But we’ve discovered that some of these lambda functions after the email gets sent out after we send it over to postmark, our email delivery service just kind of hangs out just kind of waits on lambda until the timeout happens doesn’t exit.
Just wait until lambda kills the process, we don’t know why we’re hoping that we can figure out why that’s happening so.
We can cut off these processes earlier because of course, every millisecond that we spend on lambda costs a little bit more money. So it’d be really nice to figure out what’s causing these timeouts.