Get the latest tech news

How we built ngrok's data platform

At ngrok, we manage a ~100TiB, 500+ table data lake, managed by a very small team. Here's a look at how we built it and what unique challenges we solved.

Our data stack was also not integrated very deeply in our Go monorepo and tooling, missing things like Datadog monitors and metrics, good test coverage, or style guides and enforcements via CI (see the section). Speaking of, especially in smaller companies, equipping all engineers with the technical tooling and knowledge to work on all parts of the platform (including data) is a big advantage, since it allows people not usually on your team to help on projects as needed. While there are a lot of topics we didn't cover in this article, I hope it provided both a good high-level overview about data engineering work at ngrok, as well as some details on specific challenges we faced and solved.

Get the Android app