Build your own Custom Database using TypeScript

This project will help you grasp low-level systems without requiring any prior knowledge of database engineering. I’ve included all the essential details to guide you through how everything works, and I’ve also used ChatGPT to strengthen my understanding of the concepts. Hope this helps you.

Note

This project is a work in progress, and the content of this blog may be updated over time. We’re also documenting the journey through a playlist on our YouTube channel, so be sure to subscribe!

Introduction

As a developer, I wanted to understand how databases work under the hood, including data structures like B-Trees and Red-Black Trees. Instead of just reading about them, I decided to build my own database from scratch in TypeScript. This post outlines my journey of creating LumeDB, an industry-ready, file-based, and performant database.

Technologies

Feel free to choose any other alternatives you are familiar with. These are my choices:

  • TypeScript - We'll be using TypeScript as our core language but you can go along with GO lang.
  • Bun - We'll be using Bun as our runtime.
  • Docker - We'll need to dockerize everything in order to scale & run multiple database instances in containers.
  • Hono.Js - We'll be using Hono to build Ultrafast & Lightweight API endpoints.

Project Setup

  • Language & Runtime: TypeScript + Bun
  • Folder Structure:
├── src/
│   ├── storage/
│   │   ├── file.mngr.ts  # File handling logic
│   ├── server.ts         # HTTP API for the database
│   ├── cli.ts            # Command Line Interface
│   ├── main.ts           # Entry point
├── data/                 # Stores database files
├── Dockerfile            # Docker setup
├── docker-compose.yml    # Container orchestration
  • Storage Choice: Binary files (.bin) for efficient data retrieval

File-Based Storage Implementation

  • Implemented a FileManager to handle reading and writing efficiently: