Build your own Custom Database using TypeScript
This project will help you grasp low-level systems without requiring any prior knowledge of database engineering. I’ve included all the essential details to guide you through how everything works, and I’ve also used ChatGPT to strengthen my understanding of the concepts. Hope this helps you.
Note
This project is a work in progress, and the content of this blog may be updated over time. We’re also documenting the journey through a playlist on our YouTube channel, so be sure to subscribe!
Introduction
As a developer, I wanted to understand how databases work under the hood, including data structures like B-Trees and Red-Black Trees. Instead of just reading about them, I decided to build my own database from scratch in TypeScript. This post outlines my journey of creating LumeDB, an industry-ready, file-based, and performant database.
Technologies
Feel free to choose any other alternatives you are familiar with. These are my choices:
- TypeScript - We'll be using TypeScript as our core language but you can go along with GO lang.
- Bun - We'll be using Bun as our runtime.
- Docker - We'll need to dockerize everything in order to scale & run multiple database instances in containers.
- Hono.Js - We'll be using Hono to build Ultrafast & Lightweight API endpoints.
Project Setup
- Language & Runtime: TypeScript + Bun
- Folder Structure:
├── src/
│ ├── storage/
│ │ ├── file.mngr.ts # File handling logic
│ ├── server.ts # HTTP API for the database
│ ├── cli.ts # Command Line Interface
│ ├── main.ts # Entry point
├── data/ # Stores database files
├── Dockerfile # Docker setup
├── docker-compose.yml # Container orchestration
- Storage Choice: Binary files (.bin) for efficient data retrieval
File-Based Storage Implementation
- Implemented a FileManager to handle reading and writing efficiently: