Redis Complete Production Guide

by

Redis (Remote Dictionary Server) is an open-source, in-memory data structure store used as a database, cache, message broker, and streaming engine.

What is Redis?

Redis (Remote Dictionary Server) is an open-source, in-memory data store used as a database, cache, message broker, and queue. It stores everything in RAM, which is why operations are blazing fast — typically sub-millisecond. It was created in 2009 and remains one of the most widely used tools in backend engineering.

The single most important thing to understand: Redis is single-threaded for command execution. Every command runs atomically — no two commands ever run simultaneously, which eliminates race conditions by design.

Core Architecture

Redis operates on a single-threaded event loop model. This design choice eliminates locking overhead and race conditions, making operations atomic by default. Every command in Redis is guaranteed to complete without interruption, which simplifies reasoning about concurrency. In newer versions (Redis 6+), I/O threading was introduced to parallelise network operations without breaking the single-threaded command execution model.

Data in Redis is organised as key-value pairs, where the key is always a string and the value can be one of many rich data structures. Redis optionally persists data to disk through configurable mechanisms, but its defining characteristic is that the working dataset resides entirely in memory.

Data Structures

One of Redis’s greatest strengths is its rich set of native data structures, each optimized for specific use cases.

Strings are the most basic type, capable of holding text, integers, or binary data up to 512 MB. They support atomic increment/decrement operations (INCR, DECR), making them ideal for counters. Strings can also store serialised objects like JSON or MessagePack blobs.

Lists are ordered collections of strings, implemented as doubly linked lists. They support efficient push and pop operations at both ends (LPUSH, RPUSH, LPOP, RPOP), making them perfect for queues, stacks, activity feeds, and recent-item lists. Lists can be used to implement simple message queues with blocking operations like BRPOP.

Hashes store field-value pairs within a single key, analogous to a dictionary or map. They are memory-efficient for representing objects with multiple attributes (e.g., a user profile with name, email, and age fields). Hashes allow you to retrieve and update individual fields without fetching the entire object.

Sets are unordered collections of unique strings. They support powerful set operations like union (SUNION), intersection (SINTER), and difference (SDIFF), which can be performed across multiple sets. Sets are excellent for tracking unique visitors, common friends between users, or tagging systems.

Sorted Sets (ZSets) combine the uniqueness of sets with a score for each member, maintaining elements in sorted order by their score. This makes them ideal for real-time leaderboards, priority queues, and range queries. The underlying data structure (a skiplist combined with a hash table) ensures O(log N) operations.

Bitmaps allow you to manipulate individual bits within a string value. With just a few bytes, you can track hundreds of millions of boolean flags — such as whether a user has logged in on a specific day — with incredibly compact storage and fast bitwise operations.

HyperLogLog is a probabilistic data structure that estimates the cardinality (number of unique elements) of a set using a fixed, small amount of memory (~12 KB), with an error rate of less than 1%. It’s ideal for counting unique page views, unique search queries, or unique users at a massive scale where exact counts are unnecessary.

Streams (introduced in Redis 5.0) are an append-only, log-like data structure inspired by Apache Kafka. They support consumer groups, message acknowledgement, and persistence, making them suitable for event sourcing, activity tracking, and real-time data pipelines.

Geospatial Indexes allow you to store geographic coordinates and perform proximity queries (e.g., “find all restaurants within 5 km of this location”) using commands like GEOADD and GEODIST.

Common Commands

String

The most basic type. Stores text, integers, or binary data (up to 512 MB).

SET key value
SET name "Alice"
GET name                        -- → "Alice"
Python

MSET and MGET are just SET and GET for multiple keys at once

-- Normal SET one by one
SET name "Alice"
SET age  "25"
SET city "Mumbai"

-- MSET all at once
MSET name "Alice" age "25" city "Mumbai"


-- Normal GET one by one
GET name    -- → "Alice"
GET age     -- → "25"
GET city    -- → "Mumbai"

-- MGET all at once
MGET name age city   -- → ["Alice", "25", "Mumbai"]
Python


SET counter 0
INCR counter                    -- → 1
INCRBY counter 5                -- → 6
DECR counter                    -- → 5
DECRBY counter 2                -- → 3

APPEND name " Smith"            -- → "Alice Smith"
STRLEN name                     -- → 11



SETEX key seconds value
SETEX session 3600 "abc123"   -- expires in 3600s
TTL session                  -- → 3600
Python

Hash Commands

Storing an object/record with multiple fields under one key

HSET and HGET are SET and GET, but for a Hash field 

-- Normal SET (string)
SET name "Alice"        -- stores a plain string

-- HSET (hash)
HSET user name "Alice" age "25" city "Mumbai"  -- stores inside a hash object

'''
"user": {
        "name" : "Alice",
        "age"  : "25",
        "city" : "Mumbai"
    }
'''

HGET user name                --→ "Alice"
HGET user age                 --→ "25"             
HGETALL user                  -- →  ["name","Alice","age","25","city","Mumbai"]
Python

HMSET = H (Hash) + M (Multiple) + SET

Both do exactly the same thing — HMSET is deprecated since Redis 4.0, HSET replaced it.

HSET was upgraded to handle multiple fields, so HMSET became useless and got deprecated.

-- HMSET (old way)
HMSET user:1 name "Alice" age "25" city "Mumbai"

-- HSET (new way) — does the same thing now
HSET user:1 name "Alice" age "25" city "Mumbai"
Python

HMGET = H (Hash) + M (Multiple) + GET

But HGET was never upgraded:

-- HGET way (3 commands)
HGET user name  --> "Alice"
HGET user age   --> "25"
HGET user city  --> "Mumbai"

-- HMGET way (1 command)
HMGET user:1 name age city ["Alice", "25","Mumbai"]
Python

HINCRBY user:1 age 1            -- → 26
HDEL user:1 city
HEXISTS user:1 name             -- → 1 (true)
HKEYS user:1                    -- → ["name", "age"]
HVALS user:1
Python

List Commands

Lists are ordered sequences — great for queues, activity feeds, or tracking recent items.

RPUSH queue "task1" "task2" "task3" -- adds elements to the right (tail) of the list

LPUSH queue "task0"     -- adds element to the left (head) of the list

LRANGE queue 0 -1       -- reads all elements without removing them
                        -- → ["task0","task1","task2","task3"]

LLEN queue              -- returns total number of elements → 4

LPOP queue              -- removes and returns the leftmost element → "task0"

RPOP queue              -- removes and returns the rightmost element → "task3"

LINDEX queue 0          -- reads element at index without removing → "task1"

LSET queue 0 "updated"  -- updates element at a specific index
Python

Set Commands

Sets store unique unordered elements — ideal for tags, unique visitors, or membership checks.

SADD tags "redis" "nosql" "db" -- adds one or more members to a set (duplicates ignored)
                        
SMEMBERS tags           -- returns all members → {"redis", "nosql", "db"}

SCARD tags              -- returns count of members → 3

SISMEMBER tags "redis"  -- checks if a member exists → 1 (yes) or 0 (no)

SREM tags "db"          -- removes a member from the set

SADD tags2 "redis" "cache"

SINTER tags tags2       -- returns members common to both sets → {"redis"}

SUNION tags tags2       -- returns all unique members across both sets

SDIFF  tags tags2       -- returns members in tags but NOT in tags2
Python

Sorted Set Commands

Sorted sets are like sets but each member has a score — perfect for leaderboards, rankings, or priority queues.

ZADD leaderboard 100 "Alice"   -- adds member with a numeric score
ZADD leaderboard 200 "Bob"
ZADD leaderboard 150 "Charlie"

ZRANGE leaderboard 0 -1 WITHSCORES
                        -- returns all members sorted low → high with scores
ZREVRANGE leaderboard 0 -1 WITHSCORES
                        -- returns all members sorted high → low with scores

ZRANK leaderboard "Alice"    -- returns 0-based rank (low to high) → 0

ZREVRANK leaderboard "Bob"   -- returns 0-based rank (high to low) → 0

ZSCORE leaderboard "Charlie" -- returns the score of a member → 150

ZINCRBY leaderboard 50 "Alice"  -- increases a member's score by given amount

ZREM leaderboard "Bob"          -- removes a member from the sorted set
Python

Pub/Sub Commands

Pub/Sub enables real-time messaging — one publisher sends, many subscribers receive instantly.

-- Terminal 1: listener subscribes to a channel and waits for messages
SUBSCRIBE news:sports

-- Terminal 2: publisher pushes a message to that channel
PUBLISH news:sports "Team India won!"

PSUBSCRIBE news:*       -- subscribes to all channels matching a pattern
                        -- e.g. receives from news:sports, news:cricket, etc.
Python

Transaction Commands

Transactions group multiple commands so they execute atomically — all succeed or none do.

MULTI                   -- marks the start of a transaction block
SET balance 500         -- queued (not executed yet)
INCRBY balance 100      -- queued (not executed yet)
EXEC                    -- executes all queued commands atomically

DISCARD                 -- cancels the transaction, clears the queue

WATCH balance           -- monitors a key; if it changes before EXEC,
                        -- the transaction is automatically aborted
Python

Quick cheat sheet by use case:

Use CaseCommands
Session storeSETEX to set with TTL, GET to read, DEL on logout
Rate limitingINCR per request + EXPIRE to reset window
LeaderboardZADD to update score, ZREVRANGE to show top N
Job queueRPUSH to enqueue, LPOP to dequeue
Unique visitorsSADD per visit, SCARD for total count
User profileHSET to store fields, HGETALL to fetch all
Real-time messagingPUBLISH to send, SUBSCRIBE to receive

Persistence

Although Redis is an in-memory store, it offers two primary mechanisms for durability.

RDB (Redis Database Snapshotting) takes point-in-time snapshots of the dataset at configurable intervals and writes them to a binary .rdb file on disk. RDB is compact, fast to restore, and creates minimal performance overhead during normal operation. The trade-off is that data written since the last snapshot can be lost in a crash.

AOF (Append-Only File) logs every write command to a file. Upon restart, Redis replays the log to reconstruct the dataset. AOF can be configured to fsync on every write (maximum durability), every second (a common balance), or never (leaving it to the OS). AOF files are more human-readable and durable than RDB, but they grow larger over time and can be slower to load.

Redis also supports a hybrid persistence mode (available since Redis 4.0) that combines both approaches: an RDB snapshot is embedded at the start of the AOF file, speeding up restarts while retaining the durability of AOF logging.

For use cases where durability is entirely unnecessary (pure caching), both mechanisms can be disabled.

Use Cases in Production

Redis is employed across virtually every industry. Some of its most prominent real-world uses include:

  • Session storage is one of the most universal use cases. Because HTTP is stateless, web applications store session tokens and user state in Redis for fast, centralised access across multiple application servers.
  • Real-time leaderboards and rankings leverage Sorted Sets to maintain live scoreboards for gaming platforms, sports apps, and competitive platforms with millions of players.
  • Rate limiting and throttling use atomic increment operations on keys with TTLs to enforce API rate limits and protect services from abuse.
  • Job queues and task scheduling use Redis Lists or Streams as the backbone for background job processing systems like Sidekiq (Ruby), Celery (Python), and Bull (Node.js).
  • Real-time analytics and dashboards use Redis’s Streams, HyperLogLog, and counters to aggregate and display live metrics with sub-millisecond latency.
  • Distributed locks use Redis’s atomic SET NX (set if not exists) operations to implement mutual exclusion across distributed systems — a pattern formalised by the Redlock algorithm.
  • Geolocation services use Redis’s built-in geospatial capabilities for apps that need to find nearby users, drivers, or points of interest.

Redis vs. Other Technologies

Redis is often compared to Memcached, which is a simpler, purely in-memory cache. Redis offers far richer data structures, persistence, replication, clustering, Pub/Sub, and scripting — making it a more versatile choice for most use cases. Memcached has an edge in raw multi-threaded throughput for simple string caching, but lacks Redis’s breadth.

Against traditional databases like PostgreSQL or MySQL, Redis is not a replacement but a complement. Redis excels at operations requiring microsecond latency on hot data, while relational databases handle complex queries, joins, and large datasets that don’t fit in memory.

Against NoSQL databases like MongoDB or Cassandra, Redis’s in-memory model makes it dramatically faster for small, frequently accessed datasets, but less suitable for large, durable, queryable datasets.

Redis Cluster Internals: How Keys Are Routed, Stored, and Retried

The Fundamental Problem Redis Cluster Solves

A single Redis node is limited by the RAM of one machine. If your dataset is 500 GB and your server has 64 GB of RAM, you cannot store everything on one node. Redis Cluster solves this by partitioning your data across multiple nodes, so each node owns a slice of the keyspace. The challenge is doing this in a way where any client can find any key, even as nodes are added, removed, or fail.

Redis Cluster uses a technique called consistent hashing over hash slots — a deterministic, fast system that maps every key to exactly one node, and every client can independently compute that mapping without talking to a coordinator.

Hash Slots: The Foundation of Key Routing

Redis Cluster divides the entire keyspace into exactly 16,384 hash slots, numbered 0 through 16383. Every key belongs to exactly one slot. Every slot is assigned to exactly one primary node. This is the entire routing model.

Why 16,384?

The number 16,384 (2¹⁴) was chosen deliberately. It is large enough to distribute evenly across hundreds of nodes, small enough that the cluster state (a bitmap of 16,384 bits = 2 KB) can be gossiped between nodes efficiently in heartbeat messages. With a larger number like 65,536, the gossip overhead would be impractical.

Computing a Key’s Slot

The slot for any key is computed as:

slot = CRC16(key) % 16384
Python

CRC16 is a 16-bit cyclic redundancy check.

How Slots Are Distributed Across Nodes

In a typical 3-node cluster, each node owns roughly one-third of the 16,384 slots:

Node A (primary): slots 0     – 5460    (5461 slots)
Node B (primary): slots 5461  – 10922   (5462 slots)
Node C (primary): slots 10923 – 16383   (5461 slots)
Python

This is not fixed — you can assign any distribution. When you CLUSTER ADDSLOTS, you specify exactly which slots each node owns.

Hash Tags: Forcing Keys to the Same Slot

By default, the entire key string is passed through CRC16. This means user:1001:profile and user:1001:sessions will almost certainly land on different slots, making multi-key operations (MGET, MSET, transactions, and Lua scripts that touch multiple keys) impossible across them in cluster mode.

Hash tags solve this. Only the key part is used for slot computation. The rest of the field/values are ignored.

CLUSTER KEYSLOT user:1001:profile    # uses full key → some slot
CLUSTER KEYSLOT {user:1001}.profile  # uses "user:1001" → slot X
CLUSTER KEYSLOT {user:1001}.sessions # uses "user:1001" → slot X (same!)
CLUSTER KEYSLOT {user:1001}.cart     # uses "user:1001" → slot X (same!)
Python

Because all three hash to the same slot, they live on the same node and can participate in the same MULTI/EXEC transaction or Lua script.

The Cluster State: How Nodes Know Everything

Every node in a Redis Cluster maintains a complete map of the entire cluster: which nodes exist, which slots they own, and the health of every node. This is not stored on a coordinator — it is distributed and gossiped.

Gossip Protocol

Redis Cluster uses a gossip protocol where each node periodically sends PING messages to a random subset of other nodes, including its current view of the cluster state. Recipients respond with PONG messages, including their own state. Over time, every node converges to a consistent view of the cluster.

This means:

  • No single point of failure for cluster state
  • New nodes learn the topology within seconds of joining
  • State changes (node failures, slot migrations) propagate cluster-wide within a few hundred milliseconds

The cluster state is stored in each node’s memory and includes a bitmap of all 16,384 slots, the node ID of each slot’s owner, each node’s IP, port, and role (primary/replica), and failure flags for unreachable nodes.

Cluster Bus

Nodes communicate cluster state not on the data port (6379) but on a separate cluster bus port (data port + 10000, so 16379 by default). This separation ensures that cluster gossip traffic never competes with client data traffic. Make sure firewalls allow both ports between cluster nodes.

Client-Side Routing: The Full Journey of a Request

Here is the complete path a client request takes in a Redis Cluster:

Step 1: Client Computes the Slot

Before sending any command, a cluster-aware client computes CRC16(key) % 16384 to determine the target slot.

Step 2: Client Looks Up the Slot’s Node

The client maintains a local slot map — a table mapping each of the 16,384 slots to a node address. This map is built from CLUSTER SLOTS or CLUSTER SHARDS responses and cached locally.

Slot 0–5460    → 10.0.0.1:6379
Slot 5461–10922 → 10.0.0.2:6379
Slot 10923–16383 → 10.0.0.3:6379
Python

Step 3: Client Sends the Command Directly to the Target Node

No proxy, no coordinator. The client opens a direct TCP connection to the node that owns the slot and sends the command.

Client → Node B: GET user:1001:name
         (slot 7686 belongs to Node B)
Python

Step 4: Happy Path — Node Responds

If the node owns the slot and the key exists (or doesn’t), it responds normally. This is the common case and takes one network round-trip.

MOVED: When the Slot Map Is Wrong

The client’s local slot map can become stale. Slot migrations happen when the cluster is rebalanced, nodes are added or removed, or a failover promotes a replica. If the client sends a command to the wrong node, that node responds with a MOVED redirect:

-MOVED 7686 10.0.0.4:6379
Python

This response means: “Slot 7686 is not mine. It belongs to 10.0.0.4:6379. Go there.”

The MOVED response includes the correct node address. A cluster-aware client:

  • Reads the MOVED response
  • Updates its local slot map — slot 7686 now maps to 10.0.0.4:6379
  • Automatically retries the command on the correct node
  • Returns the result to the application (transparently — the app sees no error)
Client → Node B: GET user:1001:name
Node B → Client: -MOVED 7686 10.0.0.4:6379

Client updates slot map: slot 7686 → Node D
Client → Node D: GET user:1001:name
Node D → Client: "Alice"
Python

The application code sees only the final result. The redirect is transparent.

Conclusion

Redis has earned its place as a foundational technology in modern software architecture. Its blend of raw speed, data structure versatility, operational simplicity, and rich ecosystem makes it the go-to solution for caching, real-time processing, messaging, and a dozen other patterns that slow down when pushed through disk-based systems. Whether you’re building a tiny side project or a distributed system serving millions of users, Redis offers tools that can meaningfully improve performance and simplify your architecture. Understanding Redis deeply — its data structures, persistence tradeoffs, clustering model, and design philosophy — is an investment that pays dividends across nearly every domain of backend engineering.

About Puneet Verma

Puneet Verma is a software developer specialising in backend architecture, Dynamic Programming, and SaaS solutions. He focuses on building optimised, scalable applications and sharing deep-dive technical tutorials to help developers master complex algorithmic patterns.

Leave a Comment