Standard Redundant Redis Solution: Master/Slave Replication + Sentinel Manage Failover Expected: a) Client Support + Use Sentinel Discover Current Master หรือ b) TCP Proxy Front Redis Pod Managed By Sentinel Point To Master Former = Redis Sentinel Design Latter = Growing Trend — ObjectRocket Redis Configured This Way
Effects of Failover บน Commands ใน Redis
Vast Majority Redis Operations Works As Expected Failover -> Client Detect Connection Failed -> a) Lookup Current Master Reconnect หรือ b) Reconnect Based On Scenarios Above Proxy-based = Redis Server Restart Applicable Single Instance Redis Restart
แต่ handful Redis Commands = Long-term Blocking หรือ Non-atomic PUBSUB SUBSCRIBE/PSUBSCRIBE = Non-atomic Command Register Request Receive Messages On Channel(s) But Actual Data Arrives Later (After Publish) BL* Commands = Long-term Two-phase Block Until Item Available In List To Remove
Commonality: Issued -> Registration On Server -> Data Comes Later Between Command Issued และ Server Has Data To Reply -> Failovers/Restarts Can Occur Restart/Failover -> "Registration" Gone From Server Question: What Happens To These Commands?
Before Details: Another Aspect -> 2 Types Failovers 1) Loss of Master (Triggered Failover) Master Non-responsive 2) Initiated Failover: Admin Sends Sentinel Failover Command -> Reconfigure Pod Promote Master Demote Original Master Surface Same แต่ Long-block Commands Must Understand Own Context
Key Differentiator: Sequence of Events Triggered Failover: Master Not Responding -> No Data Added/Modified, No Messages Published Initiated Failover: Brief Window 2 Masters
Techniques/Scenarios General + Demo Python "redis-py" Applies Any Client Library Background Ready -> Look Long-block Commands First: PUBSUB
PUBSUB
Redis PUBSUB = 3 Commands: PUBLISH, SUBSCRIBE, PSUBSCRIBE PUBLISH = Send Message SUBSCRIBE/PSUBSCRIBE = Register/Receive Messages Via PUBLISH First: PUBLISH Command
Execute PUBLISH -> Publish Message To Channel Immediately Return Number of Clients Received Failure Scenario Relevance? Failover Case: Client Reconnect TCP Connection Gone Proxy -> Direct Reconnect Different Server Sentinel Discovery -> Longer Still Different Server Issue: Timing Publisher Connects New Server Before Subscribers -> Message Gone Lost But Can That Happen? Look SUBSCRIBE Commands
Code Issues SUBSCRIBE -> Server Register In-memory Data Structures Connection Should Get Messages On Channel(s) Info Not Propagated To Slaves Failure -> Client Reconnect Must Re-issue SUBSCRIBE Otherwise New Server Doesn't Know Connection Needs Messages Clear Potential: Subscriber Reconnect After Publisher Conditions For Message Loss Exist Minimize/Prevent?
Options:
- Publisher Leverage PUBLISH Returns Client Count Received If Zero -> Retry Static Subscribers หรือ "At Least One Subscriber" Sufficient -> Prevent Message Loss
- Control Reconnect Windows Rule: Publisher Delay >= 3x Subscriber Delay 3 Opportunities Subscribers Reconnect Before Publish More Robust Than Uncontrolled But Still Possible Loss
- Lock Mechanism Most Complex Build/Employ Mechanism Prevent Publishers Publish Until Clients Connect Shared Location (Zookeeper, Database, Consul) Subscribers Register Ready Publishers Check Valid/Ready Subscribers Before Publish Complexity: Subscribers "Unset" When Lose TCP Connection
Above For Triggered Failover Works Because Master Non-responsive No Messages Go To It Initiated Failover: Master Accepts Messages Until No Longer Master/Disconnected Above Methods None Fully Catch This Scenario
Initiated Failovers (System Maintenance) Best Buttressing: Employ Options 1+2 Accept Message Loss Amount Depends Publishing Schedule Publish Every Few Minutes -> Minuscule Chance Loss Window Larger Interval -> Smaller Risk Conversely Hundreds/Thousands Messages/Second -> Certain Loss Managed Proxy: Depends Failover Completion Speed ObjectRocket Redis Platform Window Up To 1.5 Seconds
Interval Approach (Fails At Limits): Subscribers Reconnect Interval 1/3 Publishing Interval Publish Once/Minute -> Subscribers Reconnect Every 20s Publisher Every 1-2 Minutes Publishing Interval Approaches 3s -> Harder Failover Reconnect Process (Lookups, TCP Handshake, AUTH, SUBSCRIBE) Easily Few Seconds
BLPOP และเพื่อนๆ
Blocking List Commands Under These Conditions News Better Than Publishing Modify Data Structures = Replicated No Same Risk Lost Messages Due To Reconnect Order Producer Reconnect Prior Consumer -> No Change Expected Behavior Producer PUSH Item Onto List (POP+PUSH) Consumer Connects -> Data There One Exception
Initiated Failover: Brief Multiple-master Problem Few Moments (Milliseconds) Original Master During Initiated Failover Still Accepts Data Modifications Slave Already Promoted -> Modifications Won't Replicate Window Very Small (Single-digit Milliseconds) But Exists Rate Producers/Consumers List Modification Commands Thousands/Second -> Rare But Conceivable
Worker BRPOPLPUSH -> Item Moved "Back" Original Position After Failover BLPOP -> Same Result Item Appears Re-queued Idempotent Jobs -> No Issue Non-idempotent: Defensive Coding Based On Effect Of Job Run Twice + Frequency Modifications + Likelihood Encounter Situation Note: Only Initiated Failover -> Operational Control Advise Maintenance At Least Usage Time Minimize/Eliminate Possibility
Triggered Failover: No Data Loss Expectation Master Unresponsive -> No Modifications Made But TCP Reconnect Handling Mandatory In Any Failover
Client Reconnections
Either Failover: Client Detect Reconnect (Sentinel Lookup หรือ Brief Window Same Address) ObjectRocket Redis Managed Proxy Layer -> Client Simply Reconnect Failover Process Up To 2 Seconds Client Code Account For This Ideal: Immediate Retry Handle Network Blips Connection Dropped Back-off Algorithm With Retries Server Restarts (Standalone Redis หรือ Proxy Restart)
Upon Reconnect Subscribers Re-subscribe (Link Channels <-> New TCP Connection) Redis PUBSUB Channels Created When Subscriber/Publisher Access -> No Need Recreate Channel
How To Do This: Depends Client Library Handle Disconnects Ideal: Connection Configurable Auto-retry Ultimate Failure Returned If Boundaries Met Tested Couple Libraries Varies Dramatically Discuss Common One: redis-py
Good News: redis-py Retries On Connection Drop Bad News: Immediately Retries Too Quick Reliably Recover During Failover No Way Configure Code Must Catch/Detect Failed Reconnect Manage Reconnect Yourself
Standard redis-py Publish/Subscriber Code:
### Publisher example code r.publish(channel,message) ### Subscriber example code p = r.pubsub() p.subscribe(channel) for item in p.listen(): # do stuff with message (item)Straightforward แต่ Immediate Retry Fails During Subscriber's for-loop -> redis.ConnectionError Exception Thrown Tricky: Happens "Inside" for item in p.listen() Wrap Entire For In Try/Except -> Problematic Unnecessary Complexity
Alternative Route:
### Publisher example code p = r.pubsub() p.subscribe(channel) while True: try: message = p.get_message() except redis.ConnectionError: # Do reconnection attempts here such as sleeping and retrying p = r.pubsub() p.subscribe(channel) if message: # do something with the message time.sleep(0.001) # be nice to the system :)Method นี้ Call get_message() Directly Catch Exception At That Point Re-establish 'p' Object Connection Sleep Time Depends Code Requirements For-loop Works Better For Expected Message Count -> Still Works Publishers Simpler (Not Run On Iterator)
### Publisher example code while True: try: rcvd = r.publish(channel,message) if rcvd >0: break except redis.ConnectionError: # handle reconnect attemptsControl Whether/When/How Often Retry Critical Transparently Handle Failure Event Same For Sentinel Direct หรือ Redis Server Restarted All Publishing Code Should Follow This Mechanism
Blocking List Commands (BLPOP): Client Reconnection Order Not Strictly Important Data Persisted Try/Except Method Above Re-establish Connection When Command Execution Throws redis.ConnectionError
redis-py Specifically Against ObjectRocket Redis Platform Techniques + Retry Window 3 Seconds -> No Data Loss Expectation Triggered Failovers ~1.5 Seconds Potential Message Loss Non-stop Publishers During Initiated Failover (Vertical Resize Instance)
Code Examples redis-py Specific แต Basic Techniques ใช้กับ Any Client Code Needs Handle Server Reconnects + Uses PUBSUB/Blocking List Operations Armed With Knowledge -> Implement Techniques Highly Available Redis Pod Application Handle Failovers Without Waking Ops Team Middle Of Night
