Optimistic Concurrency Control: High-Throughput Transaction Processing Without Lock Overhead

In relational database engines, pessimistic locking mechanisms (such as Two-Phase Locking / 2PL) acquire shared and exclusive locks before accessing data items. While 2PL guarantees conflict serializability, managing lock tables, detecting deadlocks, and holding locks across long execution windows incurs substantial CPU cache overhead and blocks concurrent transactions.

Introduced by H. T. Kung and John T. Robinson in 1981, **Optimistic Concurrency Control (OCC)** operates on the premise that in many database workloads, conflicts between concurrent transactions are rare. Rather than acquiring locks upfront, transactions execute freely in private workspaces and only validate their operations against concurrent transactions at commit time, committing if no conflicts occurred and aborting/restarting if conflicts are detected.

The Three Lifecycle Phases of OCC

Every transaction under classical OCC executes through three sequential, non-overlapping phases:

1. Read / Execution Phase

Data Access: The transaction reads values from the shared database state without acquiring any shared (read) locks.
Private Workspace: All write operations (inserts, updates, deletes) are buffered strictly in the transaction's private local write buffer without modifying the shared database state.
Set Tracking: The engine tracks all accessed records in two collections: the **Read Set** (keys/tuples read) and the **Write Set** (keys/tuples modified and their new values).

2. Validation Phase

When the transaction requests to commit, the engine validates whether executing the transaction violates serializability with respect to all concurrent transactions that ran or committed during its lifetime. If validation fails, the private write buffer is discarded and the transaction aborts and restarts.

3. Write Phase

If validation succeeds, the transaction atomically applies all buffered changes from its private Write Set into the shared database storage and makes them globally visible to subsequent transactions.

Validation Strategies: Backward vs. Forward Validation

The core correctness mechanism of OCC depends on how the engine checks for intersecting Read and Write sets during validation:

1. Backward Validation (Validating Against Past Transactions)

When Transaction $T_i$ enters validation, it checks whether its own **Read Set** intersects with the **Write Set** of any transaction $T_j$ that committed after $T_i$ started:

If an intersection exists, $T_i$ read stale data that was overwritten before $T_i$ could commit, forcing $T_i$ to abort.

2. Forward Validation (Validating Against Active Future Transactions)

When Transaction $T_i$ enters validation, it checks whether its own **Write Set** intersects with the **Read Set** of any currently active, uncommitted transaction $T_k$:

Advantage: Because $T_k$ has not committed yet, the engine has flexibility: it can either abort $T_i$, abort $T_k$, or defer $T_i$'s commit until $T_k$ finishes.

OCC vs. 2PL: Workload & Contention Dynamics

The choice between Optimistic and Pessimistic concurrency control depends on transaction duration and write collision frequency:

Low Contention Workloads: OCC significantly outperforms 2PL because transactions execute with zero locking overhead, zero latch contention, and no deadlock detection sweeps.
High Contention / Hotspot Rows: In workloads with frequent concurrent writes to the same keys, OCC suffers from high abort rates ('validation thrashing'), wasting CPU cycles on transactions that repeatedly compute and abort.
Long-Running Transactions: Long-running read-write transactions under OCC have a wide validation window, making them likely to intersect with shorter committed transactions and fail validation repeatedly.

C++ Conceptual Simulation Blueprint (Backward Validation Engine)

#include <iostream>
#include <vector>
#include <unordered_map>
#include <unordered_set>
#include <algorithm>

struct CommittedTxn {
    uint64_t commitTime;
    std::unordered_set<std::string> writeSet;
};

class OCCTransaction {
public:
    uint64_t startTime;
    std::unordered_set<std::string> readSet;
    std::unordered_map<std::string, std::string> writeBuffer;

    OCCTransaction(uint64_t start) : startTime(start) {}

    void read(const std::string& key) {
        readSet.insert(key);
    }

    void write(const std::string& key, const std::string& val) {
        writeBuffer[key] = val;
    }
};

class OCCEngine {
private:
    uint64_t logicalClock = 0;
    std::unordered_map<std::string, std::string> globalStorage;
    std::vector<CommittedTxn> commitHistory;

public:
    uint64_t getClock() const {
        return logicalClock;
    }

    bool validateAndCommit(OCCTransaction& txn) {
        // 1. Backward Validation Phase
        // Check against all transactions that committed after this txn started
        for (const auto& pastTxn : commitHistory) {
            if (pastTxn.commitTime > txn.startTime) {
                for (const auto& readKey : txn.readSet) {
                    if (pastTxn.writeSet.count(readKey)) {
                        return false; // Conflict detected: Read Set intersected committed Write Set
                    }
                }
            }
        }

        // 2. Write Phase
        logicalClock++;
        CommittedTxn committedRecord;
        committedRecord.commitTime = logicalClock;

        for (const auto& [k, v] : txn.writeBuffer) {
            globalStorage[k] = v;
            committedRecord.writeSet.insert(k);
        }
        commitHistory.push_back(committedRecord);

        return true; // Successfully committed
    }
};

Real-World Modern Database Implementations

Silo & Masstree (In-Memory Transaction Engines): Multicore in-memory transactional database architectures implementing epoch-based OCC and short-phase commit validation to scale to millions of serializable transactions per second.
TicToc (Timestamp-Based OCC): Dynamically computes data item valid time ranges during validation to achieve serializability without rigid physical epoch locks.
Google Cloud Firestore & Document Stores: Exposing optimistic conditional mutations (`commit()` with precondition tokens) to validate mobile and web document writes without keeping open server-side database connections.
Software Transactional Memory (STM): Applying OCC principles inside multi-threaded runtime environments (such as Haskell, Clojure, and C++ STM libraries) to synchronize memory access without developer-managed mutexes.