This project is a distributed transaction management service. The goal is to replicate transactions over various number of nodes, keep the data consistant, and tolerate node failures. To achive these goals, we are going to use PAXOS consensus protocl. However, we are going to modify this protocol to handle multiple values in a single instance.
Clients send their transactions to servers (there is user-level sharding that map each client's account to one single machine). Servers store them inside a local storage as a block of transactions. After that, they run each transaction.
They are going to run apaxos consensus protocol when a client sends a request that has an amount more than the current balance of that client. By running this protocol, a leader node collects all other node's transactions and forms a huge block of transactions (or a list of transaction blocks). Then it replicates them over other servers.
Finally, as the majority of servers get that major block, each server starts running these transactions and stores them in a persistante storage (in our case its MongoDB).
In this section, there is a list of the system components:
gRPCserver- gets requests from both other nodes and clients
- contains a
apaxosserver,transactionsserver, andlivenessserver - it uses a consensus module to run
apaxosinstances
consensus- running an apaxos instance:
- sends prepare requests with a ballot number
- waits for the majority/a timeout perioud
- collects all transactions to create a major block
- sends accept requests with a major block
- waits for the majority/a timeout perioud
- sends a commit message
- handling input requests:
- gets propose requests and compare ballot number with its own ballot number
- returns the promise with accepted num and accepted val or its own transactions
- gets the commit request, clears the block list and executes the transactions
- running an apaxos instance:
database- connects to a
MongoDBcluster
- connects to a
memory- uses local memory of the node to keep data
worker- runs backup tasks to keep track of node's states
- new transaction file
- new transaction
- print balance (X)
- print logs
- print db
- performance (latency, throughput)
- aggregated balance (X)
- Block
- List of transactions (array of transactions)
- Ballot Number
- Transaction
- Sender
- Reciever
- Amount
- Sequence number
- Major Block (Block List)
- Ballot Number
- List of Blocks (array of blocks) ordered by their Ballot Number
- Ballot Number
- Contains a
Nnumber andIDof server
- Contains a
- Propose, Promise, and Sync
- Accept, and Accepted
- Commit
A proposer sends this request with a ballot number. Contains it's last committed Block info (UID).
An acceptor checks the propose ballot number with it's accepted num. If ballot number is greater than accepted num, it will return a promise response with it's accepted val or block list.
After collecting all logs from acceptors (waits for the majority). The proposer, creates a Major Block, or selects an accepted val with highest ballot number. Then it sends an accept request with its own ballot number.
Each acceptor checks its accepted num and accepted val with the given accept request. If it is ok it will update it's accepted val and accepted num.
Finally, the proposer waits for the majority. If it get's enough accepted responses, it will send a commit message. After getting the commit message, each node clears it's block list by comparing it to the accepted val. It will also store the accepted val and removes it to accept future messages.
During the propose process, or the accept process. If a server sends an old commit block in return, the propose sends a sync request and the list of blocks that where stored after that block. So, the node will be synced.
- Programming language:
Golang 1.23 - Communication:
gRPC v2,protoc3 - Database:
MongoDB - Logging:
zap logger - Configs:
koanf
All nodes need a .yaml config file. This file stores the required data that is needed to start up a node. An example config file is like this:
# config file for node instances
# list of nodes and their addresses
nodes:
- key: "S1"
value: "localhost:5001"
# list of clients and initial balances
clients:
- key: "S1"
value: "100"
# node information
node_id: "S1"
client: "S1"
workers_enabled: false
workers_interval: 5 # in seconds
log_level: "debug"
# consensus and gRPC values
majority: 3
grpc:
host: "localhost"
port: 5001
request_timeout: 1000 # in milliseconds
majority_timeout: 10000 # in microsends
# database configs
mongodb: # username amirhnajafiz password d3tO2kzZCse5f509
uri: ""
database: "apaxos"After compiling the application by running make compile, you can run ./main node config_1.yaml to start a node.
Now to communicate with the system, you can use the default controller. Again, you needa config file like this:
# config file for controller app
# list of nodes and their addresses
nodes:
- key: "S1"
value: "localhost:5001"
# list of clients and initial balances
clients:
- key: "S1"
value: "100"
# clients shards
clients_shards:
- key: "S1"
value: "S1"Then you can run ./main controller config-ctl.yaml and use the cli:
$ help
exit: close the controller app
help | prints help instructions
tests <csv path> | loads a csv test file
next | runs the next test-set
reset | reset all servers status to active
block <node> | get a node out of access
unblock <node> | restore a single node
ping <node> | send a ping message to a node
printbalance <client> | print the balance of a client (based on shards)
printlogs <node> | print logs of a node
printdb <node> | print database of a node
performance | gets performance of all nodes
aggrigated <client> | gets a clients balance in all servers
transaction <sender> <receiver> <amount> | make a transaction for a client