# Backend Script for Game Indexing
Make sure you have all you need before proceeding:
- You understand the concepts of CosmJS.
- You have the checkers CosmJS codebase up to the integrated GUI. If not, follow the previous steps or go ahead and clone and checkout this branch (opens new window) to get the version needed for this tutorial.
This exercise assumes that:
You are running your checkers blockchain with:
You have the following in
.env
:
Now that your blockchain is complete, you can think about additional data and services that would add value without increasing cost or complexity on-chain.
For example, how do you list all of a player's games? Currently this information is not easily available. You can find the players of a given game, but not the games of a given player. Indexing this on-chain would add storage and computation costs.
# Server idea
To implement this functionality, build a Web 2.0 server to do the indexing. The server:
- Listens to updates from the checkers chain:
- On a game creation event, adds the game ID under each player.
- On a game deletion event, removes it.
- When asked about its status, returns the latest block height up to which it has indexed players.
- When asked about a given player, returns the list of game IDs for this player.
- When a game ID is submitted, patches its information about that game ID on a best effort basis (to palliate potential de-synchronization).
# Barebones server
As a fast and simple Web 2.0 solution, navigate to the checkers CosmJS repository (opens new window) for the checkers blockchain, in the existing gui
branch, and perform the following steps:
- Create a sub-directory of the
src
folder (e.g.server
). - Use the
express
Node.js module to create an HTTP REST API. - Use a local
db.json
as a database. This is obviously primitive and not thread-safe. In a production setting use a proper database. - Poll the blockchain at regular intervals. As part of an advanced topic, you can use WebSockets.
# Data types
To keep the code type-safe, define the types of your db.json
in types.ts
:
Not only does this keep information about players, it also keeps a copy of games. This gets around a current limitation of CosmJS, where you cannot get information about a game that has just been erased from the latest state. In practice you would need to query the game at an earlier block height, but this functionality is not yet available with CosmJS. Note that nodes may prune the old state, especially if they migrate, which may unpredictably impact any query at an earlier block height.
State storage is the most expensive resource a blockchain application uses. Wasteful use of storage burdens nodes with unnecessarily large storage requirements.
From an engineering perspective, it is important to separate strict protocol needs from overall requirements. If no state-changing, on-chain logic relies on a stored value, then by definition that value's usefulness is limited to reporting. Such a value is a candidate for deletion from the blockchain state. Reporting is important, but consider more efficient methods of addressing it.
An off-chain indexer - think block explorers - can record historic facts before they are purged. This will support queries about details that are no longer present on the chain. This can also support queries about details that are still present on the chain, but at a much cheaper cost per query than querying blockchain nodes.
Preventing duplication of unique identifiers is a common requirement. The rule can be enforced even if the details are purged, as long as you keep a list of used keys or observed hashes that must not be used again. For example, keep IDs for games that used to exist while deleting the details about the moves and results that have no further relevance to the protocol because the games are "over". If needed, you can keep the complete move-by-move history of the game in cheaper, off-chain storage. This is possible if you designed your chain with proper reporting mechanisms, for instance via events.
In most cases, the most elegant code proceeds on the basis that referential integrity is guaranteed, avoiding messy exceptions like orphaned keys. Internal referential integrity is entirely the responsibility of the application developer, so consider using techniques like cascade deletions or preventing deletion if a record is referenced elsewhere.
Generally, it is a good idea to use deletes to prevent the state from simply growing forever. A simple rule of thumb is to delete everything you can, but no more.
# Empty indexer module
Define a barebones server without any Cosmos elements in an indexer.ts
. This is not CosmJS related, so start from something else if you prefer.
Note:
- The timer is set at the end of the previous poll, in case indexing takes longer than the interval.
- The database is, for now, purely in memory as it runs and is saved on exit by catching the interruption signal.
# Files for execution
Prepare these files around the indexer
to run it in the terminal:
Create an
index.ts
that describes how to run theindexer
:The
export {}
prevents Visual Studio Code from complaining.Add a specific
tsconfig.json
file if necessary:In
package.json
, add arun
target:Add your database,
db.json
by making a copy of the sample:You cannot ask for block at height
0
, so the indexer first asks for the next block at1
.
# Quick test
Check that the indexer
works:
It should print:
Now, in another terminal, test the endpoints. Omit the | jq
beautifier if it is not installed on your system:
# Add CosmJS StargateClient
You need to create a client to connect to your checkers blockchain. The client only needs read-only functionality because this server does not submit transactions. Your repository already contains useful elements:
- The
CheckersStargateClient
. - An
.env
(opens new window) file.
Add the following to indexer.ts
:
The declarations:
The pickup of
RPC_URL
:The client in the indexer:
The modified
init
:The modified
poll
:
If you have not done it yet, start your checkers chain as described at the beginning of this section.
Relaunch the indexer:
You should see the current height rising:
# Handle blocks
To index games, take each block and listen for the following relevant events:
- A transaction with a
new-game-created
event. - An
EndBlock
with agame-forfeited
event.
Start by getting each block from your last saved state. Update poll
:
This needs a new import:
The indexer:
- Declares a new function
handleBlock
. Create one and putconsole.log(block)
inside to explore what this object is and consider what actions you would take with it. - Saves the
db
after a poll, so you can watch it in real time. - Uses
process.stdout.write
andprocess.stdout.cursorTo(0)
so that the repetitive logging happens on a single line.
Observe the relevant content in handleBlock
. It must:
- Extract the events from transactions.
- Extract the events from
EndBlock
.
If you examine block.txs
directly you find transactions as they were posted. However, this does not reveal any execution results, such as if a transaction executed as expected or what game ID it used for the new game. To get this extra information:
- Calculate
txHash
from the transaction. - Call
await client.getTx(txHash)
, which returns anIndexedTx
.
The handleBlock
function can be:
This needs new imports:
while() { await }
simplifies the syntax ofawait
ing multiple times sequentially.- The hash is calculated this way as per here (opens new window).
console.log("")
puts a new line (poll
does aprocess.stdout.write
which adds no line).- The
handleBlock
function uses a new function,handleTx
. Create one and putconsole.log(indexed)
inside to explore what this object is and consider what actions you can take with it. - The
EndBlock
part has not yet been incorporated. This is explained in Prepare for EndBlock.
# Handle a transaction
Define the handleTx
function:
This needs new imports:
.flatMap
(opens new window) transforms an array of arrays into a flattened array.- The
handleTx
function uses a new function,handleEvents
. Create one and putconsole.log(events)
in it to explore what this object is and consider what actions you can take with it.
# Handle events
Define the handleEvents
function:
while() {}
simplifies the syntax ofawait
ing multiple times sequentially.- The
handleEvents
function only keeps events that emanate from thecheckers
module. - It uses a new function,
handleEvent
. Create one and putconsole.log(event)
inside to explore what this object is and consider what actions you can take with it. - It skips in case of error, as the likely cause is that the transactions in fact failed. This is not good enough if the goal is to be absolutely accurate.
# Handle one event
Define handleEvent
as follows:
new-game-created
(opens new window) andmove-played
(opens new window) are constant values defined in your Go code. They are put as the event's type (opens new window).handleEvent
uses two new functions:handleEventCreate
andhandleEventPlay
. Create them and putconsole.log(event)
inside each to explore what these objects are and consider what actions you would take with them.
# Handle one create event
Now update your db
with the information provided. First, define a convenience function in createIndexer
:
This needs a new import:
Now define handleEventCreate
as:
game-index
(opens new window),black
(opens new window), andred
(opens new window) are constants from the Go code.- You have implemented error handling.
handleEventCreate
is careful not to double-add a given game ID.- It does not save
db
as this is under the purview ofpoll()
.
# Handle one play event
Not all play events are equal. handleEventPlay
is only interested when there is a winner. Until then, there is no need to take any action.
handleEventPlay
returns quietly if there is no winner, because this means there is nothing to do.- It keeps the game information in the
db
. - It removes the id from both players' list of games.
# Test time
You can now test what happens when a game is created and played on. Restart npm run indexer-dev
locally or in Docker.
You can choose how to create and play games:
- Run the GUI prepared in the previous section with
npm start
. - Run
checkersd
command lines. - Any other way available.
Via command lines, in another terminal:
What remains is handling the games that get removed or forfeited in EndBlock
.
# Prepare for EndBlock
Nicely formatted EndBlock
events are still missing from CosmJS, so these require a little extra work:
- To get a block's
EndBlock
events, you need to ask for the block information from a CometBFT client. This client is aprivate
field (opens new window) ofStargateClient
. - The function to call is
blockResults
(opens new window). - It returns a
BlockResultsResponse
(opens new window), of whichendBlockEvents: Event
is of interest. - This
Event
(opens new window) type hasattributes: Attribute[]
of interest. - The
Attribute
(opens new window) type is coded asUint8Array
.
With this information, you can do the necessary actions:
To handle the conversion of CometBFT
Event
s intoStringEvent
s, create a helper in a newsrc/server/events.ts
:To handle the call to
blockResults
, you need access to a CometBFT client. One option is to make a copy of the private CometBFT client. You can do this only on construction, so create a child class ofCheckersStargateClient
to do that. It is recommended to keep it close byindexer.ts
. In a newindexer_stargateclient.ts
:
Now swap out CheckersStargateClient
with IndexerStargateClient
:
With this in place, go back to handleBlock
and work on the remaining TODO.
# Handle one block's EndBlock
Go to the function and update it:
The events that you have converted are compatible with those emanating from transactions, so you can just pass them on. You still need to update handleEvent
so that it acts on the new event type:
To achieve this, add a new function:
Again there is a lot of error handling. handleEvent
only soft-deletes the game, although it removes it from the list of games for the players.
# Test time of forfeit
Run the previous tests again. Create a game and see how the deletion event is picked up:
In the standalone checkers in Docker, the deadline is unfortunately set at 24 hours, so feedback is not exactly coming fast. At this state of the exercise, if you want to test the expiry quickly, you will have to run Ignite CLI and adjust the MaxTurnDuration
as described here.
# Patch a game
In the actions that the Express server exposes, app.patch
still remains to be implemented. This allows a user to inform the server that its database is no longer synchronized, and that it should look at a specific game. It is a matter of data re-synchronization:
- If the game can be found in the blockchain state, update the indexer's database accordingly:
- If there is a winner, then the game should be removed from its players' lists of games.
- If there is no winner, then the game should be added to its players' lists of games.
- If the game cannot be found in the blockchain state, but is present in the indexer's database, then the game should be removed from the lists of games of its players, and marked as soft-deleted. This shows the usefulness of keeping old games.
- If the game cannot be found either in the blockchain state nor in the indexer's database, then it is better not to do anything. To remove it from all players' lists of games is potentially expensive. This could expose the server to a DoS attack.
Code the following:
There are some issues to be aware of:
- JavaScript is not thread-safe, so you could cause two opposite actions: one coming from the polling and the other from a patch submission, or even from two concurrent patch submissions. To reduce this risk the database is not saved to disk in this function, but instead relies on the polling to save it at the next run.
- Assuming that there is no such game when you cannot find it can result in deleting data that is simply taking time to appear on your blockchain node.
Next, you need to call patchGame
from the app.patch
callback:
# Test time of patch
To simulate a case where the game is in the blockchain state but not the indexer's:
Stop your indexer.
Create a game and check at what block it is included (for example, at index
3
and block1001
).Update your indexer's
db.json
to pretend that it already indexed the game's block by setting:Restart the indexer.
From another terminal, make a call to it:
It should return:
And the indexer should log something like:
Develop your own ways to test the other scenarios.
If you started the chain in Docker, when you are done you can stop the containers with:
# Conclusion
You have created a small server that:
- Polls the blockchain to get events about created, won, and forfeited games.
- Maintains a database with information indexed in real time.
- Offers this information as a Web service.
- Accepts requests for patches.
These are examples of server-side scripts, which can improve user experience.
You can find the complete code here (opens new window).