EVM on Flow

Node setup guide for Flow Testnet/Mainnet Axelar validator

EVM on Flow integrates Ethereum’s transaction execution core into Cadence’s native transaction runtime and thus Flow’s multi node consensus. Access Nodes (AN) serve the Access API, the native entry point to Flow, accepting Cadence scripts or transactions, into which signed transactions from the EVM Gateway are included. The EVM Gateway implements the Ethereum JSON-RPC specification for web3.js clients. In the Flow context these two services together represent an Axelar validator. Access Nodes, and by extension EVM Gateways, do not vote on the network - they are rpc-only node types.

Operators can choose to run the Access Node and EVM Gateway separately or on the same logical host. Consider that Access Nodes are optimized for high throughput script/transaction ingress for Flow clients. Node operators which choose to make AN Access APIs public should use the recommended hardware requirements. If the validator’s sole purpose is to participate in the Axelar network, recommended hardware requirements may be excess to operational needs. Operators may opt to run only EVM Gateway if using public Access Nodes.

The steps first walk through the process of setting up an access node and then setting up the EVM gateway node.

  1. Generate access node information
  2. Account creation
  3. Stake the access node
  4. Build access node binary
  5. Download access node bootstrap files
  6. Create access node service
  1. Build EVM Gateway binary
  2. Create EVM Gateway service
  3. Start all services

We recommend first time operators to complete the guidance for node setup below in full using testnet.

This is a one time operation that is required for each node.

Terminal window
# get the boot-tools utility to generate node information
curl -sL -O storage.googleapis.com/flow-genesis-bootstrap/boot-tools.tar
# untar it
tar -xvf boot-tools.tar
# verify sha256sum: a06e3e9b2443c6755214150e9e101b70dd48ae30ffcfcbbcc471ba430cb104bf
sha256sum ./boot-tools/bootstrap
# create a directory
mkdir ./bootstrap
# Use a fully qualified domain name for YOUR_NODE_ADDRESS_GOES_HERE. Please also include the port number in the network address e.g. flowaccess.mycompany.com:3569
./boot-tools/bootstrap key --address "<YOUR_NODE_ADDRESS_GOES_HERE>:3569" --role access -o ./bootstrap

💡

The network address is the address that will be used by the other nodes in the network to talk to your node.

Please ensure that the address (hostname:port) that you use is accessible from outside and there is no firewall rule blocking egress and ingress traffic for that address.

This will generate the node information which includes public and private staking keys, public and private networking key and the node ID.

Terminal window
cat ./bootstrap/public-root-information/node-info.pub.[HASH].json
{
"Role": "access",
"Address": "\u003cYOUR_NODE_ADDRESS\u003e:3569",
"NodeID": "c2b79d2d4384b30ac6db6ee284d0549bf3a19f6885232e4a20af438c49e15948",
"Weight": 0,
"NetworkPubKey": "f99ba93360c6daa3b044c5a474d915f9306d450ed3a996440c378cd9e37e37cd6d57c13f9417087af148f25a216ea7b239bcadb7ce4dd72f90ad26c9ea1db03a",
"StakingPubKey": "8b027c8afbb01d63e93ee40242ab2aad03f25800e3a3c01c3615520505ab23626c016d4e7525f3bebdfe301188275ec401b403f4825a1511d22b7bcdaa8b07936933c5cae3c9880a1bf218fa3a8c21b10023b667ac9d23809b7eaa152b059dfe"
}
Terminal window
cat ./bootstrap/private-root-information/private-node-info_[HASH]/node-info.priv.json


{
  "Role": "access",
  "Address": "\u003cYOUR_NODE_ADDRESS\u003e:3569",
  "NodeID": "c2b79d2d4384b30ac6db6ee284d0549bf3a19f6885232e4a20af438c49e15948",
  "NetworkPrivKey": "aba48858482ead4a1bdeee198b8948061fa80c9720b0107024ef7d5793dbc930",
  "StakingPrivKey": "061d9bef99db63629ed28bb64ad0cc6b93418accc1a6c5e461dfcbdf087c2862"
}

Please ensure you backup the bootstrap folder as it contains the node private keys. If they keys are lost, the node will not be able to run.

For further information on this step see official documentation.

Many operators will only require the first two steps below if aggregating operational costs across Flow into a single account is acceptable, or where only a single logical host is being run for all services. Organizations who must separate operating costs between node types must include step (3) to create a distinct account for the EVM Gateway.

If you are only running the EVM Gateway and using public Access Nodes the first two account creation steps below are still required. In this case the account will only be used for the EVM Gateway.

Accounts need to be created on Flow before they can be funded and used. Visit Flow Port and select mainnet or testnet in the dropdown and click ‘Sign Up’. Flow Port will guide you through Flow Wallet download, which once initialized will have created an account for you on the selected network. You should now be able to ‘Sign in’ to Flow Port and can copy your account address from the wallet or Flow Port dashboard.

Note: to enable Flow testnet network in Flow Wallet you will need to toggle 

Settings -> Developer Mode -> Enabled

Use a supported centralized exchange to send FLOW to the account address you copied

Check your account with the network directly using Flow CLI

Terminal window
# install Flow CLI
sh -ci "$(curl -fsSL https://raw.githubusercontent.com/onflow/flow-cli/master/install.sh)"
echo 'export PATH="$PATH:/root/.local/bin"' >> ~/.profile
source ~/.profile

Paste your account address and specify network

Terminal window
% flow accounts get 0x1844efeb3fef2542  -n testnet


If you would like to fund the account with 1000 FLOW tokens for testing, visit https://testnet-faucet.onflow.org/fund-account?address=1844efeb3fef2542


Address   0x1844efeb3fef2542
Balance   744.00060373
Keys   2


Key 0  Public Key     3569d7acbc44cd61ffcca019820d4975e7c32bc0d0da38d3920e5ae7fac172bedc5905ddcf349e9ea6968bd7b1da33178c03dc45a73117d6b59c1fef79651cbe
Weight       1000
Signature Algorithm   ECDSA_secp256k1
Hash Algorithm     SHA2_256
Revoked      false
Sequence Number    7
Index        0
Contracts Deployed: 0

Use the existing operator account to create new accounts.

Terminal window
flow keys generate

It will generate an output like this. These keys are only output locally to the CLI and are used in the following step

Terminal window
🔴️ Store private key safely and don't share with anyone!
Private Key              5438b13e290b257f0c41e3b23fbf3b5f5f6d4e9b2d3797430d2d2d5484db9b37
Public Key               9b85626369efe380ebd701f7189f2746fda26d09e1b63cb003bd84a5d33b662685c56427a0526a670efa88b0cb8da371cf9dc1de5ac6bbffb56e327a5c16e708
Mnemonic                 sustain limb elbow awkward onion crouch truth trial until poverty famous feel
Derivation Path          m/44'/539'/0'/0/0
Signature Algorithm      ECDSA_P256

Create EVM Gateway operator account

  1. Obtain wallet account private key from 

    Settings -> Account List -> Choose Main account -> Private Key -> [Password prompt]

  2. Create flow.json, replacing address and privateKey values with those of your wallet account. Note address has no leading 0x

Terminal window
flow init --config-only
cat flow.json | jq '.accounts += {"my-account-alias": {"address": "$YOUR_ADDRESS", "key": {"type": "hex", "index": 0, "signatureAlgorithm": "ECDSA_P256", "hashAlgorithm": "SHA2_256", "privateKey": "$YOUR_PK"}}}' > updated_flow.json ; mv updated_flow.json flow.json
  1. Create a new account using public key generated above
Terminal window
flow accounts create --key 9b85626369efe380ebd701f7189f2746fda26d09e1b63cb003bd84a5d33b662685c56427a0526a670efa88b0cb8da371cf9dc1de5ac6bbffb56e327a5c16e708 --signer my-account-alias -n testnet


Transaction ID: efcacf408285100971f93053e28cf0d2fe0809213eb35c85d779c2fc2bca89c7


If you would like to fund the account with 1000 FLOW tokens for testing, visit https://testnet-faucet.onflow.org/fund-account?address=f3445838372e4e92


Address   0xf3445838372e4e92
Balance   0.00100000
Keys   1


Key 0  Public Key     9b85626369efe380ebd701f7189f2746fda26d09e1b63cb003bd84a5d33b662685c56427a0526a670efa88b0cb8da371cf9dc1de5ac6bbffb56e327a5c16e708
Weight       1000
Signature Algorithm   ECDSA_P256
Hash Algorithm     SHA3_256
Revoked      false
Sequence Number    0
Index        0


Contracts Deployed: 0
  1. Send funds to this account from your wallet account

Access Nodes require a minimum 100 FLOW stake for each participating node to secure a slot on the network. Follow this guide to stake using Flow Port.

💡

Use the public key values previously generated by the bootstrap utility when staking your node

When staking for the first time please select ‘Upgrade Account’ in Flow Port when requested

Once completed, your Flow Port ‘Stake and Delegate’ dashboard will display the new node registration stake in a pending status. Note the banner which outlines when your staking action will be included in the next epoch.

flow-port-stake-pending.png

Further information about staking on Flow can be found in the Staking and Epochs documentation.

On Wednesday at around 12:00 UTC, the staking auction for the current epoch will end and five nodes from candidate list of nodes will be chosen at random by the staking contract for participation in the next epoch. If all 5 slots have been taken from the previous epoch no new Access Nodes will be chosen

You can verify whether your node was chosen in the Flow Port Staking Dashboard. The ‘Staked Amount’ for your node will equal the amount staked for that node ID once it has been selected.

Using Flow CLI you can query the staking info for your node:

Terminal window
flow accounts staking-info your_account_address -n mainnet

Look for the ‘Tokens Staked’ field in the response of the above command to verify.

In the event that the epoch passes and the node you registered did not get picked your stake will be returned. You can re-stake the node using the same staking procedure as above.

Build the access node binary from the latest release tag. The access node source is available in the onflow/flow-go repo.

  1. Install the prerequisites mentioned here
  2. Checkout the latest release tag mentioned here.
  3. Build the access node binary by following this step in the flow-go repo. (eg make docker-native-build-access-binary)

Alternatively, you can also run the access node and the EVM gateway as a docker container. See here for more details.

After the access node has been successfully staked, it requires two pieces of information to startup.

  1. An initialization file called the root-protocol-state-snapshot.json to initialize its local database.
  2. A checkpoint of the execution state which contains the chain state (account balances, contracts etc.). The checkpoint consists of 18 files named as root.checkpoint, root.checkpoint.000, root.checkpoint.001…root.checkpoint.016.

The root-protocol-state-snapshot.json and the checkpoint files are published by the Foundation to a GCP bucket. Location of the bucket is available here under rootProtocolStateSnapshot and rootCheckpointFile for each of the past network upgrades. For e.g. For mainnet25 the files are here.

The transit script which is part of the boot-tools utility downloaded earlier during first step can be used to download these files and place them in the right location.

To download the files run when testnet/mainnet-${version-label} should be set to the latest version of testnet or mainnet, e.g. testnet-52, mainnet-26 etc.

The role flag (-r) has to be execution to ensure the checkpoint files are placed in the correct location.

Terminal window
./boot-tools/transit pull -b ./bootstrap -t mainnet-${version-label} -r execution --concurrency 10 --timeout 30m

Example for network the upgrade mainnet-26 the command will be as follows,

Terminal window
./boot-tools/transit pull -b ./bootstrap -t mainnet-26-execution -r execution --concurrency 10 --timeout 30m

For more information on the steps to carry out after a network upgrade, please see here.

💡

If the node was already running on the previous network before the upgrade then please follow these steps to sync the node on the upgraded network.

💡

If you are joining the network on any epoch after the last network upgrade, then please see here for instructions on downloading the bootstrap files.

Use systemctl to set up services for access-node.

Terminal window
# Define the variables
NODE_ID=$(cat ./bootstrap/public-root-information/node-info.pub.* | jq -r .NodeID)
DYNAMIC_ACCESS_ADDRESS="secure.mainnet.nodes.onflow.org:9001"
DYNAMIC_ACCESS_PUBLICKEY="28a0d9edd0de3f15866dfe4aea1560c4504fe313fc6ca3f63a63e4f98d0e295144692a58ebe7f7894349198613f65b2d960abf99ec2625e247b1c78ba5bf2eae"
Terminal window
sudo tee <<EOF >/dev/null /etc/systemd/system/access-node.service
[Unit]
Description=Access Node daemon
After=network-online.target


[Service]
User=$USER
ExecStart=/usr/bin/access-node \
--nodeid=$NODE_ID \
--bootstrapdir=$PWD/bootstrap \
--datadir=$PWD/data/protocol \
--secretsdir=$PWD/data/secrets \
--execution-data-dir=$PWD/data/execution_data \
--execution-state-checkpoint=$PWD/bootstrap/execution-state \
--rpc-addr=0.0.0.0:9000 \
--secure-rpc-addr=0.0.0.0:9001 \
--http-addr=0.0.0.0:8000 \
--rest-addr=0.0.0.0:8070 \
--admin-addr=localhost:9002 \
--bind=0.0.0.0:3569 \
--dht-enabled=false \
--grpc-compressor=gzip \
--execution-state-dir=<path-to-execution-state-dir> \
--script-execution-mode=failover \
--event-query-mode=failover \
--tx-result-query-mode=failover \
--register-cache-size=10000 \
--state-stream-addr=0.0.0.0:9000 \
--state-stream-response-limit=500.0 \
--state-stream-heartbeat-interval=50 \
--loglevel=error
Restart=always
RestartSec=3
LimitNOFILE=4096


[Install]
WantedBy=multi-user.target
EOF


cat /etc/systemd/system/access-node.service
sudo systemctl enable access-node

The following Access Node log message indicates that the node has not yet been selected for participation in the network. If your node is staked it will hopefully be selected at the next epoch. Check the staking status of your node.

Terminal window
node identity not found in the identity list of the finalized state

Checking the sync status of your Access node

Terminal window
curl localhost:8080/metrics | grep consensus_compliance_sealed_height

Further details on Access Node setup can be found in the Access Node setup documentation.

Similar to the access node, build the EVM gateway binary from the latest release tag. The EVM gateway source is available in the onflow/flow-go repo.

Terminal window
git clone https://github.com/onflow/flow-evm-gateway.git
cd flow-evm-gateway
git checkout <latest release tag mentioned here https://github.com/onflow/flow-evm-gateway/releases/latest>
make build
mv flow-evm-gateway /usr/bin/

Further details on EVM Gateway building and configuration can be found in the EVM Gateway node ops docs.

The Flow operator account must initialize its COA address in EVM which is then configured as the EVM Gateway $COINBASE address. The guidance provided for wallet account below is a GUI driven version of the CLI transaction. CLI guidance can be used on any account and network.

  • Click top left burger icon to show current profile
  • Click ‘Enable the path to EVM on Flow’ button
  • Your EVM account will now be available to use in the left nav account view

EVM operator address and private key is configured for --coa-address & --coa-key EVM Gateway flags below. If running multiple EVM Gateway hosts it is standard to share the same COA address and key for all.

Terminal window
ACCESS_NODE_GRPC_HOST="${YOUR_NODE_ADDRESS}:9000"
ACCESS_NODE_SPORK_HOSTS="access-001.mainnet25.nodes.onflow.org:9000"
FLOW_NETWORK_ID="flow-mainnet"
INIT_CADENCE_HEIGHT="85981135"
COINBASE="${LINKED_EVM_ADDRESS_WITHOUT_0x}"
COA_ADDRESS="${FLOW_OPERATOR_ACCOUNT_ADDRESS}"
COA_KEY="${FLOW_OPERATOR_ACCOUNT_PRIVATE_KEY}"
GAS_PRICE="100"
Terminal window
sudo tee <<EOF >/dev/null /etc/systemd/system/gateway.service
[Unit]
Description=Gateway daemon
After=network-online.target


[Service]
User=$USER
ExecStart=/usr/bin/flow-evm-gateway run \
--access-node-grpc-host=$ACCESS_NODE_GRPC_HOST \
--access-node-spork-hosts=$ACCESS_NODE_SPORK_HOSTS \
--flow-network-id=$FLOW_NETWORK_ID \
--init-cadence-height=$INIT_CADENCE_HEIGHT \
--ws-enabled=true \
--coinbase=$COINBASE \
--coa-address=$COA_ADDRESS \
--coa-key=$COA_KEY \
--rate-limit=9999999 \
--rpc-host=0.0.0.0 \
--gas-price=$GAS_PRICE
--tx-state-validation=local-index
Restart=always
RestartSec=3
LimitNOFILE=4096


[Install]
WantedBy=multi-user.target
EOF


cat /etc/systemd/system/gateway.service
sudo systemctl enable gateway

When running Access Node and EVM Gateway on the same logical host you must configure --state-stream-addr to use the same address/port combination which is set for Access Node --rpc-addr. This is required by the GW to allow querying both the streaming and non-streaming APIs using the same connection.

The appropriate block height to set for —init-cadence-height is the starting block of the current spork version being run for that network.

State stream configuration

The following log entries may occur when --state-stream-addr and --rpc-port are not set to the same values when running services locally

Terminal window
failure in event subscription at height ${INIT-CADENCE-HEIGHT}, with: recoverable: disconnected: error receiving event: rpc error: code = Unimplemented desc = unknown service flow.executiondata.ExecutionDataAPI”
Terminal window
component execution data indexer initialization failed: could not verify checkpoint file: could not find expected root hash e6d4f4c755666c21d7456441b4d33d3521e5e030b3eae391295577e9130fd715 in checkpoint file which contains: [e10d3c53608a1f195b7969fbc06763285281f64595be491630a1e1bdfbe69161]

Access Node not fully synced

The following log entry will occur when the EVM Gateway attempts to sync with the Access Node but it has not yet synced up to latest block

Terminal window
failure in event subscription at height ${INIT-CADENCE-HEIGHT}, with: recoverable: disconnected: error receiving event: rpc error: code = FailedPrecondition desc = could not get start height: failed to get lowest indexed height: index not initialized

Order of operations:

  1. access-node: ensure it’s fully synced in terms of block height before proceeding.
    • Query to Flow block height through Flow CLI
      Terminal window
      flow blocks get latest -n mainnet
    • Check block height on Flow block explorer or testnet
  2. gateway
Terminal window
sudo systemctl daemon-reload
sudo systemctl restart access-node
sudo systemctl restart gateway

Verify local Access Node has synced to latest block

Terminal window
flow blocks get latest --host localhost:9000

For gateway

Terminal window
curl -s -XPOST 'localhost:8545' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq

Verify block height compared to EVM block explorer or testnet

Terminal window
# change log settings to persistent if not already
sed -i 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald


journalctl -u access-node.service -f -n 100
journalctl -u gateway.service -f -n 100

Logs should look like this which means EVM Gateway is indexing the Access Node data

Terminal window
{"level":"info","component":"ingestion","cadence-height":211179523,"cadence-event-length":1,"time":"2024-09-07T15:35:08Z","message":"received new cadence evm events"}
{"level":"info","component":"ingestion","hash":"0xa3e86dbe8a454d7f84908fc3af5ea3e335f4ab33bce8127f43243eb987505020","evm-height":2853,"cadence-height":211179523,"cadence-id":"872cda9894505a8357daa493d45fa99d2632b8a36a556e013df01506c5f893da","parent-hash":"0xa9b37ac67550b5b96490270b49b78fca35ff701b92bf37a6b7f6193050681bf4","tx-hashes-root":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","time":"2024-09-07T15:35:08Z","message":"new evm block executed event"}
{"level":"info","component":"ingestion","cadence-height":211179524,"cadence-event-length":1,"time":"2024-09-07T15:35:08Z","message":"received new cadence evm events"}
{"level":"info","component":"ingestion","hash":"0x0c7706407fd79496a845c2a337ecbf551fb670ba5ff864e42d94b6897d71474b","evm-height":2854,"cadence-height":211179524,"cadence-id":"5bd1d53fe3e3125d980dee8d9a75b8998d64d1710acacc1f4df0ad52ab4140f6","parent-hash":"0xa3e86dbe8a454d7f84908fc3af5ea3e335f4ab33bce8127f43243eb987505020","tx-hashes-root":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","time":"2024-09-07T15:35:08Z","message":"new evm block executed event"}
{"level":"info","component":"ingestion","cadence-height":211179525,"cadence-event-length":1,"time":"2024-09-07T15:35:08Z","message":"received new cadence evm events"}
{"level":"info","component":"ingestion","hash":"0x42d7665cc17c04170eb77e2b14f7adbfbf30a5ac65d5b36a98a2857798a7a33f","evm-height":2855,"cadence-height":211179525,"cadence-id":"df6a5ddbd669f56c44dc1f4c6701e00c323eedc6a6cd0a14593aeeed9000ea25","parent-hash":"0x0c7706407fd79496a845c2a337ecbf551fb670ba5ff864e42d94b6897d71474b","tx-hashes-root":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","time":"2024-09-07T15:35:08Z","message":"new evm block executed event"}
{"level":"info","component":"ingestion","cadence-height":211179526,"cadence-event-length":1,"time":"2024-09-07T15:35:08Z","message":"received new cadence evm events"}
{"level":"info","component":"ingestion","hash":"0x7e248dd506b721faeece565382d1f310665c2faa5a0882b49f039b73eddb0b34","evm-height":2856,"cadence-height":211179526,"cadence-id":"d34dc88006ebd8ce3e82af11d9668f864c7a9a274e4583c392fe367fc210294c","parent-hash":"0x42d7665cc17c04170eb77e2b14f7adbfbf30a5ac65d5b36a98a2857798a7a33f","tx-hashes-root":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","time":"2024-09-07T15:35:08Z","message":"new evm block executed event"}
{"level":"info","component":"ingestion","cadence-height":211179527,"cadence-event-length":1,"time":"2024-09-07T15:35:08Z","message":"received new cadence evm events"}
{"level":"info","component":"ingestion","hash":"0x133702b182bdff543bfccd0bf380b28a6b4a9de2c1d2123b14601b522bd49a33","evm-height":2857,"cadence-height":211179527,"cadence-id":"e886a98d565f11119f6f38950145c7c86ebf438dbd6c296d88bfac8d72857ccc","parent-hash":"0x7e248dd506b721faeece565382d1f310665c2faa5a0882b49f039b73eddb0b34","tx-hashes-root":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421","time":"2024-09-07T15:35:08Z","message":"new evm block executed event"}

Edit on GitHub