Monitoring Best Practices
It is essential that you have monitoring in place on your validator. In the event that your validator is delinquent (behind the rest of the network) you want to respond immediately to fix the issue. One very useful tool to monitor your validator is
Solana watchtower is an extremely useful monitoring tool that will regularly monitor the health of your validator. It can monitor your validator for delinquency then notify you on your application of choice: slack, discord, telegram or twilio. Additionally,
solana-watchtower has the ability to monitor the health of the entire cluster so that you can be aware of any cluster wide problems.
Solana Watchtower: Getting Started
To get started with
solana-watchtower --help. From the help menu, you can see the optional flags and an explanation of the command. Here is a sample command that will monitor a validator node with an identity public key of
solana-watchtower --monitor-active-stake --validator-identity \
The command will monitor your validator, but you will not get notifications unless you added the environment variables mentioned in
solana-watchtower --help. Since getting each of these services setup for notifications is not straight forward, the next section will walk through setting up watchtower notifications on telegram.
Solana Watchtower: Telegram Setup
To send validator health notifications to your telegram account, we are going to do a few things:
- Create a bot to send the messages. The bot will be created using BotFather on Telegram.
- Send a message to the bot.
- Create a telegram group that will get the watchtower notifications
- Add the environment variables to the your command line environment.
- Restart the
Create a Bot Using BotFather
In telegram, search for @BotFather. Send the following message to @BotFather:
/newbot. Next you will have to come up with a name for the bot. The only requirement is that it cannot have dashes or spaces and it but end in the word bot. Many names have already been taken, so you may have to try a few. Once you find an available name, you will get a response for @BotFather that includes a link to chat with the bot as well as a token for the bot. Take note of the token. You will need it when you setup your environment variables.
Send a Message to The Bot
Find the bot in telegram and send it the following message:
/start. Messaging the bot will help you later when looking for the bot chatroom id.
Create Telegram Group
In telegram, click on the new message icon and then select new group. Find your newly created bot and add the bot to the group. Next, name the group whatever you'd like.
Set Environment Variables For Watchtower
Now that we have a bot setup, we will need to set the environment variables for the bot so that watchtower can send notifications. First, recall the chat message that you got from @BotFather. In the message, there was a HTTP API token for your bot. The token will have this format:
389178471:MMTKMrnZB4ErUzJmuFIXTKE6DupLSgoa7h4o. You will use that token to set the
TELEGRAM_BOT_TOKEN environment variable. In the terminal where you plan to run
solana-watchtower, run the following:
export TELEGRAM_BOT_TOKEN=<HTTP API Token>
Next, you need the chat id for your group so that
solana-watcher knows where to send the message. First, send a message to your bot in the chat group that you created. Something like
@newvalidatorbot hello. Next, in your browser, got to
https://api.telegram.org/bot<HTTP API Token>/getUpdates. Make sure to replace
<HTTP API TOKEN> with your api token that you got in the @BotFather message. Also make sure that you include the word
bot in the url before the api token. Make the request in the browser. The response should be in json. Search for the string
"chat": in the json. The
id value of that chat is your
TELEGRAM_CHAT_ID. It will be a negative number like:
-781559558. Remember to include the negative sign!. If you cannot find
"chat": in the json, then you may have to remove the bot from your chat group and add it again.
Now, export the environment variable where you plan to run
export TELEGRAM_CHAT_ID=<negative chat id number>
Now that your environment variables are set, restart
solana-watchtower. You should see output about your validator. To test that your telegram configuration is working properly, you could stop your validator briefly until it is labeled as delinquent. Up to a minute after the validator is delinquent, you should receive a message in the telegram group from your bot. Start the validator again and verify that you get another message in your telegram group from the bot. The message should say
Solana Watchtower Operations
It is a best practice to run the
solana-watchtower command on a separate server from your validator. In the case that you run
solana-watchtower on the same computer as your
solana-validator process, then during catastrophic events like a power outage, you will not be aware of the issue, because your
solana-watchtower process will stop at the same time as your
Additionally, running the
solana-watchtower process manually with environment variables set in the terminal is a good way to test out the command, it is not operationally sound because the process will not be restarted when the terminal closes or during a system restart. Instead, you could run your
solana-watchtower command as a system process similar to
solana-validator. In the system process file, you can specify the environment variables for your bot.