feat: add prometheus metrics in dedicated port

[Imod7]

Description

Relates to the issue #1008

This PR is a first approach/test on adding prometheus metrics in Sidecar. More specifically this PR :

• Uses prom-client
• Runs a separate express instance and in a different port (9100) where the Prometheus metrics are exposed
• Exposes the default Prometheus metrics
• Creates and exposes a custom metric of type counter that increments in every http error
• All metrics are prefixed with sas_ that denotes that are "Substrate Api Sidecar" related metrics.

Run the Prometheus metrics endpoint

• Checkout in the current branch
• prerequisite to have prom-client installed (you can with yarn add prom-client)
  note: if you get an error while adding prom-client try to do a yarn dedupe first
• yarn build
• You can start Sidecar & enable the metrics endpoint by running :
  • yarn start --prometheus
• Or you can start Sidecar, enable the metrics endpoint & define a custom port for the metrics endpoint by running :
  • yarn start --prometheus --prometheus-port=<YOUR_CUSTOM_PORT>
• Open Sidecar on the usual http://127.0.0.1:8080
• Open Metrics endpoint :
  • on the default port : http://127.0.0.1:9100/metrics or
  • on your custom port if you defined one : http://127.0.0.1:<YOUR_CUSTOM_PORT>/metrics
• On the metrics endpoint you should see

  # HELP sas_http_errors Number of HTTP Errors
  # TYPE sas_http_errors counter
  sas_http_errors 0
  
  # HELP sas_process_cpu_user_seconds_total Total user CPU time spent in seconds.
  # TYPE sas_process_cpu_user_seconds_total counter
  sas_process_cpu_user_seconds_total 1.335734
  
  # HELP sas_process_cpu_system_seconds_total Total system CPU time spent in seconds.
  # TYPE sas_process_cpu_system_seconds_total counter
  sas_process_cpu_system_seconds_total 0.146793
  ...
  ...
  

  The first 3 lines denote the custom metric that we defined. The rest of the lines describe the default prometheus metrics.

Testing

• If we request http://127.0.0.1:8080/blocks this will trigger an httpError
• This means that our custom metric sas_http_errors (of type counter) should be increased by 1 so we should see on the metrics endpoint the following :

# HELP sas_http_errors Number of HTTP Errors
# TYPE sas_http_errors counter
sas_http_errors 1

Additional Notes

Apart counters there are other types of metrics that can be added like Histograms, Gauges, etc. More details on metric types can be found in :

• prom-client - Custom Metrics
• Prometheus Docs - Metric Types

[TarikGul]

Overall, really good job! Just some nits on global scope, and only have things ran if prometheus is active.

[Imod7]

I have a question about the help page/message :

• If I run yarn start -h it shows me the prometheus options that I added so I get the output I expect :

  usage: main.js [-h] [-v] [-p] [-pp PROMETHEUS_PORT]
  
  optional arguments:
    -h, --help            show this help message and exit
    -v, --version         print substrate-api-sidecar version
    -p, --prometheus      enable the prometheus metrics endpoint
    -pp PROMETHEUS_PORT, --prometheus-port PROMETHEUS_PORT
                          specify the port number on which the prometheus metrics are exposed [default: 9100]
  
  

• If I run substrate-api-sidecar -h I only get the help and version options. It probably does not get my local version/folder/branch of sidecar ? thats why ?

[TarikGul]

I have a question about the help page/message :

• If I run yarn start -h it shows me the prometheus options that I added so I get the output I expect :

  usage: main.js [-h] [-v] [-p] [-pp PROMETHEUS_PORT]
  
  optional arguments:
    -h, --help            show this help message and exit
    -v, --version         print substrate-api-sidecar version
    -p, --prometheus      enable the prometheus metrics endpoint
    -pp PROMETHEUS_PORT, --prometheus-port PROMETHEUS_PORT
                          specify the port number on which the prometheus metrics are exposed [default: 9100]
  

• If I run substrate-api-sidecar -h I only get the help and version options. It probably does not get my local version/folder/branch of sidecar ? thats why ?

You need to run yarn pack and create the tarball file (executable) which is the binary then run the executable.

[TarikGul]

This is fine with the Logger since it's only being used in here which makes sense. But that being said as a side note, the logger can be a top level variable since it's only ever instantiated once. The logger file itself has a static this.create which only gets called the first time the logger is instantiated in the entry file.