= Hawkular Metrics

This project is an implementation of the Vert.x Metrics Service Provider Interface (SPI): metrics built from Vert.x
events will be sent to Hawkular, an http://www.hawkular.org[open source monitoring and management solution].

== Features

* Vert.x core tools monitoring: TCP/HTTP client and servers, `link:../../apidocs/io/vertx/core/datagram/DatagramSocket.html[DatagramSocket]`,
`link:../../apidocs/io/vertx/core/eventbus/EventBus.html[EventBus]` and handlers
* User defined metrics via an `link:../../apidocs/io/vertx/core/eventbus/EventBus.html[EventBus]` bridge.

== Prerequisites

Follow the http://www.hawkular.org/hawkular-services/docs/quickstart-guide/[instructions to get Hawkular up and running].

NOTE: You can use a standalone https://github.com/hawkular/hawkular-metrics[Hawkular Metrics] server as well.

== Getting started

The _vertx-hawkular-metrics_ module must be present in the classpath.

Maven users should add this to their project POM file:

[source,xml,subs="+attributes"]
----
<dependency>
  <groupId>io.vertx</groupId>
  <artifactId>vertx-hawkular-metrics</artifactId>
  <version>3.4.2.redhat-006</version>
</dependency>
----

And Gradle users, to their build file:

[source,groovy,subs="+attributes"]
----
compile 'io.vertx:vertx-hawkular-metrics:3.4.2.redhat-006'
----

Vert.x does not enable SPI implementations by default. You must enable metric collection in the Vert.x options:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions().setEnabled(true)
));
----

== Configuration

=== Remote Hawkular server

By default, _vertx-hawkular-metrics_ sends metrics to a Hawkular server listening on `localhost` port `8080`.
But in production, the Hawkular server will likely run on a separate machine:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setHost("hawkular.example.com")
    .setPort(8080)
));
----

=== Tenant selection

Hawkular Metrics is a multi-tenant solution, and _vertx-hawkular-metrics_ can send metrics for a tenant other than `default`:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("sales-department")
));
----

=== Connecting to a Hawkular server

Requests sent to a Hawkular server must be authenticated and tenant must be set to `hawkular`:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("hawkular")
    .setAuthenticationOptions(
      new AuthenticationOptions()
        .setEnabled(true)
        .setId("username")
        .setSecret("password")
    )
));
----

=== Openshift Metrics token authentication

When working with Openshift's internal Metrics server, you can configure token authentication with a custom HTTP header:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTenant("my-namespace")
    .setHttpHeaders(new JsonObject()
      .put("Authorization", "Bearer xkjdksf9890-shjkjhkjlkjlk")
    )
));
----

=== HTTPS and other HTTP related options

_vertx-hawkular-metrics_ communicates with the Hawkular server over HTTP. In order to communicate over HTTPS, set the
`ssl` flag to true:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setHost("hawkular.example.com")
    .setPort(443)
    .setHttpOptions(new HttpClientOptions().setSsl(true))
));
----

NOTE: The usual `link:../../apidocs/io/vertx/core/http/HttpClientOptions.html[HttpClientOptions]` properties can be used for SSL setup or client
tuning.

=== Metric tags

http://www.hawkular.org/hawkular-metrics/docs/user-guide/#_tagging[Tags] can be applied to metrics:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setTags(new JsonObject()
      .put("dc", "mars01")
      .put("rack", "web-paca")
      .put("host", "host13"))
));
----

_vertx-hawkular-metrics_ maintains a LRU cache of tagged metrics to avoid repeating tagging requests.
The cache size can be configured and defaults to `4096` metric names.

It is also possible to apply tags to a specific set of metrics defined via exact match or regex match:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setMetricTagsMatches(Arrays.asList(
      new MetricTagsMatch()
        .setValue("myapp.foo.my-metric")
        .setTags(new JsonObject().put("myapp", "foo")),
      new MetricTagsMatch()
        .setType(MatchType.REGEX).setValue(".*\\.foo\\.*")
        .setTags(new JsonObject().put("myapp", "foo"))
      )
    )
));
----

WARNING: If you use regex match, a wrong regex can potentially match a lot of metrics.

NOTE: When evaluating tags to apply, metric specific tags have higher priority than global tags.
In other words, a metric specific tag may overwrite a global tag.

Please refer to `link:../../apidocs/io/vertx/ext/hawkular/VertxHawkularOptions.html[VertxHawkularOptions]` for an exhaustive list of options.

== Vert.x core tools metrics

This section lists all the metrics generated by monitoring the Vert.x core tools.

=== Net Client

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.net.client.<host>:<port>.connections`
|Number of connections to the remote host currently opened.

|Counter
|`vertx.net.client.<host>:<port>.bytesReceived`
|Total number of bytes received from the remote host.

|Counter
|`vertx.net.client.<host>:<port>.bytesSent`
|Total number of bytes sent to the remote host.

|Counter
|`vertx.net.client.<host>:<port>.errorCount`
|Total number of errors.

|===

=== HTTP Client

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.http.client.<host>:<port>.connections`
|Number of connections to the remote host currently opened.

|Counter
|`vertx.http.client.<host>:<port>.bytesReceived`
|Total number of bytes received from the remote host.

|Counter
|`vertx.http.client.<host>:<port>.bytesSent`
|Total number of bytes sent to the remote host.

|Counter
|`vertx.http.client.<host>:<port>.errorCount`
|Total number of errors.

|Gauge
|`vertx.http.client.<host>:<port>.requests`
|Number of requests waiting for a response.

|Counter
|`vertx.http.client.<host>:<port>.requestCount`
|Total number of requests sent.

|Counter
|`vertx.http.client.<host>:<port>.responseTime`
|Cumulated response time.

|Gauge
|`vertx.http.client.<host>:<port>.wsConnections`
|Number of websockets currently opened.

|===

=== Datagram socket

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Counter
|`vertx.datagram.<host>:<port>.bytesReceived`
|Total number of bytes received on the `<host>:<port>` listening address.

|Counter
|`vertx.datagram.<host>:<port>.bytesSent`
|Total number of bytes sent to the remote host.

|Counter
|`vertx.datagram.errorCount`
|Total number of errors.

|===

=== Net Server

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.net.server.<host>:<port>.connections`
|Number of opened connections to the Net Server listening on the `<host>:<port>` address.

|Counter
|`vertx.net.server.<host>:<port>.bytesReceived`
|Total number of bytes received by the Net Server listening on the `<host>:<port>` address.

|Counter
|`vertx.net.server.<host>:<port>.bytesSent`
|Total number of bytes sent to the Net Server listening on the `<host>:<port>` address.

|Counter
|`vertx.net.server.<host>:<port>.errorCount`
|Total number of errors.

|===

=== HTTP Server

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.http.server.<host>:<port>.connections`
|Number of opened connections to the HTTP Server listening on the `<host>:<port>` address.

|Counter
|`vertx.http.server.<host>:<port>.bytesReceived`
|Total number of bytes received by the HTTP Server listening on the `<host>:<port>` address.

|Counter
|`vertx.http.server.<host>:<port>.bytesSent`
|Total number of bytes sent to the HTTP Server listening on the `<host>:<port>` address.

|Counter
|`vertx.http.server.<host>:<port>.errorCount`
|Total number of errors.

|Gauge
|`vertx.http.client.<host>:<port>.requests`
|Number of requests being processed.

|Counter
|`vertx.http.client.<host>:<port>.requestCount`
|Total number of requests processed.

|Counter
|`vertx.http.client.<host>:<port>.processingTime`
|Cumulated request processing time.

|Gauge
|`vertx.http.client.<host>:<port>.wsConnections`
|Number of websockets currently opened.

|===

=== Event Bus

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.eventbus.handlers`
|Number of event bus handlers.

|Counter
|`vertx.eventbus.errorCount`
|Total number of errors.

|Counter
|`vertx.eventbus.bytesWritten`
|Total number of bytes sent while sending messages to event bus cluster peers.

|Counter
|`vertx.eventbus.bytesRead`
|Total number of bytes received while reading messages from event bus cluster peers.

|Gauge
|`vertx.eventbus.pending`
|Number of messages not processed yet. One message published will count for `N` pending if `N` handlers
are registered to the corresponding address.

|Gauge
|`vertx.eventbus.pendingLocal`
|Like `vertx.eventbus.pending`, for local messages only.

|Gauge
|`vertx.eventbus.pendingRemote`
|Like `vertx.eventbus.pending`, for remote messages only.

|Counter
|`vertx.eventbus.publishedMessages`
|Total number of messages published (publish / subscribe).

|Counter
|`vertx.eventbus.publishedLocalMessages`
|Like `vertx.eventbus.publishedMessages`, for local messages only.

|Counter
|`vertx.eventbus.publishedRemoteMessages`
|Like `vertx.eventbus.publishedMessages`, for remote messages only.

|Counter
|`vertx.eventbus.sentMessages`
|Total number of messages sent (point-to-point).

|Counter
|`vertx.eventbus.sentLocalMessages`
|Like `vertx.eventbus.sentMessages`, for local messages only.

|Counter
|`vertx.eventbus.sentRemoteMessages`
|Like `vertx.eventbus.sentMessages`, for remote messages only.

|Counter
|`vertx.eventbus.receivedMessages`
|Total number of messages received.

|Counter
|`vertx.eventbus.receivedLocalMessages`
|Like `vertx.eventbus.receivedMessages`, for remote messages only.

|Counter
|`vertx.eventbus.receivedRemoteMessages`
|Like `vertx.eventbus.receivedMessages`, for remote messages only.

|Counter
|`vertx.eventbus.deliveredMessages`
|Total number of messages delivered to handlers.

|Counter
|`vertx.eventbus.deliveredLocalMessages`
|Like `vertx.eventbus.deliveredMessages`, for remote messages only.

|Counter
|`vertx.eventbus.deliveredRemoteMessages`
|Like `vertx.eventbus.deliveredMessages`, for remote messages only.

|Counter
|`vertx.eventbus.replyFailures`
|Total number of message reply failures.

|Counter
|`vertx.eventbus.<address>.processingTime`
|Cumulated processing time for handlers listening to the `address`.

|===

== Vert.x pool metrics

This section lists all the metrics generated by monitoring Vert.x pools.

There are two types currently supported:

* _worker_ (see `link:../../apidocs/io/vertx/core/WorkerExecutor.html[WorkerExecutor]`)
* _datasource_ (created with Vert.x JDBC client)

NOTE: Vert.x creates two worker pools upfront, _vert.x-worker-thread_ and _vert.x-internal-blocking_.

All metrics are prefixed with `<type>.<name>.`. For example, `worker.vert.x-internal-blocking.`.

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Counter
|`vertx.pool.<type>.<name>.delay`
|Cumulated time waiting for a resource (queue time).

|Gauge
|`vertx.pool.<type>.<name>.queued`
|Current number of elements waiting for a resource.

|Counter
|`vertx.pool.<type>.<name>.queueCount`
|Total number of elements queued.

|Counter
|`vertx.pool.<type>.<name>.usage`
|Cumulated time using a resource (i.e. processing time for worker pools).

|Gauge
|`vertx.pool.<type>.<name>.inUse`
|Current number of resources used.

|Counter
|`vertx.pool.<type>.<name>.completed`
|Total number of elements done with the resource (i.e. total number of tasks executed for worker pools).

|Gauge
|`vertx.pool.<type>.<name>.maxPoolSize`
|Maximum pool size, only present if it could be determined.

|Gauge
|`vertx.pool.<type>.<name>.inUse`
|Pool usage ratio, only present if maximum pool size could be determined.

|===

== Verticle metrics

[cols="15,50,35", options="header"]
|===
|Metric type
|Metric name
|Description

|Gauge
|`vertx.verticle.<name>`
|Number of verticle instances deployed.

|===

== User defined metrics

Users can send their own metrics to the Hawkular server. In order to do so, the event bus metrics bridge must be
enabled:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setMetricsBridgeEnabled(true)
));
----

By default, the metrics bus handler is listening to the `hawkular.metrics` address. But the bridge address
can be configured:

[source,java]
----
Vertx vertx = Vertx.vertx(new VertxOptions().setMetricsOptions(
  new VertxHawkularOptions()
    .setEnabled(true)
    .setMetricsBridgeEnabled(true)
    .setMetricsBridgeAddress("__hawkular_metrics")
));
----

The metrics bridge handler expects messages in the JSON format. The JSON object must at least provide a metric
`id` and a numerical `value`:

[source,java]
----
JsonObject message = new JsonObject()
  .put("id", "myapp.files.opened")
  .put("value", 7);
vertx.eventBus().publish("hawkular.metrics", message);
----

The handler will assume the metric is a gauge and will assign a timestamp corresponding to the time when the message was processed.
If the metric is a counter or availability, or if you prefer explicit configuration, set the `type` and/or `timestamp` attributes:

[source,java]
----
JsonObject counterMetric = new JsonObject()
  .put("id", "myapp.files.opened")
  .put("type", "counter")
  .put("timestamp", 189898098098908L)
  .put("value", 7);
vertx.eventBus().publish("hawkular.metrics", counterMetric);

JsonObject availabilityMetric = new JsonObject()
  .put("id", "myapp.mysubsystem.status")
  .put("type", "availability")
  .put("value", "up");
vertx.eventBus().publish("hawkular.metrics", availabilityMetric);
----

NOTE: Hawkular understands all timestamps as milliseconds since January 1, 1970, 00:00:00 UTC.