discovery-service-rs

This is the alternative to original Talos Discovery Service, that works fine with Talos discovery client requests.

To use this Sevice set the same endpoint in all configuration *.yaml you used in Sevice configuration.

E.g. if you use 10.1.1.1:3000 address for your instance of Service, configuration *.yaml files of your cluster nodes must contain something like this:

# ...
    discovery:
        enabled: true # Enable the cluster membership discovery feature.
        # Configure registries used for cluster member discovery.
        registries:
            # Kubernetes registry uses Kubernetes API server to discover cluster members and stores additional information
            kubernetes:
                disabled: true # Disable Kubernetes discovery registry.
            # Service registry is using an external service to push and pull information about cluster members.
            service:
                # External service endpoint.
                endpoint: http://10.1.1.1:3000
# ...

See more about how to work with Sevice here.

Description

Main Interface Description

• HelloRequest: This request sends the cluster ID and node version to the Service. If the Service does not store the ID of such a cluster, a new cluster record is created. If read_real_ip = true in the configuration, the Service attempts to read the connecting client’s address from the X-Real-IP header.
  The response includes redirect_endpoint if it is not an empty string in the configuration, as well as the address of the client that sent the request.
• AffiliateUpdateRequest: The request includes encrypted data, an affiliate ID, a vector of endpoints with encrypted data, the TTL for this affiliate, and the cluster ID to which the affiliate belongs. The affiliate for the specified cluster ID is either updated or a new one is created. The response is empty with a status.
• AffiliateDeleteRequest: The request includes the cluster ID and the affiliate ID to be deleted. The response is empty with a status.
• ListRequest: The request consists of a cluster ID. The response sends a vector of affiliates with their TTL, encrypted data, and all endpoints.
• WatchRequest: The request consists of a cluster ID. A persistent connection is established with the client. First, the Service responds to the client with a vector of affiliates for the specified cluster (same as in ListRequest). After that, the client receives notifications about AffiliateUpdate and AffiliateDelete events.

Program Operation

The program either loads the configuration from a file and/or environment variables or uses default settings. If the snapshots parameter is set to true, snapshots are loaded from the file at snapshots_path, and the function to save snapshots at an interval of make_snapshot seconds is activated.

Every cleanup seconds, the Service iterates through all clusters and first removes all endpoints with expired TTL, then affiliates with expired TTL.
A cluster is deleted if all its affiliates are removed or if the cluster_ttl has expired.

The server with the main interface is started. If service_tls = true, the server starts with a certificate and its corresponding key, loaded from the paths specified in the configuration.

Interaction with the Cluster Node Client

The first request sent by a node to this Service is a HelloRequest. This is followed by a WatchRequest (likely a test request or to retrieve the list of affiliates if the Service is aware of them). Next, the node sends several AffiliateUpdateRequests. After that, a WatchRequest connection is established again, this time persistent rather than terminating immediately. Once the Watch connection is terminated, the cluster is assigned a cluster_ttl, after which it will be removed from the main state.

Reading local snapshot file

Use snapshot_reader binary to read snaphot file. By default it tries to read path: /var/lib/discovery-service-rs/snapshot.bincode. Use --snapshot-path flag to reconfigure this path.

Information requests

Service starts HTTP Server for information requests on <service_ip>:<http_port> by default. Routes: /instect to get information about cluster_id, /metrics to get metrics about current Service status.

Configuration

Service has default configuration as follows (note, that duration sets in seconds, boolean values must be true or false):

* service_ip = "0.0.0.0"
* service_port = "3000"
* http_port = "2000"
* redirect_endpoint = ""
* cleanup = 30
* cluster_ttl = 0
* make_snapshot = 600
* snapshots = true
* snapshots_path = "/var/lib/discovery-service-rs/snapshot.bincode"
* service_tls = false
* certificate = ""
* cert_key = ""
* read_real_ip = false
* discovery_threads = 2

If cluster_ttl = 0 (default), Service do not remove Cluster from state after some preset ttl. Othewise Cluster will be removed after all affiliates ttls expired.

Set read_real_ip as true, when some proxy server is between Service and nodes and X-Real-IP header is used. But in local network use false by default.

Configuration could be set in configuration .ini file or via environment variables. Default path for file: /etc/discovery-service-rs/discovery-service-rs.ini (or you can change it via DISCOVERY_RS_CONFIG env var).

Environment variables you can set like this:

export snapshots_path=/var/my_path/snapshot.file

Exception: discovery_log_source need to be set only via ENV variable. Set totrue to expand logs information with modules paths of logs source. Default is false.

Usage as a container in Altlinux environment

Build image via podman or docker:

podman build --no-cache=True -t discovery-service-rs .

Prepare the path for snapshots:

sudo mkdir -p /*your-path*/
sudo chmod 700 /*your-path*/

Start Service:

podman run --rm -d \
-p 3000:3000 \
-p 2000:2000 \
--userns=keep-id \
-v /*your-path*/:/app/snapshot:U \
-e "snapshots_path=/app/snapshot/snapshot.bincode" \
-e "ALTOS=debug" \
--name discovery-rs \
discovery-service-rs:latest

Read logs:

podman logs -f discovery-rs

Stop service:

podman stop discovery-rs

Testing

Download this repository and install grpcurl:

git clone https://altlinux.space/alt-orchestra/discovery-service-rs.git
apt-get install grpcurl

Go to the repository catalogue and check discovery-service-rs Hello endpoint:

cd /[PATH TO REPOSITORY PARENT]/discovery-service-rs
grpcurl -proto cluster_grpc.proto -import-path ./proto -plaintext -d '{"clusterId": "abc", "clientVersion": "v1.10.1"}' -H 'X-Real-IP: 1.2.3.4' <IP>:<PORT> sidero.discovery.server.Cluster/Hello

Remove -plaintext if you test Service with TLS configuration.

TODO

• TLS for http server;
• dinamic loading for tls certs.