Môi trường cài đặt:
– node1: 10.144.39.21
– node2: 10.144.39.22
– node3: 10.144.39.21

OS 3 máy: ubuntu 20.04
ES cài đặt: bản mới nhất trên trang chủ: https://www.elastic.co/downloads/elasticsearch

Khai báo hostname của cả 3 máy vào file hosts, để 3 máy ping hostname của nhau được.

cat > /etc/hosts << EOF
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

10.144.39.21 es01
10.144.39.22 es02
10.144.39.23 es03
EOF

Cài đặt Elasticsearch service trên cả 3 máy
Download file cài đặt

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.11.3-amd64.deb
dpkg -i elasticsearch-8.11.3-amd64.deb


systemctl enable elasticsearch.service

Thao tác trên máy es01, đứng tại thư mục /usr/share/elasticsearch/bin

Gen cert CA, chúng ta sẽ sử dụng CA này để sinh cert cho cluster, http cert cho các bước sau

root@es01:/usr/share/elasticsearch/bin# ./elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]:       
Enter password for elastic-stack-ca.p12 : 
root@es01:/usr/share/elasticsearch/bin# 
root@es01:/usr/share/elasticsearch/bin# ls /usr/share/elasticsearch/elastic-stack-ca.p12
/usr/share/elasticsearch/elastic-stack-ca.p12

Lưu ý lưu password lại để sử dụng sau này. Chúng ta thu được file elastic-stack-ca.p12, mở file này phải có password.

Tiếp theo chúng ta sẽ sinh tiếp certificate, được ký bởi cái CA vừa sinh ở trên. Command như sau:

root@es01:/usr/share/elasticsearch/bin# ./elasticsearch-certutil cert --ca elastic-stack-ca.p12

Output như sau:

root@es01:/usr/share/elasticsearch/bin# ./elasticsearch-certutil cert --ca elastic-stack-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.


    * All certificates generated by this tool will be signed by a certificate authority (CA)
      unless the --self-signed command line option is specified.
      The tool can automatically generate a new CA for you, or you can provide your own with
      the --ca or --ca-cert command line options.


By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (elastic-stack-ca.p12) : 
Please enter the desired output file [elastic-certificates.p12]: 
Enter password for elastic-certificates.p12 : 

Certificates written to /usr/share/elasticsearch/elastic-certificates.p12

This file should be properly secured as it contains the private key for 
your instance.
This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.
root@es01:/usr/share/elasticsearch/bin# 

Bước này sẽ hỏi:
– password cho CA: nhập password của CA ở bước trên.
– password cho cái cert sẽ tạo: nhập password tuỳ ý, lưu lại sử dụng cho bước tiếp theo.
Chúng ta thu được file elastic-certificates.p12, tại đường dẫn /usr/share/elasticsearch/elastic-certificates.p12

Tiếp theo, generate cert sử dụng cho interface http. Cert này sẽ sử dụng để enable https cho port 9200 (mặc định) của elasticsearch.

./elasticsearch-certutil http

root@es01:/usr/share/elasticsearch/bin# ./elasticsearch-certutil http

## Elasticsearch HTTP Certificate Utility

The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.

This tool will ask you a number of questions in order to generate the right
set of files for your needs.

## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA). 

If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.

Generate a CSR? [y/N]N

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]y

## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
CA Path: /usr/share/elasticsearch/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press <ENTER> at the prompt
Password for elastic-stack-ca.p12:

## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 20y

## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]N

## Which hostnames will be used to connect to your nodes?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

es01
es02
es03
es04
es05
es06
es07
es08
es09
es10

You entered the following hostnames.

 - es1
 - es2
 - es3
 - es4
 - es5
 - es6
 - es7
 - es8
 - es9
 - es10

Is this correct [Y/n]Y

## Which IP addresses will be used to connect to your nodes?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

10.144.39.*
Error: 10.144.39.* is not a valid IP address
10.144.39.21
10.144.39.22
10.144.39.23
10.144.39.24
10.144.39.25
10.144.39.26
10.144.39.27
10.144.39.28
10.144.39.29
10.144.39.30
10.144.39.20

You entered the following IP addresses.

 - 10.144.39.21
 - 10.144.39.22
 - 10.144.39.23
 - 10.144.39.24
 - 10.144.39.25
 - 10.144.39.26
 - 10.144.39.27
 - 10.144.39.28
 - 10.144.39.29
 - 10.144.39.30
 - 10.144.39.20

Is this correct [Y/n]Y

## Other certificate options

The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.

Key Name: es1
Subject DN: CN=es1
Key Size: 2048

Do you wish to change any of these options? [y/N]N

## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file:  [<ENTER> for none]
Repeat password to confirm: 

## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip] 

Zip file written to /usr/share/elasticsearch/elasticsearch-ssl-http.zip
root@es01:/usr/share/elasticsearch/bin#

Các lưu ý về cấu hình khi chạy lệnh trên như sau:
– Generate a CSR? [y/N]N    => Cái này nếu các bạn cần CSR để ký với 1 CA khác, ví dụ cert đi mua bên ngoài của globalsign, Let’s Encrypt hay Digicert thì lựa chọn Y. Ở đây tôi tự sinh CA, tự ký nên tôi chọn N
– Use an existing CA? [y/N]y : chọn y để sử dụng chính cái CA đã sinh ở bước trên. y xong thì điền đường dẫn file elastic-stack-ca.p12 và password.
– For how long should your certificate be valid? [5y] 20y ====> tự ký nên sinh dài hẳn ra cho thoải mái, ở đây tôi sinh 20 năm.
– Generate a certificate per node? [y/N]N   ===> chọn Y thì cluster Elasticsearch có bao nhiêu node thì sẽ sinh bấy nhiêu file cert, cái này nếu không có yêu cầu gì bảo mật quá thì ko cần, cứ chọn N để sinh 1 certificate duy nhất sử dụng cho tất cả các node.
– ## Which hostnames will be used to connect to your nodes?  ==> liệt kê các domain sẽ sử dụng để truy cập tới elasticsearch, ở đây tôi điền hostname của máy chủ. Nếu các bạn public elasticsearch cho các đối tượng khác truy cập qua domain, hostname thì điền vào đây, ví dụ elasticsearch.tochuc.com, nếu ko thì cái http certificate sinh ra sẽ không có thông tin của domain elasticsearch.tochuc.com. Client thấy https cert ko match sẽ báo lỗi. Chỗ này các bạn điền nhiều bao nhiêu cũng được, thừa còn hơn thiếu, chấp nhận cả wildcard domain như *.tochuc.vn
– ## Which IP addresses will be used to connect to your nodes? ==> tương tự với hostname nhưng là cho IP, dự kiến cluster sẽ sử dụng những IP nào để đưa cho client kết nối thì điền hết vào, kể cả IP của các lớp Loadbalancer đằng trước.
– ## What password do you want for your private key(s)? ==> nhập password mong muốn vào.
Chúng ta sẽ thu được file /usr/share/elasticsearch/elasticsearch-ssl-http.zip, giải nén file này thu được file http.p12

elasticsearch
├── http.p12
├── README.txt
└── sample-elasticsearch.yml

trên cả 3 máy, tạo thư mục: /etc/elasticsearch/cert_custom
Copy cả 3 file elastic-certificates.p12, elastic-stack-ca.p12, http.p12 vào /etc/elasticsearch/cert_custom và cấp quyền cho user elasticsearch

# ls /etc/elasticsearch/cert_custom
elastic-certificates.p12  elastic-stack-ca.p12  http.p12

# chown -R elasticsearch:elasticsearch /etc/elasticsearch/cert_custom/

Khai báo password cho hệ thống elasticsearch để sử dụng:

• Bước 1: xoá password mặc định có khi cài đặt (có thể không có, tuỳ phiên bản, nhưng cứ xoá đi cho chắc). Chạy 3 lệnh sau:

/usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.transport.ssl.truststore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.transport.ssl.keystore.secure_password
/usr/share/elasticsearch/bin/elasticsearch-keystore remove xpack.security.http.ssl.keystore.secure_password

• Bước 2: Thêm password

  /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
  /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
  /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

  Trong đó:

  • xpack.security.transport.ssl.keystore.secure_password  ==> Nhập password của cert elastic-certificates.p12
  • xpack.security.transport.ssl.truststore.secure_password   ==> Nhập password của cert elastic-certificates.p12
  • xpack.security.http.ssl.keystore.secure_password     ====> Nhập password của http.p12
• Bước 3: Xác nhận lại password vừa nhập

  /usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password
  /usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.transport.ssl.truststore.secure_password
  /usr/share/elasticsearch/bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password

   

Tiếp theo cấu hình /etc/elasticsearch/elasticsearch.yml trên từng máy. Nội dung file từng máy có điểm khác nhau, nhưng đại khái nội dung file này như sau:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: democluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: es01
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.144.39.21
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["es01", "es02", "es03"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["es01", "es02", "es03"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 01-01-2024 10:29:11
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: cert_custom/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: required
  keystore.path: cert_custom/elastic-certificates.p12
  truststore.path: cert_custom/elastic-certificates.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
# cluster.initial_master_nodes: ["es01"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

Các điểm cần lưu ý cấu hình như sau:
cluster.name: democluster   =>   mục này cần 1 cái tên thống nhất cho tất cả các máy tham gia vào cluster.
node.name: es01        =>   tên riêng của từng máy tham gia vào cluster, tên này cần duy nhất, mặc định nó là hostname (tham khảo https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#node-name)
network.host: 10.144.39.21   => điền IP của máy vào
cluster.initial_master_nodes: [“es01”, “es02”, “es03”]   ==> tham khảo https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#initial_master_nodes
Cấu hình xpack =>

xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: cert_custom/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  client_authentication: required
  keystore.path: cert_custom/elastic-certificates.p12
  truststore.path: cert_custom/elastic-certificates.p12

Tới đây thì đã có thể khởi động elasticsearch trên tất cả các máy

systemctl restart elasticsearch.service

Tiếp theo cần reset user quản trị cao nhất của Elasticsearch:

root@es01:/usr/share/elasticsearch# ./bin/elasticsearch-reset-password -u elastic

Thu được password

root@es01:/usr/share/elasticsearch# ./bin/elasticsearch-reset-password -u elastic
This tool will reset the password of the [elastic] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]y


Password for the [elastic] user successfully reset.
New value: M3RPyfOtQAARylj13oYi

Thử kết nối tới cluster, thấy thành công

root@es01:/usr/share/elasticsearch# curl -k -u elastic:M3RPyfOtQAARylj13oYi https://10.144.39.21:9200/_cluster/health
{
   "cluster_name":"democluster",
   "status":"green",
   "timed_out":false,
   "number_of_nodes":3,
   "number_of_data_nodes":3,
   "active_primary_shards":1,
   "active_shards":2,
   "relocating_shards":0,
   "initializing_shards":0,
   "unassigned_shards":0,
   "delayed_unassigned_shards":0,
   "number_of_pending_tasks":0,
   "number_of_in_flight_fetch":0,
   "task_max_waiting_in_queue_millis":0,
   "active_shards_percent_as_number":100.0
}