Elasticsearch on Oracle Cloud Container Engine for Kubernetes

Disclaimer: Kindly note that this is based on my own research and findings and may not reflect the Oracle standpoint.

1. Introduction

Thisblob post provides an overview of the key implementation considerations for deploying Elasticsearch on Oracle Cloud Infrastructure (OCI) Kubernetes Engine (OKE). It is intended to offer high-level guidance and highlight important factors to consider. For comprehensive instructions and the latest updates, please refer to the official Elasticsearch and Oracle Cloud documentation.

2. Oracle Cloud Pre-requisites

This section outlines the requirements to prepare for Elasticsearch deployments on Oracle Cloud.

  • OCI Tenancy Access: Ensure you have access to an Oracle Cloud Infrastructure tenancy. The tenancy must be subscribed to one or more regions where Kubernetes Engine is available.
  • Quota Verification: Confirm that your tenancy has sufficient quota for the required resources (refer to Service Limits). Specifically, verify quotas for:
    o Compute instances
    o Block volumes
    o Load balancers
  • Compartment Creation: As a best practice, create a separate compartment for the Kubernetes cluster. (Refer to “Creating a Compartment” for more details.)
  • VCN Configuration: Set up the Virtual Cloud Network (VCN) and related resources required for the Kubernetes cluster. (Please refer to “Network Resource Configuration” for more details) Common configurations include:
    o Internet Gateway (if using public subnets)
    o NAT Gateway
    o Service Gateway
    o Route Tables
    o Subnets
    o Security Rules for API endpoints, worker nodes, and load balancers
  • User Permissions: Ensure the user belongs to groups with the necessary privileges to create and manage resources on Oracle Cloud. (Refer: IAM documentation)
  • IAM Configuration: Grant the appropriate IAM permissions to manage cloud resources. Oracle Cloud Infrastructure Identity and Access Management (IAM) allows fine-grained control over resource access.

3. OKE Pre-Requisites

This section covers the requirements for creating and configuring an Oracle Cloud Kubernetes Engine (OKE) cluster. For detailed instructions, refer to the official documentation.

  • RBAC Configuration: Enable Role-Based Access Control (RBAC) on the Kubernetes cluster, or use a cloud admin user to create and log in to OKE. For more details on access control, refer to the access control guide.
  • Cluster Type Recommendation: Select the Enhanced Cluster type instead of Basic, as it supports all available features and includes a financially-backed service level agreement (SLA). Reference: cluster type documentation

4. Cluster Setup and Node Configuration

  • OKE Cluster Design:
    o Use multiple worker nodes across multiple fault domains (FDs) for high availability.
    o Ensure the cluster has sufficient resources for Elasticsearch, such as CPU, memory, and storage.
    o Select suitable VM shape align with the workload. The available VM shapes can be found here.
  • Cluster Type:
    o OKE clusters can be created as either Quick or Custom. For production workloads, it is recommended to use Custom to allow for better design and configuration.
  • Network Type:
    The pod networking documentation can be found here.
    o VCN-native pod networking: Allows Kubernetes pods to connect directly to a VCN subnet and communicate natively through a VCN with other pods, services, and the Internet.
    o Flannel overlay: A per-node daemon that handles setting up an overlay network for pod communication.
  • Node Selection:
    o Worker nodes can be created as managed or virtual. Since serverless architecture (virtual) may impose limitations for this deployment, it is recommended to use managed nodes.
  • Node Sizing:
    o Master Nodes: Small but highly reliable (e.g., VM.Standard3.Flex with sufficient CPUs and memory).
    o Data Nodes: Larger nodes for storage and processing (e.g., VM.Standard.E4.Flex with block storage).
    o Client Nodes: Medium-sized nodes for coordinating queries (e.g., VM.Standard3.Flex).
  • Accessing the Cluster:
    The below are two methods to access the OKE cluster. The details can be found here.
    o Cloud Shell: Cloud Shell is a web browser-based terminal accessible from the Oracle Cloud Console.
    o Local Access: Having an instance and configure relevant packages including OCI CLI.

5. Elasticsearch Deployment Architecture

  • Master Nodes:
    o Deploy 3 master nodes to maintain quorum for cluster management.
    o Use a dedicated node pool for master nodes.
  • Data Nodes:
    o Deploy multiple data nodes to handle indexing and searching.
    o Use Oracle Cloud Block Volumes for Elasticsearch data storage. Attach volumes as Persistent Volumes (PVs) in Kubernetes.
  • Client Nodes:
    o Deploy at least 2 client nodes for handling user queries and load balancing.

6. Storage

  • Persistent Storage: Use OCI Block Volumes for Elasticsearch data storage.
  • Policy Configuration: To enable worker nodes to access Block Volume service volumes, create the following IAM policies:
    a. allow any-user to manage volumes in TENANCY where request.principal.type = ‘cluster’
    b. allow any-user to manage volume-attachments in TENANCY where request.principal.type = ‘cluster’
  • CSI Plugin: Dynamically provision block volumes using the CSI plugin, specified by the oci-bv StorageClass (provisioner: blockvolume.csi.oraclecloud.com).
  • Persistent Volume Claim (PVC) Configuration: Specify the appropriate StorageClass in the YAML definition of the PVC:
    c. For the CSI volume plugin, use: storageClassName: “oci-bv”
    d. For the FlexVolume plugin, use: storageClassName: “oci”
  • Validate StorageClass: Confirm the available StorageClass by running:
    kubectl get storageclass
    For more details, refer to the official documentation.

7. Networking

  • Load Balancing: Use OCI Load Balancer for external access to Elasticsearch.
  • Networking Options
    o Option 1: Cluster with Flannel CNI Plugin, Public Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers

o Option 2: Cluster with Flannel CNI Plugin, Private Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers

o Option 3: Cluster with OCI CNI Plugin, Public Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers

o Option 4: Cluster with OCI CNI Plugin, Private Kubernetes API Endpoint, Private Worker Nodes, and Public Load Balancers

8. Security

  • Authentication and Authorization:
    o Use Elasticsearch’s native security features or integrate with OCI Identity and Access Management (IAM).
  • Encryption:
    o Enable encryption in transit using TLS certificates. Use cert-manager to automate TLS certificate provisioning.
  • Role-Based Access Control (RBAC):
    o Use Kubernetes RBAC policies to restrict access to Elasticsearch pods and namespaces, ensuring proper segregation of duties and minimal access principles.

9. Scaling

The Oracle Kubernetes cluster supports multiple autoscaling options. Refer autoscaling documentation for more details.

  • Horizontal Scaling:
    o Use Kubernetes Horizontal Pod Autoscaler (HPA) for client nodes to scale based on traffic.
    o Scale data nodes manually or use a custom autoscaler for StatefulSets.
  • Vertical Scaling:
    o Use Flex shapes for worker nodes to adjust CPU and memory as required.

10. Fault Tolerance and High Availability

  • Deploy nodes across multiple availability domains or fault domains.
  • Use anti-affinity rules to spread pods across nodes.
  • Regularly test cluster failover scenarios.

11. Elasticsearch Installation Options

You can deploy elasticsearch as operator or using HELM as described in the official documentation.

12. References

Autoscaling
https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengautoscalingclusters.htm


Cluster Access
https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengdownloadkubeconfigfile.htm


Compute Shapes
https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm


Creating Compartment:
https://docs.oracle.com/en-us/iaas/Content/Identity/compartments/To_create_a_compartment.htm


Elasticsearch:
https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html


IAM
https://docs.oracle.com/en-us/iaas/Content/Identity/Concepts/overview.htm


Kubernetes Cluster Creation:
https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengoverview.htm


Network Resource Configuration
https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengnetworkconfig.htm


OKE Cluster Type
https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcomparingenhancedwithbasicclusters_topic.htm


Oracle Cloud Service Limits:
https://docs.oracle.com/en-us/iaas/Content/General/Concepts/servicelimits.htm


Persistent volume:
https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingpersistentvolumeclaim.htm


Pod Networking
https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengpodnetworking.htm

RBAC
https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengaboutaccesscontrol.htm

Leave a Comment

Your email address will not be published. Required fields are marked *