An Automated Method for Conducting an In-Place Engine Upgrade in Amazon OpenSearch Service

More Info

Software upgrades are essential for introducing new features, enhancing performance, and ensuring alignment with your software provider’s offerings. Yet, executing upgrades on software services can often be challenging—especially in environments where downtime is unacceptable and where the new version may contain breaking changes or deprecated features that require adjustments. This article outlines how to upgrade from the Elasticsearch engine to the OpenSearch engine on Amazon OpenSearch Service without necessitating an intermediate upgrade to Elasticsearch 7.10.

OpenSearch Service accommodates OpenSearch as an engine, with available versions in the 1.x to 2.x range. Additionally, it supports legacy Elasticsearch versions, from 1.x to 7.10. Despite the enhancements that OpenSearch offers compared to its predecessors, the prospect of upgrading both versions and engines can seem overwhelming. Fortunately, OpenSearch 1.0 is wire compatible with Elasticsearch 7.10, which simplifies the engine transition. If your current Elasticsearch version falls within the 6.x or early 7.x series on OpenSearch Service, you might assume the need to first upgrade to Elasticsearch 7.10 before moving on to OpenSearch 1.3. However, upgrading directly from Elasticsearch 6.8, 7.1, 7.2, 7.4, 7.9, or 7.10 to OpenSearch 1.3 is entirely feasible.

Before initiating an upgrade, OpenSearch Service conducts several checks:

1. Validation prior to the upgrade process
2. Configuration preparation for the intended version
3. Provisioning of new nodes with equivalent hardware specifications
4. Transferring shards from older nodes to the newly provisioned nodes
5. Dismissing older nodes and their references from OpenSearch endpoints

During the upgrade, AWS manages the intricate tasks of provisioning, deployment, and data migration to the new domain. Your responsibility is to ensure that no breaking changes hinder the data migration process to the updated OpenSearch domain. This article discusses the modifications and verifications needed before and after upgrading from Elasticsearch versions 6.8, 7.1, 7.2, 7.4, 7.9, and 7.10 to OpenSearch Service 1.3.

Pre-upgrade Breaking Changes

Several breaking changes must be acknowledged prior to upgrading:

• Dependency check for language clients and libraries: If you are utilizing the open-source high-level language clients from Elastic—such as the Java, Go, or Python client libraries—AWS suggests transitioning to the open-source OpenSearch versions. (If high-level language clients are not in use, this step can be skipped.) Here’s how to perform a dependency check:

1. Identify the client library: Select a suitable client library that works with your programming language. For a comprehensive list of supported client libraries, refer to OpenSearch language clients.
2. Add dependencies and resolve conflicts: Update your project’s dependency management system with the required dependencies from the client library. Be aware that conflicts may arise if existing dependencies clash with those of the OpenSearch client library, necessitating manual resolution.
3. Test and validate the client: Establish a connection to the OpenSearch client, perform basic operations (such as indexing and searching), and confirm the results.

• Removal of mapping types: The presence of multiple types within an index was deprecated in Elasticsearch version 6.x and entirely eliminated in version 7.0 and beyond. OpenSearch indexes are limited to a single mapping type. From OpenSearch version 2.x onwards, the mapping type must be _doc. It is necessary to inspect and correct the mapping prior to upgrading to OpenSearch 1.3.

To identify and resolve mapping issues, follow these steps:

1. Access dev tools and use the GET <index> mapping API to retrieve mapping information for all indexes:

GET /index-name/_mapping

The mapping response will provide a JSON structure representing the mapping for your index.

2. Inspect the top-level keys in the response JSON, where each key signifies a custom type within the index.
3. The _doc type serves as the default type in Elasticsearch 7.x and OpenSearch Service 1.x. However, you may discover additional types defined in prior Elasticsearch versions.

To address the multiple mapping types in your current domain, you will need to reindex the data, creating a separate index for each mapping. This is crucial since OpenSearch does not permit multiple types within a single index. Below is an example of how to convert an index with multiple mapping types into two distinct indexes, each using the _doc type:

# Create an index for "type1"
PUT /myindex_type1

# Create an index for "type2"
PUT /myindex_type2

Utilize the _reindex API to transfer data from the original index to the two new indexes. Alternatively, you may reload the data from its source if it is stored in another system.

POST _reindex
{
  "source": {
    "index": "myindex",
    "type": "type1" 
  },
  "dest": {
    "index": "myindex_type1",
    "type": "_doc" 
  }
}
POST _reindex
{
  "source": {
    "index": "myindex",
    "type": "type2" 
  },
  "dest": {
    "index": "myindex_type2",
    "type": "_doc" 
  }
}

After reindexing, any application that previously queried the original index with multiple types will need to be updated to reference the new indexes, with _doc specified as the type. For example, if your client was querying myindex, which has now been reindexed to myindex_type1 and myindex_type2, adjust your queries accordingly to target myindex*, which will search across both indexes.

For further insights, check out another informative blog post here. Additionally, for more authoritative guidelines, refer to this resource. Moreover, if you’re looking for a visual guide, here’s an excellent resource on YouTube.