Skip to content

Latest commit

 

History

History
807 lines (595 loc) · 35.5 KB

TESTING.asciidoc

File metadata and controls

807 lines (595 loc) · 35.5 KB

Testing

Creating packages

To build a distribution for your local OS and print its output location upon completion, run:

./gradlew localDistro

To create a platform-specific build, use the following depending on your operating system:

./gradlew :distribution:archives:linux-tar:assemble
./gradlew :distribution:archives:darwin(-aarch64)-tar:assemble
./gradlew :distribution:archives:windows-zip:assemble

You can build a Docker image with:

./gradlew build(Aarch64)DockerImage

Note: you almost certainly don’t want to run ./gradlew assemble as this will attempt build every single Elasticsearch distribution.

Running Elasticsearch from a checkout

In order to run Elasticsearch from source without building a package, you can run it using Gradle:

./gradlew run

Launching and debugging from an IDE

If you want to run and debug Elasticsearch from your IDE, the ./gradlew run task supports a remote debugging option. Run the following from your terminal:

./gradlew run --debug-jvm

Next start the "Debug Elasticsearch" run configuration in IntelliJ. This will enable the IDE to connect to the process and allow debug functionality.

As such the IDE needs to be instructed to listen for connections on the debug port. Since we might run multiple JVMs as part of configuring and starting the cluster it’s recommended to configure the IDE to initiate multiple listening attempts. In case of IntelliJ, this option is called "Auto restart" and needs to be checked.

Note
If you have imported the project into IntelliJ according to the instructions in CONTRIBUTING.md then a debug run configuration named "Debug Elasticsearch" will be created for you and configured appropriately.
Debugging the CLI launcher

The gradle task does not start the Elasticsearch server process directly; like in the Elasticsearch distribution, the job of starting the server process is delegated to a launcher CLI tool. If you need to debug the launcher itself, add the following option to the run task:

./gradlew run --debug-cli-jvm

This option can be specified in isolation or combined with --debug-jvm. Since the CLI launcher lifespan may overlap with the server process lifespan, the CLI launcher process will be started on a different port (5107 for the first node, 5108 and following for additional cluster nodes).

As with the --debug-jvm command, the IDE needs to be instructed to listen for connections on the debug port. You need to configure and start an appropriate Remote JVM Debug configuration, e.g. by cloning and editing the "Debug Elasticsearch" run configuration to point to the correct debug port.

Disabling assertions

When running Elasticsearch with ./gradlew run, assertions are enabled by default. To disable them, add the following command line option:

-Dtests.jvm.argline="-da -dsa"

Distribution

By default a node is started with the zip distribution. In order to start with a different distribution use the -Drun.distribution argument.

To for example start the open source distribution:

./gradlew run -Drun.distribution=oss

License type

By default a node is started with the basic license type. In order to start with a different license type use the -Drun.license_type argument.

In order to start a node with a trial license execute the following command:

./gradlew run -Drun.license_type=trial

This enables security and other paid features and adds a superuser with the username: elastic-admin and password: elastic-password.

Other useful arguments

  • In order to start a node with a different max heap space add: -Dtests.heap.size=4G

  • In order to use a custom data directory: --data-dir=/tmp/foo

  • In order to preserve data in between executions: --preserve-data

  • In order to remotely attach a debugger to the server process: --debug-jvm

  • In order to remotely attach a debugger to the CLI launcher process: --debug-cli-jvm

  • In order to set a different keystore password: --keystore-password

  • In order to set an Elasticsearch setting, provide a setting with the following prefix: -Dtests.es.

  • In order to pass a JVM setting, e.g. to disable assertions: -Dtests.jvm.argline="-da"

  • In order to use HTTPS: ./gradlew run --https

  • In order to start a mock logging APM server on port 9999 and configure ES cluster to connect to it, use ./gradlew run --with-apm-server

Customizing the test cluster for ./gradlew run

You may need to customize the cluster configuration for the ./gradlew run task. The settings can be set via the command line, but other options require updating the task itself. You can simply find the task in the source code and configure it there. (The task is currently defined in build-tools-internal/src/main/groovy/elasticsearch.run.gradle) However, this requires modifying a source controlled file and is subject to accidental commits. Alternatively, you can use a Gradle init script to inject custom build logic with the -I flag to configure this task locally.

For example:

To use a custom certificate for HTTPS with ./gradlew run, you can do the following. Create a file (for example ~/custom-run.gradle) with the following contents:

rootProject {
  if(project.name == 'elasticsearch') {
    afterEvaluate {
        testClusters.matching { it.name == "runTask"}.configureEach {
          extraConfigFile 'http.p12', file("<path/to/file>/http.p12")
        }
    }
  }
}

Now tell Gradle to use this init script:

./gradlew run -I ~/custom-run.gradle \
-Dtests.es.xpack.security.http.ssl.enabled=true \
-Dtests.es.xpack.security.http.ssl.keystore.path=http.p12

Now the http.p12 file will be placed in the config directory of the running cluster and available for use. Assuming you have the http.ssl.keystore setup correctly, you can now use HTTPS with ./gradlew run without the risk of accidentally committing your local configurations.

Multiple nodes in the test cluster for ./gradlew run

Another desired customization for ./gradlew run might be to run multiple nodes with different setting for each node. For example, you may want to debug a coordinating only node that fans out to one or more data nodes. To do this, increase the numberOfNodes and add specific configuration for each of the nodes. For example, the following will instruct the first node (:9200) to be a coordinating only node, and all other nodes to be master, data_hot, data_content nodes.

testClusters.register("runTask") {
     ...
    numberOfNodes = 2
    def cluster = testClusters.named("runTask").get()
    cluster.getNodes().each { node ->
      node.setting('cluster.initial_master_nodes', cluster.getLastNode().getName())
      node.setting('node.roles', '[master,data_hot,data_content]')
    }
    cluster.getFirstNode().setting('node.roles', '[]')
   ...
}

You can also place this config in custom init script (see above) to avoid accidental commits. If you are passing in the --debug-jvm flag with multiple nodes, you will need multiple remote debuggers running. One for each node listening at port 5007, 5008, 5009, and so on. Ensure that each remote debugger has auto restart enabled.

Use ./gradlew run-ccs to launch 2 clusters wired together for the purposes of cross cluster search. For example send a search request "my_remote_cluster:*/_search" to the querying cluster (:9200) to query data in the fulfilling cluster.

If you are passing in the --debug-jvm flag, you will need two remote debuggers running. One at port 5007 and another one at port 5008. Ensure that each remote debugger has auto restart enabled.

Test case filtering.

You can run a single test, provided that you specify the Gradle project. See the documentation on simple name pattern filtering.

Run a single test case in the server project:

./gradlew :server:test --tests org.elasticsearch.package.ClassName

Run all tests in a package and its sub-packages:

./gradlew :server:test --tests 'org.elasticsearch.package.*'

Run all tests that are waiting for a bugfix (disabled by default)

./gradlew test -Dtests.filter=@awaitsfix

Seed and repetitions.

Run with a given seed (seed is a hex-encoded long).

./gradlew test -Dtests.seed=DEADBEEF

Repeats all tests of ClassName N times.

Every test repetition will have a different method seed (derived from a single random master seed).

./gradlew :server:test -Dtests.iters=N --tests org.elasticsearch.package.ClassName

Repeats all tests of ClassName N times.

Every test repetition will have exactly the same master (0xdead) and method-level (0xbeef) seed.

./gradlew :server:test -Dtests.iters=N -Dtests.seed=DEAD:BEEF --tests org.elasticsearch.package.ClassName

Repeats a given test N times

(note the filters - individual test repetitions are given suffixes, ie: testFoo[0], testFoo[1], etc…​ so using testmethod or tests.method ending in a glob is necessary to ensure iterations are run).

./gradlew :server:test -Dtests.iters=N --tests org.elasticsearch.package.ClassName.methodName

Repeats N times but skips any tests after the first failure or M initial failures.

./gradlew test -Dtests.iters=N -Dtests.failfast=true ...
./gradlew test -Dtests.iters=N -Dtests.maxfailures=M ...

Test groups.

Test groups can be enabled or disabled (true/false).

Default value provided below in [brackets].

./gradlew test -Dtests.awaitsfix=[false] - known issue (@AwaitsFix)

Load balancing and caches.

By default the tests run on multiple processes using all the available cores on all available CPUs. Not including hyper-threading. If you want to explicitly specify the number of JVMs you can do so on the command line:

./gradlew test -Dtests.jvms=8

Or in ~/.gradle/gradle.properties:

systemProp.tests.jvms=8

It’s difficult to pick the "right" number here. Hypercores don’t count for CPU intensive tests and you should leave some slack for JVM-internal threads like the garbage collector. And you have to have enough RAM to handle each JVM.

Test compatibility.

It is possible to provide a version that allows to adapt the tests behaviour to older features or bugs that have been changed or fixed in the meantime.

./gradlew test -Dtests.compatibility=1.0.0

Miscellaneous.

Run all tests without stopping on errors (inspect log files).

./gradlew test -Dtests.haltonfailure=false

Run more verbose output (slave JVM parameters, etc.).

./gradlew test -verbose

Change the default suite timeout to 5 seconds for all tests (note the exclamation mark).

./gradlew test -Dtests.timeoutSuite=5000! ...

Change the logging level of ES (not Gradle)

./gradlew test -Dtests.es.logger.level=DEBUG

Print all the logging output from the test runs to the commandline even if tests are passing.

./gradlew test -Dtests.output=always

Configure the heap size.

./gradlew test -Dtests.heap.size=512m

Pass arbitrary jvm arguments.

# specify heap dump path
./gradlew test -Dtests.jvm.argline="-XX:HeapDumpPath=/path/to/heapdumps"
# enable gc logging
./gradlew test -Dtests.jvm.argline="-verbose:gc"
# enable security debugging
./gradlew test -Dtests.jvm.argline="-Djava.security.debug=access,failure"

Pass build arguments.

# Run tests against a release build. License key must be provided, but usually can be anything.
./gradlew test -Dbuild.snapshot=false -Dlicense.key="x-pack/license-tools/src/test/resources/public.key"

Running verification tasks

To run all verification tasks, including static checks, unit tests, and integration tests:

./gradlew check

Note that this will also run the unit tests and precommit tasks first. If you want to just run the in memory cluster integration tests (because you are debugging them):

./gradlew internalClusterTest

If you want to just run the precommit checks:

./gradlew precommit

Some of these checks will require docker-compose installed for bringing up test fixtures. If it’s not present those checks will be skipped automatically. The host running Docker (or VM if you’re using Docker Desktop) needs 4GB of memory or some of the containers will fail to start. You can tell that you are short of memory if containers are exiting quickly after starting with code 137 (128 + 9, where 9 means SIGKILL).

Debugging tests

If you would like to debug your tests themselves, simply pass the --debug-jvm flag to the testing task and connect a debugger on the default port of 5005.

./gradlew :server:test --debug-jvm

For REST tests, if you’d like to debug the Elasticsearch server itself, and not your test code, use the --debug-server-jvm flag and use the "Debug Elasticsearch" run configuration in IntelliJ to listen on the default port of 5007.

./gradlew :rest-api-spec:yamlRestTest --debug-server-jvm
Note
In the case of test clusters using multiple nodes, multiple debuggers will need to be attached on incrementing ports. For example, for a 3 node cluster ports 5007, 5008, and 5009 will attempt to attach to a listening debugger. You can use the "Debug Elasticsearch (node 2)" and "(node 3)" run configurations should you need to debug a multi-node cluster.

You can also use a combination of both flags to debug both tests and server. This is only applicable to Java REST tests.

./gradlew :modules:kibana:javaRestTest --debug-jvm --debug-server-jvm

Testing the REST layer

The REST layer is tested through specific tests that are executed against a cluster that is configured and initialized via Gradle. The tests themselves can be written in either Java or with a YAML based DSL.

YAML based REST tests should be preferred since these are shared between all the elasticsearch official clients. The YAML based tests describe the operations to be executed and the obtained results that need to be tested.

The YAML tests support various operators defined in the rest-api-spec and adhere to the Elasticsearch REST API JSON specification In order to run the YAML tests, the relevant API specification needs to be on the test classpath. Any gradle project that has support for REST tests will get the primary API on it’s class path. However, to better support Gradle incremental builds, it is recommended to explicitly declare which parts of the API the tests depend upon.

For example:

restResources {
  restApi {
    includeCore '_common', 'indices', 'index', 'cluster', 'nodes', 'get', 'ingest'
  }
}

YAML REST tests that include x-pack specific APIs need to explicitly declare which APIs are required through a similar includeXpack configuration.

The REST tests are run automatically when executing the "./gradlew check" command. To run only the YAML REST tests use the following command (modules and plugins may also include YAML REST tests):

./gradlew :rest-api-spec:yamlRestTest

A specific test case can be run with the following command:

./gradlew ':rest-api-spec:yamlRestTest' \
  --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT" \
  -Dtests.method="test {yaml=cat.segments/10_basic/Help}"

You can run a group of YAML test by using wildcards:

./gradlew :rest-api-spec:yamlRestTest \
  --tests "org.elasticsearch.test.rest.ClientYamlTestSuiteIT.test {yaml=index/*/*}"

or

./gradlew :rest-api-spec:yamlRestTest \
  --tests org.elasticsearch.test.rest.ClientYamlTestSuiteIT -Dtests.method="test {yaml=cat.segments/10_basic/*}"

The latter method is preferable when the YAML suite name contains . (period).

Note that if the selected test via the --tests filter is not a valid test, i.e., the YAML test runner is not able to parse and load it, you might get an error message indicating that the test was not found. In such cases, running the whole suite without using the --tests could show more specific error messages about why the test runner is not able to parse or load a certain test.

The YAML REST tests support all the options provided by the randomized runner, plus the following:

  • tests.rest.blacklist: comma separated globs that identify tests that are blacklisted and need to be skipped e.g. -Dtests.rest.blacklist=index//Index document,get/10_basic/

Java REST tests can be run with the "javaRestTest" task.

For example :

./gradlew :modules:mapper-extras:javaRestTest

A specific test case can be run with the following syntax (fqn.test {params}):

./gradlew ':modules:mapper-extras:javaRestTest' \
  --tests "org.elasticsearch.index.mapper.TokenCountFieldMapperIntegrationIT.testSearchByTokenCount {storeCountedFields=true loadCountedFields=false}"

yamlRestTest’s and javaRestTest’s are easy to identify, since they are found in a respective source directory. However, there are some more specialized REST tests that use custom task names. These are usually found in "qa" projects commonly use the "integTest" task.

If in doubt about which command to use, simply run <gradle path>:check

Testing packaging

The packaging tests are run on different build vm cloud instances to verify that installing and running Elasticsearch distributions works correctly on supported operating systems. These tests should really only be run on ephemeral systems because they’re destructive; that is, these tests install and remove packages and freely modify system settings, so you will probably regret it if you execute them on your development machine.

Reproducing packaging tests

To reproduce or debug packaging tests failures we recommend using using our provided buildkite tools

Testing backwards compatibility

Backwards compatibility tests exist to test upgrading from each supported version to the current version. To run them all use:

./gradlew bwcTest

A specific version can be tested as well. For example, to test bwc with version 5.3.2 run:

./gradlew v5.3.2#bwcTest

Use -Dtests.class and -Dtests.method to run a specific bwcTest test. For example to run a specific tests from the x-pack rolling upgrade from 7.7.0:

./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \
 -Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
 -Dtests.method="test {p0=*/40_ml_datafeed_crud/*}"

Tests are ran for versions that are not yet released but with which the current version will be compatible with. These are automatically checked out and built from source. See BwcVersions and distribution/bwc/build.gradle for more information.

When running ./gradlew check, minimal bwc checks are also run against compatible versions that are not yet released.

BWC Testing against a specific remote/branch

Sometimes a backward compatibility change spans two versions. A common case is a new functionality that needs a BWC bridge in an unreleased versioned of a release branch (for example, 5.x). Another use case, since the introduction of serverless, is to test BWC against main in addition to the other released branches. To do so, specify the bwc.refspec remote and branch to use for the BWC build as origin/main. To test against main, you will also need to create a new version in Version.java, increment elasticsearch in version.properties, and hard-code the project.version for ml-cpp in ml/build.gradle.

In general, to test the changes, you can instruct Gradle to build the BWC version from another remote/branch combination instead of pulling the release branch from GitHub. You do so using the bwc.refspec.{VERSION} system property:

./gradlew check -Dtests.bwc.refspec.8.15=origin/main

The branch needs to be available on the remote that the BWC makes of the repository you run the tests from. Using the remote is a handy trick to make sure that a branch is available and is up to date in the case of multiple runs.

Example:

Say you need to make a change to main and have a BWC layer in 5.x. You will need to: . Create a branch called index_req_change off your remote ${remote}. This will contain your change. . Create a branch called index_req_bwc_5.x off 5.x. This will contain your bwc layer. . Push both branches to your remote repository. . Run the tests with ./gradlew check -Dbwc.remote=${remote} -Dbwc.refspec.5.x=index_req_bwc_5.x.

Skip fetching latest

For some BWC testing scenarios, you want to use the local clone of the repository without fetching latest. For these use cases, you can set the system property tests.bwc.git_fetch_latest to false and the BWC builds will skip fetching the latest from the remote.

Testing in FIPS 140-2 mode

We have a CI matrix job that periodically runs all our tests with the JVM configured to be FIPS 140-2 compliant with the use of the BouncyCastle FIPS approved Security Provider. FIPS 140-2 imposes certain requirements that affect how our tests should be set up or what can be tested. This section summarizes what one needs to take into consideration so that tests won’t fail when run in fips mode.

Muting tests in FIPS 140-2 mode

If the following limitations cannot be observed, or there is a need to actually test some use case that is not available/allowed in fips mode, the test can be muted. For unit tests or Java rest tests one can use

assumeFalse("Justification why this cannot be run in FIPS mode", inFipsJvm());

For specific YAML rest tests one can use

- skip:
    features: fips_140
    reason: "Justification why this cannot be run in FIPS mode"

For disabling entire types of tests for subprojects, one can use for example:

if (BuildParams.inFipsJvm){
  // This test cluster is using a BASIC license and FIPS 140 mode is not supported in BASIC
  tasks.named("javaRestTest").configure{enabled = false }
}

in build.gradle.

Limitations

The following should be taken into consideration when writing new tests or adjusting existing ones:

TLS

JKS and PKCS#12 keystores cannot be used in FIPS mode. If the test depends on being able to use a keystore, it can be muted when needed ( see ESTestCase#inFipsJvm ). Alternatively, one can use PEM encoded files for keys and certificates for the tests or for setting up TLS in a test cluster. Also, when in FIPS 140 mode, hostname verification for TLS cannot be turned off so if you are using *.verification_mode: none , you’d need to mute the test in fips mode.

When using TLS, ensure that private keys used are longer than 2048 bits, or mute the test in fips mode.

Password hashing algorithm

Test clusters are configured with xpack.security.fips_mode.enabled set to true. This means that FIPS 140-2 related bootstrap checks are enabled and the test cluster will fail to form if the password hashing algorithm is set to something else than a PBKDF2 based one. You can delegate the choice of algorithm to i.e. SecurityIntegTestCase#getFastStoredHashAlgoForTests if you don’t mind the actual algorithm used, or depend on default values for the test cluster nodes.

Password length

While using pbkdf2 as the password hashing algorithm, FIPS 140-2 imposes a requirement that passwords are longer than 14 characters. You can either ensure that all test user passwords in your test are longer than 14 characters and use i.e. SecurityIntegTestCase#getFastStoredHashAlgoForTests to randomly select a hashing algorithm, or use pbkdf2_stretch that doesn’t have the same limitation.

Keystore Password

In FIPS 140-2 mode, the elasticsearch keystore needs to be password protected with a password of appropriate length. This is handled automatically in fips.gradle and the keystore is unlocked on startup by the test clusters tooling in order to have secure settings available. However, you might need to take into consideration that the keystore is password-protected with keystore-password if you need to interact with it in a test.

How to write good tests?

Base classes for test cases

There are multiple base classes for tests:

  • ESTestCase: The base class of all tests. It is typically extended directly by unit tests.

  • ESSingleNodeTestCase: This test case sets up a cluster that has a single node.

  • ESIntegTestCase: An integration test case that creates a cluster that might have multiple nodes.

  • ESRestTestCase: An integration tests that interacts with an external cluster via the REST API. This is used for Java based REST tests.

  • ESClientYamlSuiteTestCase : A subclass of ESRestTestCase used to run YAML based REST tests.

Good practices

What kind of tests should I write?

Unit tests are the preferred way to test some functionality: most of the time they are simpler to understand, more likely to reproduce, and unlikely to be affected by changes that are unrelated to the piece of functionality that is being tested.

The reason why ESSingleNodeTestCase exists is that all our components used to be very hard to set up in isolation, which had led us to having a number of integration tests but close to no unit tests. ESSingleNodeTestCase is a workaround for this issue which provides an easy way to spin up a node and get access to components that are hard to instantiate like IndicesService. Whenever practical, you should prefer unit tests.

Many tests extend ESIntegTestCase, mostly because this is how most tests used to work in the early days of Elasticsearch. However the complexity of these tests tends to make them hard to debug. Whenever the functionality that is being tested isn’t intimately dependent on how Elasticsearch behaves as a cluster, it is recommended to write unit tests or REST tests instead.

In short, most new functionality should come with unit tests, and optionally REST tests to test integration.

Refactor code to make it easier to test

Unfortunately, a large part of our code base is still hard to unit test. Sometimes because some classes have lots of dependencies that make them hard to instantiate. Sometimes because API contracts make tests hard to write. Code refactors that make functionality easier to unit test are encouraged. If this sounds very abstract to you, you can have a look at this pull request for instance, which is a good example. It refactors IndicesRequestCache in such a way that: - it no longer depends on objects that are hard to instantiate such as IndexShard or SearchContext, - time-based eviction is applied on top of the cache rather than internally, which makes it easier to assert on what the cache is expected to contain at a given time.

Bad practices

Use randomized-testing for coverage

In general, randomization should be used for parameters that are not expected to affect the behavior of the functionality that is being tested. For instance the number of shards should not impact date_histogram aggregations, and the choice of the store type (niofs vs mmapfs) does not affect the results of a query. Such randomization helps improve confidence that we are not relying on implementation details of one component or specifics of some setup.

However it should not be used for coverage. For instance if you are testing a piece of functionality that enters different code paths depending on whether the index has 1 shards or 2+ shards, then we shouldn’t just test against an index with a random number of shards: there should be one test for the 1-shard case, and another test for the 2+ shards case.

Abuse randomization in multi-threaded tests

Multi-threaded tests are often not reproducible due to the fact that there is no guarantee on the order in which operations occur across threads. Adding randomization to the mix usually makes things worse and should be done with care.

Test coverage analysis

Generating test coverage reports for Elasticsearch is currently not possible through Gradle. However, it is possible to gain insight in code coverage using IntelliJ’s built-in coverage analysis tool that can measure coverage upon executing specific tests.

Test coverage reporting used to be possible with JaCoCo when Elasticsearch was using Maven as its build system. Since the switch to Gradle though, this is no longer possible, seeing as the code currently used to build Elasticsearch does not allow JaCoCo to recognize its tests. For more information on this, see the discussion in issue #28867.

Building with extra plugins

Additional plugins may be built alongside elasticsearch, where their dependency on elasticsearch will be substituted with the local elasticsearch build. To add your plugin, create a directory called elasticsearch-extra as a sibling of elasticsearch. Checkout your plugin underneath elasticsearch-extra and the build will automatically pick it up. You can verify the plugin is included as part of the build by checking the projects of the build.

./gradlew projects

Environment misc

There is a known issue with macOS localhost resolve strategy that can cause some integration tests to fail. This is because integration tests have timings for cluster formation, discovery, etc. that can be exceeded if name resolution takes a long time. To fix this, make sure you have your computer name (as returned by hostname) inside /etc/hosts, e.g.:

127.0.0.1       localhost ElasticMBP.local
255.255.255.255 broadcasthost
::1             localhost ElasticMBP.local`

Benchmarking

For changes that might affect the performance characteristics of Elasticsearch you should also run macrobenchmarks. We maintain a macrobenchmarking tool called Rally which you can use to measure the performance impact. It comes with a set of default benchmarks that we also run every night. To get started, please see Rally’s documentation.

Test doc builds

The Elasticsearch docs are in AsciiDoc format. You can test and build the docs locally using the Elasticsearch documentation build process. See https://github.com/elastic/docs.