Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Design of an inherited xarray duck array test suite #6

Open
TomNicholas opened this issue Aug 22, 2024 · 2 comments
Open

Design of an inherited xarray duck array test suite #6

TomNicholas opened this issue Aug 22, 2024 · 2 comments

Comments

@TomNicholas
Copy link
Member

TomNicholas commented Aug 22, 2024

Hi @Zac-HD and @tomwhite 👋

@keewis and I spent today trying to design a duck array test suite for xarray using hypothesis. This (temporary) repo contains the sketch of what we're trying to do, with the eventual aim being that this sort of code lives upstream in xarray and in downstream duck array libraries like pint/dask/sparse/cubed etc.

The problem we are trying to solve is that we want to create a test suite that can be used to test xarray's wrapping of any duck array type, including cubed and pint as representative examples. We want this test suite to:

  1. Specify expected behaviour of an xarray-wrapped duck array. i.e. once complete, if the suite passes, we can be confident that xarray users can use xarray with that duck array type without issues. Note this is not the same as just checking that the duck array obeys the array API standard (there are things in xarray that aren't in the standard, and things in the standard that aren't in xarray).
  2. Be defined upstream in xarray, so that we have control over what features are tested downstream. For example if xarray added support for scipy.skew via a new reduction method da.skew, we would want to be able to add a test to this to the test suite for duck array libraries just by changing code in the xarray repository.
  3. Be inherited by downstream duck array libraries (either in their main packages or in glue packages such as cubed-xarray and pint-xarray). That way xarray devs don't have to maintain all the tests, and failures are first reported downstream, and it's on the devs of those packages to report any failures upstream in xarray if they think its actually xarray's fault.
  4. Test a wide variety of cases with few lines of code. For example we want the downstream tester to just be able to import a TestDatasetReductions class and that automatically runs many different reductions on many different xarray objects.
  5. Allow the downstream tester to specify test behaviour unique to their duck array type, including:
    • array creation (e.g. you need to use the downstream package's array creation functions, and may want to parameterize over additional options such as chunking pattern)
    • array result comparison (e.g. pint requires calling .magnitude, and cubed/dask requires calling .compute)
  6. Allow the downstream tester to mark certain parameter combinations as expected failures. For example cubed.mean() currently doesn't support taking means of integers (because it's not actually required by the array API standard), but xarray.DataArray.mean() expects this to be possible. So we want to the test suite to test means of integers but give cubed's downstream tests the opportunity to mark that case as an expected failure.

(4) is the reason why we are using hypothesis, and why we made the hypothesis strategies for generating arbitrary xarray.Variable objects.

(4), (5) and (6) are the most difficult parts of this to achieve simultaneously. We need a lot of control over test cases upstream in xarray, but also give a lot of control to the downstream tester to override things.

We would appreciate it if you could take a look at what we have done (both in main and in #4) and tell us if you think we are headed in a good direction or not?

xref pydata/xarray#4972 pydata/xarray#4972 pydata/xarray#6908 cubed-dev/cubed-xarray#20

@tomwhite
Copy link

+1 for the direction.

Presumably the code in #4 will eventually end up in cubed-xarray, but you are keeping it here while the design evolves?

@keewis
Copy link
Contributor

keewis commented Aug 28, 2024

yes, indeed, it helps to have multiple examples while writing up the actual tests. So far we also expect to move to xarray and archive this repository once we're confident that the structure of the testing framework is okay.

By the way, the tests in #4 already exposed some issues in cubed / cubed-xarray (not sure which, could also be xarray).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants