Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integration test #7

Open
maxspahn opened this issue Feb 16, 2022 · 3 comments
Open

Add integration test #7

maxspahn opened this issue Feb 16, 2022 · 3 comments
Labels
enhancement New feature or request

Comments

@maxspahn
Copy link
Collaborator

Make sure that basic functionality is not destroyed by changes made during structure improvement.
This can be done with GitHub actions and a CI.

@maxspahn maxspahn added the enhancement New feature or request label Feb 16, 2022
@niketagrawal niketagrawal changed the title Add simple unit test Add integration test Feb 16, 2022
@niketagrawal
Copy link
Collaborator

niketagrawal commented Feb 23, 2022

Approach to writing the integration/blackbox test: Compare the output of running an experiment with a specific type of planner and robot with the expected output.

The planners are deterministic, so the planners are expected to produce the same output for the same input configuration. Some of the fields are non-deterministic; they produce values that depend on the platform or OS that the experiment is run on. For the integration tests only the deterministic output fields can be compared each time.

Requirements

  • Input configurations and results produced by the planners for those configurations.
  • Pseudo code/logic to run experiments, store results in temporary directories, compare them against expected output, and delete the temporary result directories.

Workflow actions to be tested:

  • Test if the runner command resulted in creation of a results folder with the expected no. and names of the files: res.csv, exp.yml, etc.
  • test if the deterministic contents of the results produced are the same for a specific robot type and experiment.
  • The test can be made more robust by testing runner execution and results produced for different combinations such as one planner and one robot type, two panners and one robot type, one planner and two robot typs, etc.

The integration test can be developed for the example planner now. It is not possible to use fabric and mpc planner in the tests because these planner are not open source; they can't be installed and run by GitHub actions. In future, open source versions of these two planners could be developed.

@maxspahn
Copy link
Collaborator Author

maxspahn commented Mar 2, 2022

Introduced initial tests in a1c69c6

@maxspahn
Copy link
Collaborator Author

maxspahn commented Mar 4, 2022

See #12 for switching from bash script to pytest only syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants