Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Passing packer variables to specify custom overrides does not seem to work #2372

Open
vasyl-purchel opened this issue Aug 10, 2023 · 10 comments
Labels
bug Something isn't working

Comments

@vasyl-purchel
Copy link
Contributor

What happened:

Command in use to build Ubuntu 22.04 based EKSA BM image:

image-builder-dev4 build --os ubuntu --os-version 22.04 --hypervisor baremetal --release-channel 1-27 -v 3 --force --baremetal-config /home/image-builder/packer_vars.json

I have /home/image-builder/packer_vars.json file with overrides to use a custom Ansible role through image provisioning.
When I attempt to pass it with --baremetal-config /home/image-builder/packer_vars.json this path is nowhere to be seen in end command under PACKER_VAR_FILES environment variable.

Also I've noticed following warning during execution:

Makefile:46: warning: undefined variable 'PACKER__VAR_FILES'

which looks like comes from:

PACKER_TYPE_VAR_FILES?=$(PACKER_$(call TO_UPPER,$(IMAGE_FORMAT))_VAR_FILES)

where IMAGE_FORMAT is empty instead of raw.

I've also attempted to modify commandEnvVars for Ubuntu build with no impact:

                 case Ubuntu:
                         buildCommand = fmt.Sprintf("make -C %s local-build-raw-ubuntu-%s", imageBuilderProjectPath, b.OsVersion)
+                        commandEnvVars = append(commandEnvVars, fmt.Sprintf("PACKER__VAR_FILES='/home/image-builder/packer_vars.json'"), fmt.Sprintf("PACKER_RAW_VAR_FILES='/home/image-builder/packer_vars.json'")
                 case RedHat:

End result packer command executed:

/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/.local/bin/packer build -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/kubernetes.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/cni.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/containerd.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/wasm-shims.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/ansible-args.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/goss-args.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/common.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/config/additional_components.json"  -force -color=true -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/packer/raw/raw-ubuntu-2204-efi.json" -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2204/config/kubernetes.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2204/config/common.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2204/config/cni.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/_output/1-27/raw/ubuntu/2204/config/additional_components.json"  -var-file="/home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/packer/config/baremetal.json"  packer/raw/packer.json

What you expected to happen:

Being able to supply overrides from https://image-builder.sigs.k8s.io/capi/capi, for example:

{
        "disable_public_repos": true,
        "reenable_public_repos": false,
        "extra_repos": "/home/image-builder/ubuntu.sources.list"
}

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • EKS Anywhere Release: 1-27
  • EKS Distro Release: Ubuntu 22.04
  • EKS Anywhere image-builder: built from sources with support for 22.04
@vasyl-purchel vasyl-purchel added the bug Something isn't working label Aug 10, 2023
@jaxesn
Copy link
Member

jaxesn commented Aug 11, 2023

Thanks for the detailed report!

Quick question, do you have export EKSA_USE_DEV_RELEASE=true set?

@vasyl-purchel
Copy link
Contributor Author

Don't think I do:

$ echo $EKSA_USE_DEV_RELEASE

@jaxesn
Copy link
Member

jaxesn commented Aug 11, 2023

Ok I think that explains Makefile:46: warning: undefined variable 'PACKER__VAR_FILES', but I'm not convinced even if you set it that what you want is going to work. I am still investigating and will try to get something that we can test by Monday.

@jaxesn
Copy link
Member

jaxesn commented Aug 14, 2023

I did confirm we do not currently have a way to pass these kinds of additional options/overrides in yet. We are going to take a look internally at options we have and will get back to you with an idea of a timeline.

@abhay-krishna
Copy link
Member

abhay-krishna commented Aug 15, 2023

Also, the filepath passed in as baremetal-config is not used directly as a Packer var-file. It is unmarshaled and then fields are validated and then the content is re-marshaled into the location $HOME/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/packer/config/baremetal.json, which I see in the packer build command.

Are you able to see this file on your host? If so could you post its contents?

@abhay-krishna
Copy link
Member

abhay-krishna commented Aug 15, 2023

Also, the filepath passed in as baremetal-config is not used directly as a Packer var-file. It is unmarshaled and then fields are validated and then the content is re-marshaled into the location $HOME/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/packer/config/baremetal.json, which I see in the packer build command.

Are you able to see this file on your host? If so could you post its contents?

Ideally, you should see something like

{
    "files":null,
    "rhel_username":"",
    "rhel_password":"",
    "extra_repos":"/home/image-builder/ubuntu.sources.list"
}

@vasyl-purchel
Copy link
Contributor Author

hi, yes I do see that file under /home/image-builder/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/packer/config/baremetal.json, and contents are as follows:

{"rhel_username":"","rhel_password":"","extra_repos":"/home/image-builder/ubuntu.sources.list"}

@vasyl-purchel
Copy link
Contributor Author

so if this file is parsed I see the only options it supports is:

type BaremetalConfig struct {
	IsoConfig
	RhelConfig
	ExtraPackagesConfig
}
type ExtraPackagesConfig struct {
	ExtraDebs  string `json:"extra_debs,omitempty"`
	ExtraRepos string `json:"extra_repos,omitempty"`
	ExtraRpms  string `json:"extra_rpms,omitempty"`
}

how do other customers pass in custom provisioner after the build or anything else to customise the image? or is it just extra repos + extra packages the only option available?

@abhay-krishna
Copy link
Member

so if this file is parsed I see the only options it supports is:

type BaremetalConfig struct {
	IsoConfig
	RhelConfig
	ExtraPackagesConfig
}
type ExtraPackagesConfig struct {
	ExtraDebs  string `json:"extra_debs,omitempty"`
	ExtraRepos string `json:"extra_repos,omitempty"`
	ExtraRpms  string `json:"extra_rpms,omitempty"`
}

how do other customers pass in custom provisioner after the build or anything else to customise the image? or is it just extra repos + extra packages the only option available?

Currently we support only these parameters that are present in these Go struct types. We are working on a potential solution to expand this list and provide more customization through image-builder.

@vasyl-purchel
Copy link
Contributor Author

Hi, so I've been trying to get addition of ansible roles:

type BaremetalConfig struct {
	AdditionalFiles []File `json:"files"`
	IsoConfig
	RhelConfig
	ExtraPackagesConfig
	ExtraRolesConfig
}

type ExtraRolesConfig struct {
	firstboot_custom_roles_pre  string `json:"firstboot_custom_roles_pre"`
	firstboot_custom_roles_post string `json:"firstboot_custom_roles_post"`
	node_custom_roles_pre       string `json:"node_custom_roles_pre"`
	node_custom_roles_post      string `json:"node_custom_roles_post"`
	roles_from                  string `json:"roles_from"`
}

to match what upstream kubernetes-sigs support and building locally it I'm still not getting necessary content from the config:

{
        "extra_repos": "/home/image-builder/ubuntu.sources.list",
        "roles_from": "/home/image-builder/wd_ansible_roles",
        "node_custom_roles_pre": "wd.eksa.pre_node"
}

but still variable b.BaremetalConfig.ExtraRolesConfig.roles_from is empty... any points at what am I missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants