I’m a big fan of linters. They detect problems earlier (also known as “shifting to the left”), and the earlier problem detection is, the more efficient remediation is. Therefore, I want to lint as much as possible. Lately, I’ve been working a lot with GitLab CI YAML which oftentimes has shell script embedded in it which naturally would benefit from linting. So I figured out how to do so using a combination of yq and shellcheck as demonstrated by this GitLab CI job:
shellcheck:
image: alpine:3.19.1
script:
- |
# shellcheck shell=sh
# Check *.sh
git ls-files --exclude='*.sh' --ignored -c -z | xargs -0r shellcheck -P SCRIPTDIR -x
# Check files with a shell shebang
git ls-files -c | while IFS= read -r file; do
if head -n1 "${file}" |grep -q "^#\\! \?/.\+\(ba|d|k\)\?sh" ; then
shellcheck -P SCRIPTDIR -x "${file}"
fi
done
# Check script embedded in GitLab CI YAML named *.gitlab-ci.yml
newline="$(printf '\nq')"
newline=${newline%q}
git ls-files --exclude='*.gitlab-ci.yml' --ignored -c | while IFS= read -r file; do
documentCount=$(yq eval-all '[.] | length' "${file}")
documentIndex=-1
while [ "$documentIndex" -lt "$documentCount" ]; do
true $((documentIndex=documentIndex+1))
yq eval 'select(documentIndex == '${documentIndex}') | .[] | select(tag=="!!map") | (.before_script,.script,.after_script) | select(. != null ) | path | "select(documentIndex == '${documentIndex}') | .[\"" + join("\"].[\"") + "\"]"' "${file}" | while IFS= read -r selector; do
set +e
script=$(yq eval "${selector} | join(\"${newline}\")" "${file}")
status=$?
set -e
if [ $status -ne 0 ]; then
>&2 printf "\nError getting the contents of the selector %s in the file %s:\n\nThe YAML may be malformed." "${selector}" "${file}"
exit 1
fi
if ! printf '%s' "${script}" | shellcheck -x -; then
>&2 printf "\nError in %s in the script specified in %s:\n%s\n" "${file}" "${selector}" "${script}"
exit 1
fi
done
done
done
This GitLab CI job will run shellcheck on:
*.shfiles- files with a shell shebang
- script embedded in GitLab CI YAML in
script,before_script, andafter_script(which are which are all the places where shell script can appear in GitLab CI YAML). GitLab CI YAML files are expected to follow the naming convention*.gitlab-ci.yml.
This job will not handle !reference tags in GitLab CI YAML, so on the off chance that that feature is used in a before_script, after_script, or script, the results won’t be correct.
Happy linting!
Shellcheck Scripts Embedded in GitLab CI YAML by Craig Andrews is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Hi Craig Andrews, is there any commands available to implement linting on shell-script present in yaml file using shellcheck and yq, rather than this CI job.
The commands to implement linting on shell script present in GitLab CI yaml files using shellcheck and yq are in the job, here they are:
newline="$(printf '\nq')" newline=${newline%q} git ls-files --exclude='*.gitlab-ci.yml' --ignored -c | while IFS= read -r file; do yq eval '.[] | select(tag=="!!map") | (.before_script,.script,.after_script) | select(. != null ) | path | ".[\"" + join("\"].[\"") + "\"]"' "${file}" | while IFS= read -r selector; do script=$(yq eval "${selector} | join(\"${newline}\")" "${file}") if ! printf '%s' "${script}" | shellcheck -; then >&2 printf "\nError in %s in the script specified in %s:\n%s\n" "${file}" "${selector}" "${script}" exit 1 fi done doneHi Craig,
Thanks for your inspiring post.
Unfortunately, try as I might, I have not been able to get this line to work even when invoked on the command line independently of GitLab and your full script:
yq eval ‘.[] | select(tag==”!!map”) | (.before_script,.script,.after_script) | select(. != null ) | path | “.[\”” + join(“\”].[\””) + “\”]”‘ “${file}”
I cannot find any reference to `select(tag==”!!map”)` anywhere so I cannot figure out what it is doing, and `path` also seems to not be correct, at least not for jq. jq has `paths` and `path(.)` but not `path` AFAICT.
Is there any chance this link managed to get typos into it when you published? Would you mind testing this again, and if it works as-is may I ask you to explain those two aspects?
Thank you very much in advance.
There are two
yqprojects unfortunately – I suspect you’re looking at the wrong one. Also, it’syq(which is for YAML) and notjq(which is for JSON). Here’s the documentation formapin the yq used in this post: https://mikefarah.gitbook.io/yq/operators/mapYou can see a working example here:
https://gitlab.com/candrews/jumpstart/-/blob/6c9fa0b5c6c8805aeebf964bf9be9c4342e0650a/.gitlab-ci.yml#L26
And here’s the GitHub Actions version:
https://github.com/candrews/jumpstart/blob/6c9fa0b5c6c8805aeebf964bf9be9c4342e0650a/.github/workflows/build.yml#L17
Thanks for the quick reply, and thanks for the details.
Indeed, I am using the other one, the one installed by NixShell, which is the one I have hassle-free access to. That one is a wrapper around `jq` which is why I mentioned `jq`.
I did see the docs for those `map` docs — though I wasn’t sure which `yq` you were using — but I didn’t see anything explaining the double exclamations you used in `!!map`.
Anyway, at this point I think I am going to just write the logic in Go rather than waste more time trying to get this Bash script to behave. Still, thanks for showing me how to get started with spellcheck for GitLab; that definitely saved me from having to trial-and-error it.
> I didn’t see anything explaining the double exclamations you used in `!!map`.
`!!map` means the type/tag of map. All tags (aka types) are denoted with `!!` prefixes, for example, there is also `!!str`, `!!seq`, `!!bool`, and `!!int`. Tags are documented at https://mikefarah.gitbook.io/yq/operators/tag
What the script is doing at that point is looking for maps.
Piping into while read without checking for empty is risky behavior, bash will grab the outer context. I once found passwords this way. It depends on a lot of factors, but I’d avoid it.
Use xargs or check that the result is not null before running while read on it.
For a fast git command like yours this might be enough:
[ $(git ls-files -c | wc -l) -gt 0] && git ls-files -c | while read etc.
I don’t see the risk here.
First, in my testing, with an empty repository (so git doesn’t find any files), the two `git ls-files` that pipe to `read` don’t run their loop bodies as expected. No errors occur and no commands in the `do` bodies are run.
Second, I’m not finding anything that discusses the issue you suggest. Can you please link me to something or otherwise explain the vulnerability?