Gametrekking Omnibus Mac OS
Chef Software currently uses Omnibus to package up a number of our client and server software packages for distribution as traditional software binaries (e.g. .deb
, .rpm
, etc). These packages are built in Buildkite using a shim pipeline definition (e.g. .expeditor/release.omnibus.yml) which allows maintainers to easily define on which of our supported platforms they wish to build and/or test their product.
Creating your Omnibus pipelines
- Mac gaming refers to the use of video games on Macintosh personal computers. In the 1990s, Apple computers did not attract the same level of video game development as Microsoft Windows computers due to the high popularity of Microsoft Windows and, for 3D gaming, Microsoft's DirectX technology. In recent years, the introduction of Mac OS X and support for Intel processors has eased porting of.
- Today I am releasing the Gametrekking Omnibus, a downloadable collection of all of my Gametrekking creations to date. It is my tenth release of the Gametrekking project, and it will also be my final 'official' release. Platforms: Linux, Mac OS X, Windows. Get it from: TIGdb. Passage is a very short art game about life and death.
Buildkite Omnibus pipelines are an example of a named pipeline that requires the following configuration:
The omnibus/release pipeline is your main release pipeline that, when triggered, builds omnibus artifacts that are promoted to the current channel when complete.; The omnibus/adhoc pipeline, by setting the ADHOC environment variable to true, tells our automation to include a timestamp with the version and does not promote the build to the current channel (indicating it is not suitable for.
- Two Omnibus Buildkite pipelines:
omnibus/release
andomnibus/adhoc
. - A shim pipeline definition file named .expeditor/release.omnibus.yml (which is shared by both pipelines).
- A valid Omnibus software definition.
We use the term “release” instead of “build” (as we do with other pipelines) to maintain continuity with our old Jenkins pipelines which were named “release” and to reinforce this pipeline is used to create releasable artifacts, rather than one-off builds. If you prefer, you can replace the usage of “release” with “build” (or just about anything else); Expeditor makes no hard-coded assumptions that your omnibus pipeline is named “release.”
As with other named pipelines, prefixing your pipeline name with omnibus/
informs Expeditor that this Buildkite pipeline is different than a general purpose pipeline and tells it to expect a shim pipeline definition rather than a traditional pipeline definition. Remember, shim pipeline definition files are not supported by the native Buildkite DSL.
- The
omnibus/release
pipeline is your main release pipeline that, when triggered, builds omnibus artifacts that are promoted to the current channel when complete. - The
omnibus/adhoc
pipeline, by setting theADHOC
environment variable totrue
, tells our automation to include a timestamp with the version and does not promote the build to the current channel (indicating it is not suitable for external consumption).
Triggering release builds
There are three ways we recommend to trigger your omnibus/release
pipeline.
- Via the trigger_pipeline:omnibus/release action. The most common pattern is to trigger this action in response to a pull_request_merged workload. Projects that have Omnibus pipelines are also good candidates for staging areas as Omnibus builds are sometimes slow and are susceptible to build storms.
- Via the Buildkite UI. Triggering a release build via the Buildkite UI is useful when you need to trigger a fresh build out of band of a code change to your project.
- Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the
bk build create
command.
Triggering adhoc builds
What makes an adhoc build different from a release build is the presence of the ADHOC
environment variable, which we’ve configured to present in all our builds by default as part of pipelines configuration above.
There are three ways we recommend to trigger you omnibus/adhoc
pipeline.
- Via the trigger_pipeline:omnibus/adhoc action. The most common pattern is to trigger this action in response to an external workload. One common use case is triggering an adhoc build when an chef/omnibus-software definition is updated. Another common example is running the build on a schedule.
- Via the Buildkite UI. Triggering a release build via the Buildkite UI is useful when you need to trigger a fresh build out of band of a code change to your project.
- Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the
bk build create
command.
Building subset of platforms using OMNIBUS_BUILD_FILTER
By specifying a comma-separated list of globs in the OMNIBUS_BUILD_FILTER
environment variable, you can build only a subset of the platforms you have specified without needing to modify your shim pipeline definition file.
Note
Warning
Example Environment Variable | Result |
---|---|
mac_os_x* | Build mac_os_x-10.14-x86_64 and mac_os_x-11-arm64 . |
el*,ubuntu* | Build el-6-x86_64 , el-7-x86_64 , and ubuntu-16.04 . |
mac*arm64 | Build only mac_os_x-11-arm64 . |
Building and/or testing subset of platforms using OMNIBUS_FILTER
By specifying a comma-separated list of globs in the OMNIBUS_FILTER
environment variable, you can build and/or test only a subset of the platforms you have specified without needing to modify your shim pipeline definition file.
Warning
Example Environment Variable | Result |
---|---|
mac_os_x-10* | Build mac_os_x-10.14-x86_64 . Test only on mac_os_x-10.14-x86_64 and mac_os_x-10.15-x86_64 . |
el*,ubuntu* | Build el-6-x86_64 , el-7-x86_64 , and ubuntu-16.04 . Test all el and ubuntu platforms. |
mac*arm64 | Build and test only mac_os_x-11-arm64 . |
Triggering a build with an expired cache
You may wish to trigger a build on your omnibus/release
or omnibus/adhoc
pipelines with an expired cached. This is handled by setting the EXPIRE_CACHE
environment variable to true
. Expiring the cache ensures that the entire Omnibus cache directory is deleted from the build systems. This can be especially useful when you’ve built the latest major version N
of an omnibus project and now you need to clear the cache to successfully build the N-1
version of the project.
There is no way to pass an environment variable in as part of the trigger_pipeline action, and we do not recommend permanently configuring your pipelines to expire the cache on every build, so the only way to trigger a build with an expired cache is manually.
- Via the Buildkite UI. You can specify the
EXPIRE_CACHE=true
environment variable when triggering a manual build via the Buildkite UI by clicking on the “Options” link in the “New Build” window. - Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the
bk build create
command.
Triggering a build with MAC_NOTARIZATION disabled
You may wish to trigger a build on your omnibus/release
or omnibus/adhoc
pipelines without notarizing the macOS package. This is handled by setting the MAC_NOTARIZE
environment variable to false
. By default expeditor cli will enable macOS Notarization, which means the package created during the build will be uploaded as a buildkite artifact which will be run through “Notarize macOS Package” phase where the artifact will be notarized from apple and the ticket will be stapled to the package before it is published to artifactory. Apple is going to make notarizing all build macOS packages mandatory starting February 3rd 2020. If your pipeline is not ready for notarizing the package during the build you may disable notarization for your pipeline by passing the environment variable “MAC_NOTARIZE” in your build settings and setting it to “false”
There is no way to pass an environment variable in as part of the trigger_pipeline action, and we do not recommend permanently configuring your pipelines to skip macOS notarization on every build, so the only way to trigger a build macOS without notarization manually. This value does not affect pipelines that do not contain macOS builds.
- Via the Buildkite UI. You can specify the
MAC_NOTARIZE=false
environment variable when triggering a manual build via the Buildkite UI by clicking on the “Options” link in the “New Build” window. - Via the Buildkite CLI. If you have the Buildkite CLI configured, you can trigger a release pipeline manually using the
bk build create
command.
Walk-through of what happens in the pipeline
Let’s break down the steps of an Omnibus Buildkite pipeline build and walk through all the processes.
- Parse the shim pipeline definition. Our trigger step reads in the .expeditor/release.omnibus.yml shim pipeline definition file and performs the following sub-steps.
- Determine on which platforms we need to build. Expeditor reads through your builder-to-testers-map setting and determines on which platform it needs to build. It also looks at the fips-platforms setting to determine whether or not to build with FIPS compatibility.
- Determine on which platforms we need to test. While reading through your builder-to-testers-map setting, Expeditor also determines on which platforms certain builds should be tested. It configures the jobs accordingly, using the test-path or test-path-windows setting for each job as appropriate.
- Determine whether or not to promote to current at the end. Based on whether or not the
ADHOC
environment variable is set totrue
, Expeditor will tack a final step which will promote all the builds to the current channel in Artifactory.
- Build all the artifacts and publish them to the unstable channel. An individual Buildkite job is created for each Omnibus build so that all the builds can occur in parallel. If the Omnibus build succeeds, the resulting artifact is published to the unstable channel of Chef’s internal Artifactory.
- Create the Build Record. Upon the completion of all the builds, a build record representing all the artifacts is created inside Artifactory.
- Test all of the artifacts. Once the build record has been completed it is time to test the artifacts using the test script specified in either test-path or test-path-windows. Each test platform downloads the appropriate artifact (as determined by our mixlib-install logic) and tested.
- Promote to the current channel (release only). Once all of the tests scripts have passed successfully, our builds are promoted to the current channel, where they can be consumed by early adopters and other beta/QA testers.
release.omnibus.yml
The release.omnibus.yml file is a shim pipeline definition file with a particular schema that informs the pipeline how to build, test, and upload to Artifactory an Omnibus package.
Placeholder Definitions
PROJECT_NAME
- The name of the project as defined in our internal Artifactory instance: typically the same as the product_key with few exceptions.
build-options
Additional parameters to pass into the omnibus
CLI at build time.
builder-to-testers-map
A hash that maps each builder platform used for building a package to its corresponding tester platforms.
You can find names of the currently supported platforms in omnibus-toolchain’s release.omnibus.yml file.
config
The relative path to the omnibus configuration file in the local repository. Defaults to omnibus.rb
.
fips-platforms
A list of platforms that should have OMNIBUS_FIPS_MODE
environment variable set to true
. This list can include glob patterns such as el-*-x86_64
.
install-dir
Install path of the omnibus project. Defaults to /opt/<project-name>
.
install-dir-windows
Install path of the omnibus project on a Windows platform. Defaults to C:/opscode/<project-name>
.
project-name
The name of the omnibus project.
skip-artifactory-platforms
A list of platforms that will not upload to artifactory and will not fail the entire pipeline if their specific step fails. Can be used to target multiple platforms with the *
character such as el-*-x86_64
. This will match BOTH builders and testers.
test-channel
Gametrekking Omnibus Mac Os Download
The channel from which to download the omnibus package for testing. Defaults to unstable
. If set, we skip the build step.
test-dep-channel
The channel from which to download dependent omnibus packages for testing. Defaults to current
.
test-path
The relative path to the script used to test the omnibus package. If unset, we skip the testing step.
Gametrekking Omnibus Mac Os X
test-path-windows
The relative path to the Windows batch or PowerShell script used to test the omnibus package on a Windows platform. If unset, we skip the testing step for Windows.
test-version
Gametrekking Omnibus Mac Os 11
The omnibus package version to be downloaded for testing. Defaults to the value stored in Buildkite metadata by the build stage. If unset, we skip the build step.