This repository holds the Workspace Manager service, the MC-Terra component responsible for managing resources and the applications/resources they use.
To push versions of this repository to different environments (including per-developer integration environments), use deliverybot: Deliverybot Dashboard
A swagger-ui page is available at /api/swagger-ui.html on any running instance. TODO: Once we have a stable and persistent dev instance, a link to its swagger-ui page should go here.
We use Spring Boot as our framework for REST servers. The objective is to use a minimal set of Spring features; there are many ways to do the same thing and we would like to constrain ourselves to a common set of techniques.
We only use Java configuration. We never use XML files.
In general, we use type-safe configuration parameters as shown here:
Type-safe Configuration Properties.
That allows proper typing of parameters read from property files or environment variables. Parameters are
then accessed with normal accessor methods. You should never need to use an @Value annotation.
When the applications starts, Spring wires up the components based on the profiles in place. Setting different profiles allows different components to be included. This technique is used as the way to choose the cloud platform (Google, Azure, AWS) code to include.
We use the Spring idiom of the postSetupInitialization, found in ApplicationConfiguration.java,
to perform initialization of the application between the point of having the entire application initialized and
the point of opening the port to start accepting REST requests.
The typical pattern when using Spring is to make singleton classes for each service, controller, and DAO. You do not have to write the class with its own singleton support. Instead, annotate the class with the appropriate Spring annotation. Here are ones we use:
@ComponentRegular singleton class, like a service.@RepositoryDAO component@ControllerREST Controller@ConfigurationDefinition of properties
There are other annotations that are handy to know about.
Spring wires up the singletons and other beans when the application is launched. That allows us to use Spring profiles to control the collection of code that is run for different environments. Perhaps obviously, you can only autowire singletons to each other. You cannot autowire dynamically created objects.
There are two styles for declaring autowiring. The preferred method of autowiring, is to put the annotation on the constructor of the class. Spring will autowire all of the inputs to the constructor.
@Component
public class Foo {
private Bar bar;
private Fribble fribble;
@Autowired
public Foo(Bar bar, Fribble fribble) {
this.bar = bar;
this.foo = foo;
}
Spring will pass in the instances of Bar and Fribble into the constructor. It is possible to autowire a specific class member, but that is rarely necessary:
@Component
public class Foo {
@Autowired
private Bar bar;
@RequestBodyMarks the controller input parameter receiving the body of the request@PathVariable("x")Marks the controller input parameter receiving the parameterx@RequestParam("y")Marks the controller input parameter receiving the query parametery
We use the Jackson JSON library for serializing objects to and from JSON. Most of the time, you don't need to use JSON annotations. It is sufficient to provide setter/getter methods for class members and let Jackson figure things out with interospection. There are cases where it needs help and you have to be specific.
The common JSON annotations are:
@JsonValueMarks a class member as data that should be (de)serialized to(from) JSON. You can specify a name as a parameter to specify the JSON name for the member.@JsonIgnoreMarks a class member that should not be (de)serialized@JsonCreatorMarks a constructor to be used to create an object from JSON.
For more details see Jackson JSON Documentation
This section explains the code structure of the template. Here is the directory structure:
/src
/main
/java
/bio/terra/TEMPLATE
/app
/configuration
/controller
/common
/exception
/service
/ping
/resources
/appFor the top of the application, including Main and the StartupInitializer/app/configurationFor all of the bean and property definitions/app/controllerFor the REST controllers. The controllers typically do very little. They invoke a service to do the work and package the service output into the response. The controller package also defines the global exception handling./commonFor common models and common exceptions; for example, a model that is shared by more than one service./common/exceptionThe template provides abstract base classes for the commonly used HTTP status responses. They are all based on the ErrorReportException that provides the explicit HTTP status and "causes" information for our standard ErrorReport model./serviceEach service gets a package within. We handle cloud-platform specializations within each service./service/pingThe example service; please delete./resourcesProperties definitions, database schema definitions, and the REST API definition
There are sample tests for the ping service to illustrate two styles of unit testing.
- New commit is merged to master
- The master_push workflow is triggered. It builds the image, tags the image & commit, and pushes the image to GCR. It then sends a dispatch with the new version for the service to the framework-version repo.
- This triggers the update workflow, which updates the JSON that maps services to versions to map to the new version for the service whose repo sent the dispatch. The JSON is then committed and pushed.
- This triggers the tag workflow, which tags the new commit in the framework-version repo with a bumped semantic version, yielding a new version of the whole stack incorporating the newly available version of the service.
- The new commit corresponding to the above version of the stack is now visible on the deliverybot dashboard. It can now be manually selected for deployment to an environment.
- Deploying a version of the stack to an environment from the dashboard triggers the deploy workflow. This sends a dispatch to the framework-env repo with the version that the chosen commit is tagged with, and the desired environment.
- The dispatch triggers the update workflow in that repo, which similarly to the one in the framework-version one, updates a JSON. This JSON maps environments to versions of the stack. It is updated to reflect the desired deployment of the new stack version to the specified environment and the change is pushed up.
- The change to the JSON triggers the apply workflow, which actually deploys the desired resources to k8s. It determines the services that must be updated by diffing the stack versions that the environment in question is transitioning between and re-deploys the services that need updates.
Workspace Manager publishes an API client library based on the OpenAPI Spec v3.
...
compile(group: 'bio.terra', name: 'terra-workspace-manager-client', version: '0.0.1-SNAPSHOT')
...
Note that the publishing of this artifact is currently manual. Whenever the OpenAPI definitions change, we should publish a new version of this library to artifactory. Backwards compatible changes should have a minor version bump, and breaking changes should have a major version bump. We will try to avoid breaking changes at all costs.
To publish, you will need to export the ARTIFACTORY_USERNAME and ARTIFACTORY_PASSWORD environment variables for the Broad artifactory. To build, publish, and clean, run:
./gradlew terra-workspace-manager-client:generateApi terra-workspace-manager-client:artifactoryPublish terra-workspace-manager-client:clean