Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Component configuration template and test system #194

Draft
wants to merge 23 commits into
base: master
Choose a base branch
from
Draft

Component configuration template and test system #194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
d909bf5
[occ] Dump incoming configuration payloads to file
teo Nov 24, 2020
f272b20
[core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
teo Nov 26, 2020
5bd0a29
[core] Print out varStack for debug purposes
teo Nov 27, 2020
2368076
[core] Fix StringWrapMap unmarshal occasionally breaking the var system
teo Nov 27, 2020
5c2f309
[core] Add utils.TimeTrack to be used with defer to measure exec time
teo Nov 27, 2020
8456d2a
[core] Add support for config template inclusion
teo Nov 30, 2020
91ad13e
[core] Add support for RUNTYPE and rolename in config paths
teo Nov 30, 2020
a1fd76a
[coconut] Initial support for RUNTYPE/rolename in config paths
teo Nov 30, 2020
d19fc66
[occ] Print out OCC version from dummyprocess
teo Dec 9, 2020
8c4ecd1
[core] Add NewID function to workflow template environment
teo Dec 10, 2020
162f584
[coconut] Improve coconut conf output
teo Dec 10, 2020
fed53ea
[coconut] Fix coconut conf import behavior
teo Dec 11, 2020
eb98edc
[configuration] Improve Get, GetKeysByPrefix; add IsDir
teo Dec 11, 2020
6a1ff3f
[coconut] Update coconut conf show for new Consul structure
teo Dec 14, 2020
4c12d56
[core][configuration] Move high level conf mgmt to configuration package
teo Dec 14, 2020
ce7968a
[configuration] Move component cfg template processing into confSvc
teo Dec 14, 2020
d5f29a4
[configuration] Add IsDir to configuration.YamlSource
teo Dec 14, 2020
2404622
[configuration] Fix GetConfig logic for unversioned configuration
teo Dec 14, 2020
a063538
[common] Suppress bothersome log output
teo Dec 14, 2020
167b62f
[coconut] New --simulate flag to coconut conf show
teo Dec 14, 2020
fa5c8b4
[core] Migrate from gogo/protobuf to protobuf generator v2
teo Dec 15, 2020
7c9b784
[coconut] Update inline documentation for coconut conf
teo Dec 16, 2020
b55d1ce
Replace sed with perl in makefile
miltalex Jan 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 15 additions & 10 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,10 @@ PROD :=-X=$(REPOPATH)/common/product
EXTLDFLAGS :="-static"
LDFLAGS=-ldflags "-extldflags $(EXTLDFLAGS) $(PROD).VERSION_MAJOR=$(VERSION_MAJOR) $(PROD).VERSION_MINOR=$(VERSION_MINOR) $(PROD).VERSION_PATCH=$(VERSION_PATCH) $(PROD).BUILD=$(BUILD)" -tags osusergo,netgo

# We expect to find the gogo protobuf executables in $GOPATH/bin
# We expect to find the protoc-gen-go executable in $GOPATH/bin
GOPATH := $(shell go env GOPATH)
GOGOPATH=$(GOPATH)/bin/protoc-gen-gofast
HAS_GOGOPROTO := $(shell command -v $(GOGOPATH) 2> /dev/null)

GO_GET_U1 := $(addprefix github.com/gogo/protobuf/, proto protoc-gen-gofast protoc-gen-gogofast protoc-gen-gogofaster protoc-gen-gogoslick gogoproto)
GO_GET_U2 := $(addprefix github.com/golang/protobuf/, proto protoc-gen-go)
GO_GET_U2 += google.golang.org/grpc
GOPROTOCPATH=$(GOPATH)/bin/protoc-gen-go
HAS_PROTOC := $(shell command -v $(GOPROTOCPATH) 2> /dev/null)

.PHONY: build all install generate test debugtest vet fmt clean cleanall help $(WHAT) tools vendor doc docs

Expand Down Expand Up @@ -97,7 +93,7 @@ $(INSTALL_WHAT):
@$(WHAT_$(@:install_%=%)_BUILD_FLAGS) go install -mod=vendor $(VERBOSE_$(V)) $(LDFLAGS) ./cmd/$(@:install_%=%)

generate:
ifndef HAS_GOGOPROTO
ifndef HAS_PROTOC
$(MAKE) tools/protoc
endif
@for gendir in $(GENERATE_DIRS); do \
Expand Down Expand Up @@ -132,6 +128,15 @@ vendor:
@echo -e "\033[1;33mcurl odc.proto\033[0m"
@mkdir -p odcshim/odcprotos
@curl -s -L $(ODC_PROTO) -o odcshim/odcprotos/odc.proto

# WORKAROUND: In order to avoid the following issues:
# https://github.com/golang/protobuf/issues/992
# https://github.com/golang/protobuf/issues/1158
# we insert a go_package specification into the ODC protofile right
# after we download it.
@echo -e "\033[1;33mpatch odc.proto\033[0m"
@perl -pi -e '$$_.="option go_package = \"odcprotos;odc\";\n" if (/^package/)' odcshim/odcprotos/odc.proto

# vendor: tools/dep
# @echo -e "\033[1;33mdep ensure\033[0m"
# @./tools/dep ensure
Expand All @@ -147,8 +152,8 @@ tools: tools/protoc

tools/protoc:
@echo "installing Go protoc"
go get -u $(GO_GET_U1)
go get -u $(GO_GET_U2)
go get -u google.golang.org/grpc
go install google.golang.org/protobuf/cmd/protoc-gen-go

docs: doc

Expand Down
4 changes: 3 additions & 1 deletion coconut/cmd/configuration_dump.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@ var configurationDumpCmd = &cobra.Command{
Short: "dump configuration subtree",
Long: `The configuration dump command requests from O² Configuration
a subtree of key-values, and dumps it to standard output in the specified
format.`,
format. This command has full read access to the O² Configuration store
and performs a raw query with no additional processing or access control
semantics.`,
Run: configuration.WrapCall(configuration.Dump),
Args: cobra.ExactArgs(1),
}
Expand Down
2 changes: 1 addition & 1 deletion coconut/cmd/configuration_history.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ var configurationHistoryCmd = &cobra.Command{
Aliases: []string{"h"},
Example: `coconut conf history <component>
coconut conf history <component> <entry>
coconut conf history <component>/<entry>`,
coconut conf history <component>/<run type>/<machine role>/<entry>`,
Short: "List all existing entries with timestamps of a specified component in Consul",
Long: `The configuration history command returns all entries with
all of their associated timestamps or returns all timestamps for a specified component and entry`,
Expand Down
4 changes: 2 additions & 2 deletions coconut/cmd/configuration_import.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ var configurationImportCmd = &cobra.Command{
Use: "import <component> <entry> <file_path>",
Aliases: []string{"i", "imp"},
Example: `coconut conf import <component> <entry> <file_path>
coconut conf import <component>/<entry> <file_path>
coconut conf import <component>/<run type>/<machine role>/<entry> <file_path>
coconut conf import <component> <entry> <file_path> --new-component
coconut conf import <component>/<entry> <file_path> --format=json
coconut conf import <component>/<run type>/<machine role>/<entry> <file_path> --format=json
coconut conf import <component> <entry> <file_path>.json
coconut conf import <component> <entry> <file_path> --no-versioning
coconut conf import <component> <entry> <file_path> --no-versioning --new-component
Expand Down
10 changes: 7 additions & 3 deletions coconut/cmd/configuration_show.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,10 @@ var configurationShowCmd = &cobra.Command{
Aliases: []string{"s"},
Example: `coconut conf show <component> <entry>
coconut conf show <component> <entry> -t <timestamp>
coconut conf show <component>/<entry>
coconut conf show <component>/<entry> -t <timestamp>
coconut conf show <component>/<entry>@<timestamp>`,
coconut conf show <component>/<run type>/<machine role>/<entry>
coconut conf show <component>/<run type>/<machine role>/<entry> -t <timestamp>
coconut conf show <component>/<run type>/<machine role>/<entry>@<timestamp>
coconut conf show <component> <entry> -s -e '{"key1": "value1", "key2": "value2"}'`,
Short: "Show configuration for the component and entry specified",
Long: `The configuration show command returns the most recent
configuration revision for the specified component and entry.
Expand All @@ -48,4 +49,7 @@ It can also return a specific revision, requested with the --timestamp/-t flag`,
func init() {
configurationCmd.AddCommand(configurationShowCmd)
configurationShowCmd.Flags().StringP("timestamp", "t", "", "request configuration for this timestamp")
configurationShowCmd.Flags().BoolP("simulate", "s", false, "simulate runtime template processing on the configuration payload")
// The following only applies if simulate is set:
configurationShowCmd.Flags().StringP("extra-vars", "e", "", "values passed using key=value CSV or JSON syntax, interpreted as strings `key1=val1,key2=val2` or `{\"key1\": \"value1\", \"key2\": \"value2\"}`")
}
2 changes: 1 addition & 1 deletion coconut/cmd/environment.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
* Intergovernmental Organization or submit itself to any jurisdiction.
*/

//go:generate protoc -I ../../core --gofast_out=plugins=grpc:.. protos/o2control.proto
//go:generate protoc -I ../../core --go_out=plugins=grpc:.. protos/o2control.proto

package cmd

Expand Down
2 changes: 1 addition & 1 deletion coconut/cmd/repo.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
* Intergovernmental Organization or submit itself to any jurisdiction.
*/

//go:generate protoc -I ../../core --gofast_out=plugins=grpc:.. protos/o2control.proto
//go:generate protoc -I ../../core --go_out=plugins=grpc:.. protos/o2control.proto

package cmd

Expand Down
Loading