PROP-27 - Implement Registry proxy #32

nyagamunene · 2024-12-09T18:16:42Z

What type of PR is this?

What does this do?

Which issue(s) does this PR fix/relate to?

Resolves Feature: Implement Registry proxy #27

Have you included tests for your changes?

Did you document any new/modified features?

Notes

cmd/proxy/.env

cmd/proxy/main.go

proxy/config/http.go

proxy/mqtt/mqtt.go

cmd/proxy/main.go

proxy/config/http.go

proxy/config/mqtt.go

cmd/proxy/main.go

proxy/README.md

proxy/config/http.go

proxy/config/mqtt.go

Signed-off-by: nyagamunene <[email protected]>

cmd/proxy/main.go

rodneyosodo · 2024-12-19T21:18:35Z

cmd/proxy/main.go

+	logger := slog.New(logHandler)
+	slog.SetDefault(logger)
+
+	mqttCfg := config.MQTTProxyConfig{}


There is some stutter here in config MQTT Proxy Config

You can move the configs to the proxy package and import it as

mqttConfig := proxy.MQTTConfig{}

rodneyosodo · 2024-12-19T21:20:56Z

proplet/requests.go

+	ID                   string
+	FunctionName         string
+	WasmFile             []byte
+	WasmFileDownloadPath string


Can this be changed to image URL?

rodneyosodo · 2024-12-19T21:22:14Z

proxy/service.go

+		case chunk := <-s.dataChan:
+			if err := s.mqttClient.PublishContainer(ctx, chunk); err != nil {
+				s.logger.Error("failed to publish container chunk",
+					"error", err,


use logger args

rodneyosodo · 2024-12-19T21:22:26Z

proxy/service.go

+
+			s.logger.Info("published container chunk",
+				"chunk_name", chunk.AppName,
+				"chunk_no", chunk.ChunkIdx,


rodneyosodo · 2024-12-19T21:22:50Z

proxy/service.go

+				continue
+			}
+
+			s.logger.Info("published container chunk",


Should this be debug information?

Signed-off-by: nyagamunene <[email protected]>

rodneyosodo · 2024-12-20T11:18:11Z

test.md

@@ -102,3 +102,28 @@ export PROPLET_THING_ID=""
 export PROPLET_THING_KEY=""
 propeller-proplet
 ```
+


Move this to absmach/propeller-docs#6 and delete it here

rodneyosodo · 2024-12-20T11:23:00Z

cmd/proxy/main.go

+	LogLevel string `env:"PROXY_LOG_LEVEL"           envDefault:"info"`
+
+	BrokerURL  string `env:"PROXY_MQTT_ADDRESS"          envDefault:"tcp://localhost:1883"`
+	PropletKey string `env:"PROXY_PROPLET_KEY,notEmpty"`


Align with the other env variable config

PROPLET_CHANNEL_ID PROPLET_THING_ID PROPLET_THING_KEY

rodneyosodo · 2024-12-20T11:23:20Z

cmd/proxy/main.go

+
+	ChunkSize    int    `env:"PROXY_CHUNK_SIZE"             envDefault:"512000"`
+	Authenticate bool   `env:"PROXY_AUTHENTICATE"           envDefault:"false"`
+	Token        string `env:"PROXY_REGISTRY_TOKEN"         envDefault:""`


Do we need token and password or it is either one?

Its either of the 2

rodneyosodo · 2024-12-20T11:24:23Z

task/task.go

@@ -35,6 +33,7 @@ type Task struct {
 	ID         string    `json:"id"`
 	Name       string    `json:"name"`
 	State      State     `json:"state"`
+	ImageURL   URLValue  `json:"image_url,omitempty"`


What about just string?

rodneyosodo · 2024-12-20T11:24:29Z

task/url.go

@@ -0,0 +1,45 @@
+package task
+


Signed-off-by: nyagamunene <[email protected]>

drasko · 2024-12-20T12:22:01Z

Makefile

-		if [[ ! "$$file" =~ \.wasm$$ ]]; then \
-			cp "$$file" $(GOBIN)/propeller-`basename "$$file"`; \
-		fi \
+		if [ "$$file" = "$${file%%.wasm}" ]; then \


Can we use Makefile wildcard directive here to contract the filtering and selection for copying?

Signed-off-by: nyagamunene <[email protected]>

drasko · 2024-12-20T16:18:54Z

Makefile

-			cp "$$file" $(GOBIN)/propeller-`basename "$$file"`; \
-		fi \
-	done
+	$(foreach f,$(wildcard $(BUILD_DIR)/*[!.wasm]),cp $(f) $(patsubst $(BUILD_DIR)/%,$(GOBIN)/propeller-%,$(f));)


Would it be possible, in order to simplify handling of binaries, to have a subdir wasm within build dir, and put all Wasm binaries there. I guess these binaries are coming from examples, right? If really needed you can also add build/bin and build/wasm, but I do not think that we need build/bin (as most of the time we'll need just Propeller services (and not examples), and they can live directly in the top build dir, for convenience), but if this simplifies copying, then please go ahead and make both build/bin and build/wasm.

In fact, as you would need to prefix each bin anyway - I will merge as is.

Would it be possible, in order to simplify handling of binaries, to have a subdir wasm within build dir, and put all Wasm binaries there. I guess these binaries are coming from examples, right? If really needed you can also add build/bin and build/wasm, but I do not think that we need build/bin (as most of the time we'll need just Propeller services (and not examples), and they can live directly in the top build dir, for convenience), but if this simplifies copying, then please go ahead and make both build/bin and build/wasm.

We can implement this .

nyagamunene self-assigned this Dec 9, 2024

nyagamunene force-pushed the PROP-27 branch from 11bc6ce to f183783 Compare December 11, 2024 11:45

nyagamunene marked this pull request as ready for review December 11, 2024 13:37

rodneyosodo requested changes Dec 11, 2024

View reviewed changes

rodneyosodo requested changes Dec 13, 2024

View reviewed changes

cmd/proxy/main.go Outdated Show resolved Hide resolved

cmd/proxy/main.go Outdated Show resolved Hide resolved

cmd/proxy/main.go Outdated Show resolved Hide resolved

proxy/config/http.go Outdated Show resolved Hide resolved

proxy/config/mqtt.go Outdated Show resolved Hide resolved

nyagamunene force-pushed the PROP-27 branch from 7377a6d to 07093dd Compare December 13, 2024 16:42

rodneyosodo requested changes Dec 19, 2024

View reviewed changes

cmd/proxy/main.go Outdated Show resolved Hide resolved

proxy/README.md Outdated Show resolved Hide resolved

proxy/config/http.go Outdated Show resolved Hide resolved

proxy/config/mqtt.go Outdated Show resolved Hide resolved

nyagamunene added 23 commits December 19, 2024 15:01

Initial implementation of HTTP to MQTT proxy

57bcec1

Signed-off-by: nyagamunene <[email protected]>

update mqtt client and http client

3a10e82

Signed-off-by: nyagamunene <[email protected]>

refactor oras http

2c29619

Signed-off-by: nyagamunene <[email protected]>

add env file and main file

b83e12d

Signed-off-by: nyagamunene <[email protected]>

add env file and main file

3fedc22

Signed-off-by: nyagamunene <[email protected]>

update go.mod and go.sum file

0401587

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

e2ac8c5

Signed-off-by: nyagamunene <[email protected]>

fix tag align

e9755d2

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

16a8e47

Signed-off-by: nyagamunene <[email protected]>

fix start method

046a167

Signed-off-by: nyagamunene <[email protected]>

remove white spaces

ee09666

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

b442273

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

26a6706

Signed-off-by: nyagamunene <[email protected]>

address comments and change how data is sent

48bf08f

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

ebea46b

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

2f4621a

Signed-off-by: nyagamunene <[email protected]>

add logging

7751314

Signed-off-by: nyagamunene <[email protected]>

add documentation and debug connection issue

182f89f

Signed-off-by: nyagamunene <[email protected]>

update go mod and go sum file

a86c749

Signed-off-by: nyagamunene <[email protected]>

remove password

1f4b1a5

Signed-off-by: nyagamunene <[email protected]>

add comments

0b323e6

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

b1d7f38

Signed-off-by: nyagamunene <[email protected]>

add validation

df6c46c

Signed-off-by: nyagamunene <[email protected]>

nyagamunene added 11 commits December 19, 2024 15:03

adjust size of data sent via nats

7356002

Signed-off-by: nyagamunene <[email protected]>

add contants

b9a4d20

Signed-off-by: nyagamunene <[email protected]>

refactor FetchFromReg

f9aa43e

Signed-off-by: nyagamunene <[email protected]>

add logging after all chunks were sent sucessfully

50f4f32

Signed-off-by: nyagamunene <[email protected]>

change chunk_payload type

fa04558

Signed-off-by: nyagamunene <[email protected]>

update test documentation

4c718e4

Signed-off-by: nyagamunene <[email protected]>

update env variables

27608f0

Signed-off-by: nyagamunene <[email protected]>

fix failing linter

fb81865

Signed-off-by: nyagamunene <[email protected]>

update env file

c31f6d5

Signed-off-by: nyagamunene <[email protected]>

remove proxy read me

93cb573

Signed-off-by: nyagamunene <[email protected]>

intergrate proplet and manager with proxy

471d22e

Signed-off-by: nyagamunene <[email protected]>

nyagamunene force-pushed the PROP-27 branch from 0e2c914 to 471d22e Compare December 19, 2024 16:15

remove unused variable

b439ca0

Signed-off-by: nyagamunene <[email protected]>

rodneyosodo requested changes Dec 19, 2024

View reviewed changes

nyagamunene added 2 commits December 20, 2024 13:42

address comments

6c13917

Signed-off-by: nyagamunene <[email protected]>

remove linter check

03b75aa

Signed-off-by: nyagamunene <[email protected]>

rodneyosodo requested changes Dec 20, 2024

View reviewed changes

nyagamunene added 2 commits December 20, 2024 14:55

address comments

a1f4940

Signed-off-by: nyagamunene <[email protected]>

remove test.md file

cf7a9eb

Signed-off-by: nyagamunene <[email protected]>

rodneyosodo approved these changes Dec 20, 2024

View reviewed changes

drasko requested changes Dec 20, 2024

View reviewed changes

nyagamunene added 3 commits December 20, 2024 15:46

update make install command

0f70398

Signed-off-by: nyagamunene <[email protected]>

update make install command

febf61c

Signed-off-by: nyagamunene <[email protected]>

add comments to make install command

2da6d97

Signed-off-by: nyagamunene <[email protected]>

drasko requested changes Dec 20, 2024

View reviewed changes

drasko merged commit 8a91d1c into absmach:main Dec 20, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PROP-27 - Implement Registry proxy #32

PROP-27 - Implement Registry proxy #32

nyagamunene commented Dec 9, 2024 •

edited

Loading

rodneyosodo Dec 19, 2024

rodneyosodo Dec 19, 2024

rodneyosodo Dec 19, 2024

rodneyosodo Dec 19, 2024

rodneyosodo Dec 19, 2024

rodneyosodo Dec 20, 2024

rodneyosodo Dec 20, 2024

rodneyosodo Dec 20, 2024

nyagamunene Dec 20, 2024

rodneyosodo Dec 20, 2024

rodneyosodo Dec 20, 2024

drasko Dec 20, 2024

drasko Dec 20, 2024

drasko Dec 20, 2024

nyagamunene Dec 20, 2024

PROP-27 - Implement Registry proxy #32

PROP-27 - Implement Registry proxy #32

Conversation

nyagamunene commented Dec 9, 2024 • edited Loading

What type of PR is this?

What does this do?

Which issue(s) does this PR fix/relate to?

Have you included tests for your changes?

Did you document any new/modified features?

Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nyagamunene commented Dec 9, 2024 •

edited

Loading