update to dropshot 0.15 #7196

hawkw · 2024-12-03T17:51:05Z

No description provided.

As of oxidecomputer/dropshot@f0e6a46, the OpenAPI documents for APIs that don't use custom user-defined error types should be unchanged from the previous Dropshot release. This commit updates that and undoes the diff to the OpenAPI docs that broke progenitor.

iliana · 2024-12-05T22:05:41Z

I think this is ready to merge barring any test failures, but I want to merge it after we branch for R12.

iliana · 2024-12-06T17:54:16Z

I'm not completely sure yet but I think this is failing in CI because sled-agent is using dropshot 0.15's default_request_body_max_bytes to write a config file for MGS, but MGS is on an older dropshot. https://buildomat.eng.oxide.computer/wg/0/artefact/01JECD8056131TGQ88YT5461PR/vf0F9NvP9NS6utkc5JpUxVireyQNiKooCBzR9UHjjaHeQJGo/01JECD8EAYXDC7E33WV6TV4JNC/01JECG53D8QTFM4MT6VK7BZH49/oxide-mgs:default.log?format=x-bunyan

EDIT: wait no that's not a generated config.

davepacheco · 2024-12-06T21:19:26Z

We'll want to update this to 0.15.1, which I expect to be available shortly. (See oxidecomputer/dropshot#1195.)

hawkw · 2024-12-11T20:18:38Z

IIRC we wanted to wait until after the release to merge this, is that still the case?

iliana · 2024-12-11T20:36:38Z

After we branch for R12 yeah, but I think that's just my risk assessment. I don't know how soon we're branching.

sunshowers · 2024-12-12T23:57:58Z

nexus/examples/config-second.toml

@@ -60,7 +60,7 @@ bind_address = "127.0.0.1:12222"
 #
 # This should be brought back down to a more reasonable value once per-endpoint
 # request body limits are implemented.
-request_body_max_bytes = 3221225472
+default_request_body_max_bytes = 3221225472


Thank you for picking this up!

Think the comment above here may need to be updated since 0.15 implements these limits. (Just saying that this should be brought down is good, I think.)

sunshowers · 2024-12-12T23:58:07Z

nexus/examples/config.toml

@@ -46,7 +46,7 @@ bind_address = "127.0.0.1:12220"
 #
 # This should be brought back down to a more reasonable value once per-endpoint
 # request body limits are implemented.
-request_body_max_bytes = 3221225472
+default_request_body_max_bytes = 3221225472


sunshowers · 2024-12-12T23:59:02Z

sled-agent/src/services.rs

@@ -2359,7 +2359,7 @@ impl ServiceManager {
                            bind_address: SocketAddr::new(*opte_ip, nexus_port),
                            // This has to be large enough to support:
                            // - bulk writes to disks


This one too I think.

sunshowers · 2024-12-12T23:59:50Z

smf/sled-agent/gimlet-standalone/config.toml

 # Host OS images are just over 800 MiB currently; set this to 2 GiB to give some
 # breathing room.
-request_body_max_bytes = 2_147_483_648
+default_request_body_max_bytes = 2_147_483_648


Same here -- just a todo mentioning that it's now possible to bring this down and use the per-endpoint limit.

sunshowers · 2024-12-12T23:59:55Z

smf/sled-agent/gimlet/config.toml

 # Host OS images are just over 800 MiB currently; set this to 2 GiB to give some
 # breathing room.
-request_body_max_bytes = 2_147_483_648
+default_request_body_max_bytes = 2_147_483_648


sunshowers · 2024-12-12T23:59:59Z

smf/sled-agent/non-gimlet/config.toml

 # Host OS images are just over 800 MiB currently; set this to 2 GiB to give some
 # breathing room.
-request_body_max_bytes = 2_147_483_648
+default_request_body_max_bytes = 2_147_483_648


iliana · 2024-12-13T02:46:01Z

I figure since I intend this to land post-R12 I may as well make the values sensible now. If we log request body sizes it might be good to pull those logs from dogfood as well to see if we would be overlooking any endpoints.

iliana · 2024-12-16T19:30:48Z

I've found most of the larger-than-necessary API-wide limits but I'm still finding some more.

iliana · 2024-12-16T21:28:43Z

I think that's everything, except for a few defaults that remain set above 1 MiB:

A handful of Oximeter producer endpoints use 10 MiB as the max body size. I'm not entirely sure why? (Oximeter retrieves these metrics without sending large request bodies, right?)
External/internal DNS use 100 MiB as the max body size but I don't have the necessary context to safely lower this.
wicketd is 8 MiB (this was the previous value before it was bumped to 4 GiB to support TUF repos).

And for completion, testing-only values:

The simulated Crucible Pantry in sled-agent/src/sim has an 8 MiB limit.
Transient DNS servers use a 4 MiB limit; the DNS server used in nexus-test-utils and for testing the resolver implementation use an 8 MiB limit.

Given how wide-reaching of a change this has the opportunity to be, I'm running mupdate, rack init, and live-tests on dublin from this branch.

hawkw · 2024-12-16T22:14:32Z

Given how wide-reaching of a change this has the opportunity to be, I'm running mupdate, rack init, and live-tests on dublin from this branch.

Yeah, that sounds like a good call --- thanks for taking this over while I was moving!

iliana · 2024-12-16T22:15:24Z

[switch 1 00:00:08] Completed .. 4a (8/8) Checking the new RoT bootloader: after 3.01s
[switch 1 00:00:08] Completed (4/6) Updating RoT bootloader: after 6.86s
[switch 1 00:00:08]   Running (5/6) Updating RoT
[switch 1 00:00:08]   Skipped (5/6) Updating RoT: RoT active slot already at version 1.0.30
[switch 1 00:00:08]   Running (6/6) Updating SP
[switch 1 00:00:08]   Running .. 6a (1/4) Sending data to MGS (slot 0)
[switch 1 00:00:08]    Failed .. 6a (1/4) Sending data to MGS (slot 0): after 42.87ms: SP component update failed at stage "sending" for switch_sp:sidecar-c (version 1.0.32)
[switch 1 00:00:08]           .. 6a   Caused by:
[switch 1 00:00:08]           .. 6a   - Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "0b1e394e-1722-4ff9-81b5-4767c196b43f", "content-length": "126", "date": "Sun, 28 Dec 1986 00:09:02 GMT"}; value: Error { error_code: None, message: "request body exceeded maximum size of 1048576 bytes", request_id: "0b1e394e-1722-4ff9-81b5-4767c196b43f" }

haha woooo

iliana · 2024-12-17T21:49:39Z

Okay, mupdate, rack init, and basic operation including instance launch all seem fine on a racklette with this installed. Mupdate was performed from a switch zone built on this branch. Trying to get live-tests going but running into unrelated issues.

There appear to be no rejected requests due to unexpected large request bodies:

root@oxz_switch0:~# pilot host exec -O -c "grep 'request body exceeded maximum size' /var/svc/log/* /zone/oxz_switch/root/var/svc/log/* /pool/ext/*/crypt/zone/*/root/var/svc/log/* /pool/ext/*/crypt/debug/*" 14-17
=== Failure from BRM42220026 (cubby 14):
exit code 1

=== Failure from BRM27230037 (cubby 15):
exit code 2: grep: can't open "/zone/oxz_switch/root/var/svc/log/*"

=== Failure from BRM23230018 (cubby 16):
exit code 1

=== Failure from BRM23230010 (cubby 17):
exit code 2: grep: can't open "/zone/oxz_switch/root/var/svc/log/*"

sunshowers · 2024-12-17T22:08:08Z

This is awesome, thanks for being so thorough iliana!

iliana · 2024-12-17T22:09:18Z

I may have found an issue while running live tests related to internal DNS, but it's unclear as of yet if that's caused by this PR or something else.

[WIP] update to oxidecomputer/dropshot#1180

1994534

hawkw mentioned this pull request Dec 3, 2024

Allow handlers to return user-defined error types oxidecomputer/dropshot#1180

Merged

hawkw and others added 3 commits December 3, 2024 12:11

draw the rest of the owl

b08747a

Merge remote-tracking branch 'origin/main' into eliza/dropshot-1180

5e71ab8

iliana changed the title ~~[WIP] update to oxidecomputer/dropshot#1180~~ update to dropshot 0.15 Dec 5, 2024

iliana marked this pull request as ready for review December 5, 2024 21:58

update to dropshot 0.15.0

f7cfe6d

iliana force-pushed the eliza/dropshot-1180 branch from 45b9156 to f7cfe6d Compare December 6, 2024 17:56

iliana added 2 commits December 9, 2024 17:55

Merge remote-tracking branch 'origin/main' into eliza/dropshot-1180

3aa1bfe

update to dropshot 0.15.1

8dd03e9

sunshowers reviewed Dec 13, 2024

View reviewed changes

iliana added 2 commits December 16, 2024 19:07

clean up high default_request_body_max_bytes values

459618b

Merge remote-tracking branch 'origin/main' into eliza/dropshot-1180

097619e

even more

ee6c690

fix mupdate

fb10fad

hawkw mentioned this pull request Dec 18, 2024

refine error messages about not having a propolis address #7263

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update to dropshot 0.15 #7196

update to dropshot 0.15 #7196

hawkw commented Dec 3, 2024

iliana commented Dec 5, 2024

iliana commented Dec 6, 2024 •

edited

Loading

davepacheco commented Dec 6, 2024

hawkw commented Dec 11, 2024

iliana commented Dec 11, 2024

sunshowers Dec 12, 2024

sunshowers Dec 12, 2024

sunshowers Dec 12, 2024

sunshowers Dec 12, 2024

sunshowers Dec 12, 2024

sunshowers Dec 12, 2024

iliana commented Dec 13, 2024 •

edited

Loading

iliana commented Dec 16, 2024

iliana commented Dec 16, 2024

hawkw commented Dec 16, 2024

iliana commented Dec 16, 2024

iliana commented Dec 17, 2024

sunshowers commented Dec 17, 2024

iliana commented Dec 17, 2024

update to dropshot 0.15 #7196

Are you sure you want to change the base?

update to dropshot 0.15 #7196

Conversation

hawkw commented Dec 3, 2024

iliana commented Dec 5, 2024

iliana commented Dec 6, 2024 • edited Loading

davepacheco commented Dec 6, 2024

hawkw commented Dec 11, 2024

iliana commented Dec 11, 2024

sunshowers Dec 12, 2024

Choose a reason for hiding this comment

sunshowers Dec 12, 2024

Choose a reason for hiding this comment

sunshowers Dec 12, 2024

Choose a reason for hiding this comment

sunshowers Dec 12, 2024

Choose a reason for hiding this comment

sunshowers Dec 12, 2024

Choose a reason for hiding this comment

sunshowers Dec 12, 2024

Choose a reason for hiding this comment

iliana commented Dec 13, 2024 • edited Loading

iliana commented Dec 16, 2024

iliana commented Dec 16, 2024

hawkw commented Dec 16, 2024

iliana commented Dec 16, 2024

iliana commented Dec 17, 2024

sunshowers commented Dec 17, 2024

iliana commented Dec 17, 2024

iliana commented Dec 6, 2024 •

edited

Loading

iliana commented Dec 13, 2024 •

edited

Loading