Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update CI benchmarks workflows #1255

Merged
merged 1 commit into from
Nov 18, 2024
Merged

Update CI benchmarks workflows #1255

merged 1 commit into from
Nov 18, 2024

Conversation

GitGab19
Copy link
Collaborator

@GitGab19 GitGab19 commented Nov 14, 2024 •
edited
Loading

This PR updates the workflows used to run benchmarks on our CI. According to @epompeii, we need to add permissions: checks: write to solve errors we are getting when merging PRs to main.

This PR also adds a new workflow to automatically archive benches from forks after PRs are merged/closed.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

🐰 Bencher Report

Branchupdate-bencher-workflow
Testbedsv2
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
client_sv2_handle_message_common📈 view plot
🚷 view threshold		44.09
(-1.20%)		47.84
(92.15%)
client_sv2_handle_message_mining📈 view plot
🚷 view threshold		92.60
(+19.02%)		118.84
(77.92%)
client_sv2_mining_message_submit_standard📈 view plot
🚷 view threshold		14.70
(+0.21%)		14.76
(99.61%)
client_sv2_mining_message_submit_standard_serialize📈 view plot
🚷 view threshold		266.42
(-0.01%)		323.34
(82.40%)
client_sv2_mining_message_submit_standard_serialize_deserialize📈 view plot
🚷 view threshold		650.35
(+8.62%)		766.09
(84.89%)
client_sv2_open_channel📈 view plot
🚷 view threshold		149.34
(+1.36%)		155.38
(96.11%)
client_sv2_open_channel_serialize📈 view plot
🚷 view threshold		278.09
(-2.55%)		320.40
(86.79%)
client_sv2_open_channel_serialize_deserialize📈 view plot
🚷 view threshold		379.23
(+0.62%)		416.22
(91.11%)
client_sv2_setup_connection📈 view plot
🚷 view threshold		155.78
(-2.97%)		182.12
(85.53%)
client_sv2_setup_connection_serialize📈 view plot
🚷 view threshold		477.57
(+2.86%)		578.17
(82.60%)
client_sv2_setup_connection_serialize_deserialize📈 view plot
🚷 view threshold		989.36
(+1.88%)		1,086.72
(91.04%)
🐰 View full continuous benchmarking report in Bencher

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

🐰 Bencher Report

Branchupdate-bencher-workflow
Testbedsv1
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
client-submit-serialize📈 view plot
🚷 view threshold		6,672.90
(-0.49%)		7,155.73
(93.25%)
client-submit-serialize-deserialize📈 view plot
🚷 view threshold		7,500.30
(-1.51%)		8,065.02
(93.00%)
client-submit-serialize-deserialize-handle/client-submit-serialize-deserialize-handle📈 view plot
🚷 view threshold		8,158.80
(-0.26%)		8,595.01
(94.92%)
client-sv1-authorize-serialize-deserialize-handle/client-sv1-authorize-serialize-deserialize-handle📈 view plot
🚷 view threshold		884.40
(+1.28%)		950.66
(93.03%)
client-sv1-authorize-serialize-deserialize/client-sv1-authorize-serialize-deserialize📈 view plot
🚷 view threshold		682.67
(+1.31%)		738.20
(92.48%)
client-sv1-authorize-serialize/client-sv1-authorize-serialize📈 view plot
🚷 view threshold		244.19
(-1.54%)		264.70
(92.25%)
client-sv1-get-authorize/client-sv1-get-authorize📈 view plot
🚷 view threshold		155.86
(-0.94%)		165.95
(93.92%)
client-sv1-get-submit📈 view plot
🚷 view threshold		6,377.80
(-1.42%)		7,199.47
(88.59%)
client-sv1-get-subscribe/client-sv1-get-subscribe📈 view plot
🚷 view threshold		298.44
(+6.29%)		332.90
(89.65%)
client-sv1-subscribe-serialize-deserialize-handle/client-sv1-subscribe-serialize-deserialize-handle📈 view plot
🚷 view threshold		742.64
(+1.07%)		814.09
(91.22%)
client-sv1-subscribe-serialize-deserialize/client-sv1-subscribe-serialize-deserialize📈 view plot
🚷 view threshold		597.00
(+0.01%)		633.61
(94.22%)
client-sv1-subscribe-serialize/client-sv1-subscribe-serialize📈 view plot
🚷 view threshold		202.79
(-1.92%)		230.95
(87.81%)
🐰 View full continuous benchmarking report in Bencher

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

🐰 Bencher Report

Branchupdate-bencher-workflow
Testbedsv1
Click to view all benchmark results
BenchmarkEstimated CyclesBenchmark Result
estimated cycles
(Result Δ%)		Upper Boundary
estimated cycles
(Limit %)		InstructionsBenchmark Result
instructions
(Result Δ%)		Upper Boundary
instructions
(Limit %)		L1 AccessesBenchmark Result
accesses
(Result Δ%)		Upper Boundary
accesses
(Limit %)		L2 AccessesBenchmark Result
accesses
(Result Δ%)		Upper Boundary
accesses
(Limit %)		RAM AccessesBenchmark Result
accesses
(Result Δ%)		Upper Boundary
accesses
(Limit %)
get_authorize📈 view plot
🚷 view threshold		8,425.00
(-0.36%)		8,830.86
(95.40%)		📈 view plot
🚷 view threshold		3,689.00
(-1.55%)		3,892.73
(94.77%)		📈 view plot
🚷 view threshold		5,155.00
(-1.90%)		5,462.75
(94.37%)		📈 view plot
🚷 view threshold		10.00
(+39.13%)		12.94
(77.27%)		📈 view plot
🚷 view threshold		92.00
(+1.74%)		96.22
(95.61%)
get_submit📈 view plot
🚷 view threshold		95,483.00
(+0.05%)		96,078.05
(99.38%)		📈 view plot
🚷 view threshold		59,346.00
(-0.20%)		59,768.99
(99.29%)		📈 view plot
🚷 view threshold		85,218.00
(-0.23%)		85,855.52
(99.26%)		📈 view plot
🚷 view threshold		44.00
(-2.59%)		64.62
(68.09%)		📈 view plot
🚷 view threshold		287.00
(+2.54%)		289.60
(99.10%)
get_subscribe📈 view plot
🚷 view threshold		7,906.00
(-1.34%)		8,388.12
(94.25%)		📈 view plot
🚷 view threshold		2,765.00
(-2.41%)		2,972.69
(93.01%)		📈 view plot
🚷 view threshold		3,846.00
(-2.89%)		4,155.95
(92.54%)		📈 view plot
🚷 view threshold		14.00
(+16.82%)		20.69
(67.67%)		📈 view plot
🚷 view threshold		114.00
(-0.08%)		120.16
(94.87%)
serialize_authorize📈 view plot
🚷 view threshold		12,248.00
(-0.21%)		12,653.61
(96.79%)		📈 view plot
🚷 view threshold		5,266.00
(-0.99%)		5,462.52
(96.40%)		📈 view plot
🚷 view threshold		7,323.00
(-1.27%)		7,624.09
(96.05%)		📈 view plot
🚷 view threshold		12.00
(+32.30%)		15.87
(75.61%)		📈 view plot
🚷 view threshold		139.00
(+1.12%)		143.67
(96.75%)
serialize_deserialize_authorize📈 view plot
🚷 view threshold		24,412.00
(-1.09%)		25,287.05
(96.54%)		📈 view plot
🚷 view threshold		9,838.00
(-0.53%)		10,056.69
(97.83%)		📈 view plot
🚷 view threshold		13,887.00
(-0.54%)		14,212.47
(97.71%)		📈 view plot
🚷 view threshold		33.00
(-5.04%)		44.27
(74.55%)		📈 view plot
🚷 view threshold		296.00
(-1.75%)		315.81
(93.73%)
serialize_deserialize_handle_authorize📈 view plot
🚷 view threshold		30,110.00
(-0.68%)		30,796.22
(97.77%)		📈 view plot
🚷 view threshold		12,015.00
(-0.54%)		12,227.44
(98.26%)		📈 view plot
🚷 view threshold		17,000.00
(-0.63%)		17,322.01
(98.14%)		📈 view plot
🚷 view threshold		60.00
(+7.61%)		68.38
(87.75%)		📈 view plot
🚷 view threshold		366.00
(-0.92%)		381.93
(95.83%)
serialize_deserialize_handle_submit📈 view plot
🚷 view threshold		126,441.00
(-0.02%)		127,148.01
(99.44%)		📈 view plot
🚷 view threshold		73,200.00
(-0.12%)		73,652.02
(99.39%)		📈 view plot
🚷 view threshold		104,911.00
(-0.14%)		105,611.17
(99.34%)		📈 view plot
🚷 view threshold		106.00
(-1.08%)		133.59
(79.35%)		📈 view plot
🚷 view threshold		600.00
(+0.64%)		605.62
(99.07%)
serialize_deserialize_handle_subscribe📈 view plot
🚷 view threshold		27,524.00
(-1.08%)		28,650.81
(96.07%)		📈 view plot
🚷 view threshold		9,584.00
(-0.65%)		9,791.58
(97.88%)		📈 view plot
🚷 view threshold		13,544.00
(-0.74%)		13,852.97
(97.77%)		📈 view plot
🚷 view threshold		66.00
(+3.98%)		76.03
(86.81%)		📈 view plot
🚷 view threshold		390.00
(-1.53%)		417.22
(93.48%)
serialize_deserialize_submit📈 view plot
🚷 view threshold		115,163.00
(-0.06%)		115,962.02
(99.31%)		📈 view plot
🚷 view threshold		68,060.00
(-0.06%)		68,477.12
(99.39%)		📈 view plot
🚷 view threshold		97,653.00
(-0.09%)		98,312.69
(99.33%)		📈 view plot
🚷 view threshold		65.00
(+3.42%)		78.58
(82.72%)		📈 view plot
🚷 view threshold		491.00
(0.00%)		499.69
(98.26%)
serialize_deserialize_subscribe📈 view plot
🚷 view threshold		22,914.00
(-1.42%)		24,077.43
(95.17%)		📈 view plot
🚷 view threshold		8,143.00
(-0.72%)		8,348.13
(97.54%)		📈 view plot
🚷 view threshold		11,464.00
(-0.78%)		11,760.23
(97.48%)		📈 view plot
🚷 view threshold		36.00
(-3.54%)		44.70
(80.54%)		📈 view plot
🚷 view threshold		322.00
(-2.02%)		349.08
(92.24%)
serialize_submit📈 view plot
🚷 view threshold		99,872.00
(+0.02%)		100,462.87
(99.41%)		📈 view plot
🚷 view threshold		61,408.00
(-0.16%)		61,811.62
(99.35%)		📈 view plot
🚷 view threshold		88,077.00
(-0.20%)		88,699.68
(99.30%)		📈 view plot
🚷 view threshold		49.00
(+4.27%)		65.19
(75.16%)		📈 view plot
🚷 view threshold		330.00
(+1.68%)		334.20
(98.74%)
serialize_subscribe📈 view plot
🚷 view threshold		11,351.00
(-0.39%)		11,767.14
(96.46%)		📈 view plot
🚷 view threshold		4,118.00
(-1.50%)		4,318.33
(95.36%)		📈 view plot
🚷 view threshold		5,706.00
(-1.91%)		6,011.37
(94.92%)		📈 view plot
🚷 view threshold		16.00
(+24.73%)		20.54
(77.90%)		📈 view plot
🚷 view threshold		159.00
(+0.93%)		164.15
(96.86%)
🐰 View full continuous benchmarking report in Bencher

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 14, 2024 •
edited
Loading

🐰 Bencher Report

Branchupdate-bencher-workflow
Testbedsv2
🚨 5 ALERTS: Threshold Boundary Limits exceeded!
BenchmarkMeasure
Units		ViewBenchmark Result
(Result Δ%)		Lower Boundary
(Limit %)		Upper Boundary
(Limit %)
client_sv2_mining_message_submit_standard_serialize_deserializeEstimated Cycles
estimated cycles		📈 plot
🚨 alert
🚷 threshold		27,758.00
(+0.84%)		27,742.47
(100.06%)
client_sv2_mining_message_submit_standard_serialize_deserializeInstructions
instructions		📈 plot
🚨 alert
🚷 threshold		10,645.00
(+0.62%)		10,643.34
(100.02%)
client_sv2_mining_message_submit_standard_serialize_deserializeL1 Accesses
accesses		📈 plot
🚨 alert
🚷 threshold		15,503.00
(+0.71%)		15,494.64
(100.05%)
client_sv2_mining_message_submit_standard_serialize_deserializeL2 Accesses
accesses		📈 plot
🚨 alert
🚷 threshold		92.00
(+12.97%)		90.41
(101.76%)
client_sv2_open_channel_serialize_deserializeL2 Accesses
accesses		📈 plot
🚨 alert
🚷 threshold		87.00
(+16.90%)		85.60
(101.64%)
Click to view all benchmark results
BenchmarkEstimated CyclesBenchmark Result
estimated cycles
(Result Δ%)		Upper Boundary
estimated cycles
(Limit %)		InstructionsBenchmark Result
instructions
(Result Δ%)		Upper Boundary
instructions
(Limit %)		L1 AccessesBenchmark Result
accesses
(Result Δ%)		Upper Boundary
accesses
(Limit %)		L2 AccessesBenchmark Result
accesses
(Result Δ%)		Upper Boundary
accesses
(Limit %)		RAM AccessesBenchmark Result
accesses
(Result Δ%)		Upper Boundary
accesses
(Limit %)
client_sv2_handle_message_common📈 view plot
🚷 view threshold		2,099.00
(+0.32%)		2,203.56
(95.25%)		📈 view plot
🚷 view threshold		473.00
(-0.04%)		488.74
(96.78%)		📈 view plot
🚷 view threshold		734.00
(-0.23%)		757.85
(96.85%)		📈 view plot
🚷 view threshold		7.00
(+27.64%)		14.24
(49.15%)		📈 view plot
🚷 view threshold		38.00
(+0.06%)		41.09
(92.47%)
client_sv2_handle_message_mining📈 view plot
🚷 view threshold		8,166.00
(-0.47%)		8,289.33
(98.51%)		📈 view plot
🚷 view threshold		2,137.00
(-0.03%)		2,143.08
(99.72%)		📈 view plot
🚷 view threshold		3,161.00
(+0.01%)		3,170.58
(99.70%)		📈 view plot
🚷 view threshold		35.00
(-1.58%)		42.23
(82.88%)		📈 view plot
🚷 view threshold		138.00
(-0.74%)		140.96
(97.90%)
client_sv2_mining_message_submit_standard📈 view plot
🚷 view threshold		6,289.00
(+0.17%)		6,386.95
(98.47%)		📈 view plot
🚷 view threshold		1,750.00
(-0.04%)		1,767.13
(99.03%)		📈 view plot
🚷 view threshold		2,549.00
(-0.18%)		2,576.15
(98.95%)		📈 view plot
🚷 view threshold		20.00
(+17.49%)		24.77
(80.75%)		📈 view plot
🚷 view threshold		104.00
(+0.02%)		106.96
(97.23%)
client_sv2_mining_message_submit_standard_serialize📈 view plot
🚷 view threshold		14,732.00
(+0.01%)		14,965.15
(98.44%)		📈 view plot
🚷 view threshold		4,694.00
(-0.02%)		4,711.13
(99.64%)		📈 view plot
🚷 view threshold		6,752.00
(-0.10%)		6,782.87
(99.54%)		📈 view plot
🚷 view threshold		49.00
(+12.48%)		53.66
(91.32%)		📈 view plot
🚷 view threshold		221.00
(-0.25%)		227.71
(97.05%)
client_sv2_mining_message_submit_standard_serialize_deserialize📈 view plot
🚨 view alert
🚷 view threshold		27,758.00
(+0.84%)		27,742.47
(100.06%)		📈 view plot
🚨 view alert
🚷 view threshold		10,645.00
(+0.62%)		10,643.34
(100.02%)		📈 view plot
🚨 view alert
🚷 view threshold		15,503.00
(+0.71%)		15,494.64
(100.05%)		📈 view plot
🚨 view alert
🚷 view threshold		92.00
(+12.97%)		90.41
(101.76%)		📈 view plot
🚷 view threshold		337.00
(+0.59%)		341.28
(98.75%)
client_sv2_open_channel📈 view plot
🚷 view threshold		4,417.00
(+0.45%)		4,611.96
(95.77%)		📈 view plot
🚷 view threshold		1,461.00
(-0.01%)		1,476.74
(98.93%)		📈 view plot
🚷 view threshold		2,157.00
(-0.12%)		2,185.41
(98.70%)		📈 view plot
🚷 view threshold		11.00
(+22.22%)		17.77
(61.91%)		📈 view plot
🚷 view threshold		63.00
(+0.56%)		68.39
(92.12%)
client_sv2_open_channel_serialize📈 view plot
🚷 view threshold		14,024.00
(-0.26%)		14,448.77
(97.06%)		📈 view plot
🚷 view threshold		5,064.00
(-0.00%)		5,079.74
(99.69%)		📈 view plot
🚷 view threshold		7,324.00
(-0.02%)		7,353.16
(99.60%)		📈 view plot
🚷 view threshold		38.00
(+8.06%)		43.43
(87.50%)		📈 view plot
🚷 view threshold		186.00
(-0.75%)		198.59
(93.66%)
client_sv2_open_channel_serialize_deserialize📈 view plot
🚷 view threshold		22,866.00
(+0.89%)		22,907.78
(99.82%)		📈 view plot
🚷 view threshold		8,040.00
(+0.25%)		8,074.89
(99.57%)		📈 view plot
🚷 view threshold		11,686.00
(+0.17%)		11,755.22
(99.41%)		📈 view plot
🚨 view alert
🚷 view threshold		87.00
(+16.90%)		85.60
(101.64%)		📈 view plot
🚷 view threshold		307.00
(+1.13%)		311.80
(98.46%)
client_sv2_setup_connection📈 view plot
🚷 view threshold		4,689.00
(+0.11%)		4,771.48
(98.27%)		📈 view plot
🚷 view threshold		1,502.00
(-0.01%)		1,517.74
(98.96%)		📈 view plot
🚷 view threshold		2,274.00
(-0.19%)		2,299.53
(98.89%)		📈 view plot
🚷 view threshold		14.00
(+49.58%)		15.92
(87.93%)		📈 view plot
🚷 view threshold		67.00
(-0.58%)		69.80
(95.99%)
client_sv2_setup_connection_serialize📈 view plot
🚷 view threshold		16,164.00
(-0.06%)		16,466.01
(98.17%)		📈 view plot
🚷 view threshold		5,963.00
(-0.00%)		5,978.74
(99.74%)		📈 view plot
🚷 view threshold		8,659.00
(-0.05%)		8,692.74
(99.61%)		📈 view plot
🚷 view threshold		45.00
(+12.43%)		53.63
(83.90%)		📈 view plot
🚷 view threshold		208.00
(-0.40%)		216.49
(96.08%)
client_sv2_setup_connection_serialize_deserialize📈 view plot
🚷 view threshold		35,704.00
(+0.43%)		35,747.34
(99.88%)		📈 view plot
🚷 view threshold		14,888.00
(+0.27%)		14,906.81
(99.87%)		📈 view plot
🚷 view threshold		21,879.00
(+0.31%)		21,918.42
(99.82%)		📈 view plot
🚷 view threshold		98.00
(+5.47%)		114.39
(85.67%)		📈 view plot
🚷 view threshold		381.00
(+0.46%)		384.65
(99.05%)
🐰 View full continuous benchmarking report in Bencher

@codecov Codecov
Copy link

codecov bot commented Nov 14, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 19.30%. Comparing base (b900d0a) to head (c54af5e).
Report is 2 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #1255   +/-   ##
=======================================
  Coverage   19.30%   19.30%           
=======================================
  Files         164      164           
  Lines       10849    10849           
=======================================
  Hits         2094     2094           
  Misses       8755     8755
Flag Coverage Δ
binary_codec_sv2-coverage 0.00% <ø> (ø)
binary_serde_sv2-coverage 3.65% <ø> (ø)
binary_sv2-coverage 5.46% <ø> (ø)
bip32_derivation-coverage 0.00% <ø> (ø)
buffer_sv2-coverage 25.02% <ø> (ø)
codec_sv2-coverage 0.01% <ø> (ø)
common_messages_sv2-coverage 0.13% <ø> (ø)
const_sv2-coverage 0.00% <ø> (ø)
error_handling-coverage 0.00% <ø> (ø)
framing_sv2-coverage 0.29% <ø> (ø)
jd_client-coverage 0.00% <ø> (ø)
jd_server-coverage 7.79% <ø> (ø)
job_declaration_sv2-coverage 0.00% <ø> (ø)
key-utils-coverage 2.39% <ø> (ø)
mining-coverage 2.49% <ø> (-0.02%) ⬇️
mining_device-coverage 0.00% <ø> (ø)
mining_proxy_sv2-coverage 0.70% <ø> (ø)
noise_sv2-coverage 4.35% <ø> (ø)
pool_sv2-coverage 1.38% <ø> (ø)
protocols 24.72% <ø> (ø)
roles 6.55% <ø> (ø)
roles_logic_sv2-coverage 8.06% <ø> (ø)
sv1-mining-device-coverage 0.00% <ø> (ø)
sv2_ffi-coverage 0.00% <ø> (ø)
template_distribution_sv2-coverage 0.00% <ø> (ø)
translator_sv2-coverage 9.60% <ø> (ø)
utils 25.13% <ø> (ø)
v1-coverage 2.47% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@GitGab19 GitGab19 force-pushed the update-bencher-workflow branch from 45d2ce4 to 28ba220 Compare November 14, 2024 20:43
Shourya742
Shourya742 suggested changes Nov 15, 2024
View reviewed changes
Copy link
Contributor

@Shourya742 Shourya742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nits.

.github/workflows/run-and-track-benchmarks-on-main.yaml Outdated Show resolved Hide resolved
.github/workflows/run-and-track-benchmarks-on-main.yaml Outdated Show resolved Hide resolved
.github/workflows/run-and-track-benchmarks-on-main.yaml Show resolved Hide resolved
.github/workflows/fork_pr_benchmark_closed.yaml Outdated Show resolved Hide resolved
@GitGab19 GitGab19 force-pushed the update-bencher-workflow branch from 28ba220 to 5c60255 Compare November 15, 2024 08:45
@GitGab19 GitGab19 force-pushed the update-bencher-workflow branch from 5c60255 to c54af5e Compare November 15, 2024 08:48
Shourya742
Shourya742 approved these changes Nov 17, 2024
View reviewed changes
Copy link
Contributor

@Shourya742 Shourya742 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK

@GitGab19 GitGab19 merged commit 8c98647 into stratum-mining:main Nov 18, 2024
38 checks passed
@GitGab19 GitGab19 deleted the update-bencher-workflow branch November 18, 2024 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@plebhash plebhash plebhash approved these changes

@Shourya742 Shourya742 Shourya742 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GitGab19 @Shourya742 @plebhash