Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Excessive rucio metadata lookup #118

Open
FaroutYLq opened this issue Dec 11, 2023 · 0 comments
Open

Excessive rucio metadata lookup #118

FaroutYLq opened this issue Dec 11, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@FaroutYLq
Copy link
Collaborator

We crashed rucio server, and one suspect is that outsource jobs are doing overwhelming rucio metadata lookup. One example from a peaklet job

[root@rucio-xenon rucio_httpd]# grep mwt2-c071.campuscluster.illinois.edu httpd_access_log | grep 11/Dec/2023:12
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:29:12 -0600] "GET /auth/x509_proxy HTTP/1.1" 200 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:29:12 -0600] "GET /accounts/whoami HTTP/1.1" 303 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:29:12 -0600] "GET /accounts/production HTTP/1.1" 200 202
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/aqmon_hits-gsmdrzd6gz HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/peak_classification_bayes-gkrokfhp5z HTTP/1.1" 404 143
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/corrected_areas-o44rjrrdci HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records-rfzvpzj4mf HTTP/1.1" 200 186
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records-rfzvpzj4mf/rules HTTP/1.1" 200 1630
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_he-rfzvpzj4mf HTTP/1.1" 200 189
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_he-rfzvpzj4mf/rules HTTP/1.1" 200 1632
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon-rfzvpzj4mf HTTP/1.1" 200 192
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon-rfzvpzj4mf/rules HTTP/1.1" 200 1638
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_nv-rfzvpzj4mf HTTP/1.1" 200 189
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_nv-rfzvpzj4mf/rules HTTP/1.1" 200 1632
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon_nv-rfzvpzj4mf HTTP/1.1" 200 195
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aqmon_nv-rfzvpzj4mf/rules HTTP/1.1" 200 1643
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:27 -0600] "GET /dids/xnt_049794/raw_records_aux_mv-rfzvpzj4mf HTTP/1.1" 200 193
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_aux_mv-rfzvpzj4mf/rules HTTP/1.1" 200 1639
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_mv-rfzvpzj4mf HTTP/1.1" 200 189
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_mv-rfzvpzj4mf/rules HTTP/1.1" 200 1632
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/detector_time_offsets-xyl6d4sclw HTTP/1.1" 404 139
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/distinct_channels-qzbcespqp3 HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/energy_estimates-c2fdcw5wcr HTTP/1.1" 404 134
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_ambience-b7qndgom42 HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_area_per_channel-ywkbseoogl HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_n_channel-ywkbseoogl HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_basics-o5culyytu5 HTTP/1.1" 404 130
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_info-5ovx4l2iul HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_info_double-22sfnn5kim HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_ms_naive-epbbboxkuj HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_pattern_fit-cahobtcvru HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_per_event-klsx7noczh HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_positions-dayaj7r4yj HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s1_positions_cnn-hvvmp65uq2 HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s2_positions_cnn-bh27t2hzlw HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s2_positions_gcn-d5gu4vewsv HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_s2_positions_mlp-f6ytzu6qqn HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_shadow-zum2aprycl HTTP/1.1" 404 130
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_top_bottom_params-urabkrxdqc HTTP/1.1" 404 141
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_waveform-loszf2t5d6 HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_w_bayes_class-yhok6eq7b3 HTTP/1.1" 404 137
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/raw_records_diagnostic-5jnmouazik HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/individual_peak_monitor-spjp2pl4hl HTTP/1.1" 404 141
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/afterpulses-uf63o427z7 HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/led_calibration-lsdigsccxn HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/event_local_min_info-hazyatbm5u HTTP/1.1" 404 138
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/merged_s2s-wpzg6lsm2m HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/merged_s2s_he-hohrh5yh6o HTTP/1.1" 404 131
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/online_monitor_mv-if3qqljiby HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/online_monitor_nv-z2cushqctj HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/online_peak_monitor-etr2pjxerr HTTP/1.1" 404 137
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_ambience-aglg7ad3jl HTTP/1.1" 404 131
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_basics-6bdyxhzzfz HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_basics_he-4bjhf4bqfd HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_corrections-tqortpgfmp HTTP/1.1" 404 134
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_positions_cnn-iongm54rho HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_positions_gcn-sgi5hgfujv HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_positions_mlp-zfge4gdktl HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_proximity-abfkn7nmwv HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_s1_positions_cnn-vfvtclcw3u HTTP/1.1" 404 139
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_shadow-pctjku7ovb HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peak_top_bottom_params-366morvvx5 HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklet_classification-p3m6pr2fhz HTTP/1.1" 404 140
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklet_classification_he-jc5epv3vtq HTTP/1.1" 404 143
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklets-euocvpkv3y HTTP/1.1" 200 183
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklets-euocvpkv3y/rules HTTP/1.1" 200 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/lone_hits-euocvpkv3y HTTP/1.1" 200 184
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/lone_hits-euocvpkv3y/rules HTTP/1.1" 200 -
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaklets_he-qsiaztqf6c HTTP/1.1" 404 129
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaks-5i3zhnt5vx HTTP/1.1" 404 123
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/peaks_he-iakptlspfu HTTP/1.1" 404 126
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/records-hwcemk7dbt HTTP/1.1" 404 125
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:28 -0600] "GET /dids/xnt_049794/veto_regions-hwcemk7dbt HTTP/1.1" 200 187
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/veto_regions-hwcemk7dbt/rules HTTP/1.1" 200 803
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/pulse_counts-hwcemk7dbt HTTP/1.1" 200 187
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/pulse_counts-hwcemk7dbt/rules HTTP/1.1" 200 803
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/records_he-cbkz5nmyc4 HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/pulse_counts_he-cbkz5nmyc4 HTTP/1.1" 404 133
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/s2_recon_pos_diff-5srxt25yui HTTP/1.1" 404 135
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/veto_intervals-hyu26wo3bv HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/veto_proximity-c2u2t7vf5z HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/events_sync_mv-e3gwcxlgee HTTP/1.1" 404 132
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/events_mv-5wxwp4k6sz HTTP/1.1" 404 127
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/hitlets_mv-yf3ezxotyn HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/records_mv-gsrdzepgyt HTTP/1.1" 404 128
mwt2-c071.campuscluster.illinois.edu - - [11/Dec/2023:12:31:29 -0600] "GET /dids/xnt_049794/event_positions_nv-l6qg7r6k5v HTTP/1.1" 404 136
mwt2-c071.campuscluster.illinois.edu...

It is suspected to be what strax's rucio backend is doing. The question here:

  • Can we bypass this in strax?
  • If not, can we download manually what we need and then not using rucio remote?
@FaroutYLq FaroutYLq added the enhancement New feature or request label Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@FaroutYLq