Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARM64: social-network-mcrouter pod has error while running ./wrk AsyncSocketException: AsyncMcClient, type = Socket not open #264

Open
ahalarnkar-ampere opened this issue Jan 30, 2023 · 3 comments

Comments

@ahalarnkar-ampere
Copy link

Hi,

I have built mcrouter from github/facebook/mcrouter for arm64. I have the mcrouter to docker hub and using this link in the helm-chart/socialnetwork/charts/mcrouter/values.yaml to pull the image from dockerhub.

The mcrouter container with 3 memcached instances in a cluster...
I am running into an error when I try to run a workload via a LoadBalancer...

Any hints would be helpful. Thanks in advance. Attached logs below.

I0124 10:22:47.529297 1 StandaloneUtils.cpp:479] .0 mcrouter startup (1)
I0124 10:22:47.529446 1 StandaloneUtils.cpp:192] Starting Memcache router
I0124 10:22:47.529476 1 Server-inl.h:496] Spawning AsyncMcServer
I0124 10:22:47.530237 1 FunctionScheduler.cpp:397] Starting FunctionScheduler with 0 functions.
I0124 10:22:47.530498 1 CarbonRouterInstance-inl.h:628] creating config builder
I0124 10:22:47.531168 1 CarbonRouterInstance-inl.h:598] started reconfiguring
I0124 10:22:47.531427 1 CarbonRouterInstance-inl.h:616] reconfigured 1 proxies with 1 pools, 3 clients 6423f6330c93e5d05cbdf195daf66b7b)
W0124 10:22:47.531690 1 Server-inl.h:56] acl checker will not be enabled.

FAILURE 1674556123.762147 1 [AsyncMcClient] [bad-environment] [IOThreadPool0] network/AsyncMcClientImpl.cpp:362] AsyncSocketException: AsyncMcClient, type = Socket not open
"libmcrouter.mcrouter.5000":
{
"allow_only_gets":"0",
"async_spool":"/var/spool/mcrouter",
"asynclog_disable":"0",
"asynclog_port_override":"0",
"big_value_batch_size":"10",
"big_value_split_threshold":"0",
"client_queue_no_notify_rate":"0",
"client_queue_size":"1024",
"client_queue_wait_threshold_us":"0",
"collect_rxmit_stats_every_hz":"0",
"config":"",
"config_dump_root":"/var/mcrouter/config",
"config_file":"/etc/mcrouter/config.json",
"config_params":"",
"config_str":"",
"connect_timeout_retries":"0",
"constantly_reload_configs":"0",
"cross_cluster_timeout_ms":"0",
"cross_region_timeout_ms":"0",
"debug_fifo_root":"/var/mcrouter/fifos",
"default_qos_class":"0",
"default_qos_path":"0",
"default_route":"/././",
"disable_miss_on_arith_errors":"0",
"disable_reload_configs":"0",
"disable_request_deadline_check":"0",
"disable_shard_split_route":"0",
"disable_tko_tracking":"0",
"enable_axonlog":"0",
"enable_compression":"0",
"enable_failure_logging":"1",
"enable_flush_cmd":"0",
"enable_logging_route":"0",
"enable_partial_reconfigure":"0",
"enable_qos":"0",
"enable_send_to_main_shard_split":"1",
"enable_service_router":"0",
"enable_ssl_tfo":"0",
"external_carbon_connection_log_max_burst":"500",
"external_carbon_connection_log_rate_per_hour":"3600",
"external_carbon_connection_log_sample_rate":"10000",
"external_carbon_connection_logging_enabled":"0",
"failures_until_tko":"3",
"fibers_max_pool_size":"1000",
"fibers_pool_resize_period_ms":"60000",
"fibers_record_stack_size_every":"100000",
"fibers_stack_size":"24576",
"fibers_use_guard_pages":"1",
"file_observer_poll_period_ms":"100",
"file_observer_sleep_before_update_ms":"1000",
"flavor_name":"unknown",
"group_remote_errors":"0",
"jemalloc_nodump_buffers":"0",
"keepalive_cnt":"0",
"keepalive_idle_s":"300",
"keepalive_interval_s":"60",
"logging_rtt_outlier_threshold_us":"0",
"max_dumped_config_age":"43200",
"max_no_flush_event_loops":"5",
"max_rxmit_reconnect_threshold":"0",
"max_shadow_token_map_size":"1024",
"min_rxmit_reconnect_threshold":"0",
"miss_on_get_errors":"1",
"num_proxies":"1",
"pem_ca_path":"",
"pem_cert_path":"",
"pem_key_path":"",
"pool_stats_config_file":"",
"post_reconfiguration_delay_ms":"0",
"probe_delay_initial_ms":"10000",
"probe_delay_max_ms":"60000",
"proxy_max_inflight_requests":"0",
"proxy_max_inflight_shadow_requests":"0",
"proxy_max_throttled_requests":"0",
"reconfiguration_delay_ms":"1000",
"reconfiguration_jitter_ms":"0",
"reset_inactive_connection_interval":"60000",
"router_name":"5000",
"runtime_vars_file":"",
"rxmit_latency_deviation_us":"0",
"send_invalid_route_to_default":"0",
"server_timeout_ms":"1000",
"service_name":"mcrouter",
"ssl_connection_cache":"0",
"ssl_handshake_offload":"0",
"ssl_service_identity":"",
"ssl_service_identity_authorization_enforce":"0",
"ssl_service_identity_authorization_log":"0",
"ssl_verify_peers":"0",
"stats_async_queue_length":"50",
"stats_logging_interval":"10000",
"stats_root":"/var/mcrouter/stats",
"target_max_inflight_requests":"0",
"target_max_pending_requests":"100000",
"target_max_shadow_requests":"1000",
"tcp_rto_min":"-1",
"test_mode":"0",
"thread_affinity":"0",
"thrift_compression_threshold":"0",
"tls_prefer_ocb_cipher":"0",
"use_asynclog_version2":"0",
"version":".0 mcrouter",
"waiting_request_timeout_ms":"0",
"within_cluster_timeout_ms":"0"
}

This is the command:
mcrouter -p 5000 --config-file=/etc/mcrouter/config.json

Contents of config.json:

{
"pools": {
"A": {
"servers": [
// hosts of replicated pool, https://github.com/facebook/mcrouter/wiki/Replicated-pools-setup e.g.:
"social-network-0.social-network.social-network.svc.cluster.local:11211",
"social-network-1.social-network.social-network.svc.cluster.local:11211",
"social-network-2.social-network.social-network.svc.cluster.local:11211",
]
}
},
"route": {
"type": "OperationSelectorRoute",
"default_policy": "PoolRoute|A",
"operation_policies": {
"add": "AllFastestRoute|Pool|A",
"delete": "AllFastestRoute|Pool|A",
"get": "LatestRoute|Pool|A",
"set": "AllFastestRoute|Pool|A"
}
}
}

@ahalarnkar-ampere ahalarnkar-ampere changed the title ARM64: ARM64: social-network-mcrouter pod has error while running ./wrk AsyncSocketException: AsyncMcClient, type = Socket not open Jan 30, 2023
@przmk0
Copy link
Contributor

przmk0 commented Jan 30, 2023

Hey,
Are you sure it's the right name? social-network-0.social-network.social-network.svc.cluster.local:11211?
Do you really have double social-network in the FQDN?

@ahalarnkar-ampere
Copy link
Author

ahalarnkar-ampere commented Jan 31, 2023

Hi,

Here is the statement in bold inside charts/mcrouter/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "fullname" . }}
labels:
app: {{ template "name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
config.json: |-
{{- if .Values.mcrouterCommandParams.config }}
{{ .Values.mcrouterCommandParams.config | nindent 4}}
{{- else }}
{{- if .Values.memcached.enabled }}
{
"pools": {
"A": {
"servers": [
// hosts of replicated pool, https://github.com/facebook/mcrouter/wiki/Replicated-pools-setup e.g.:
{{- $global := . }}
{{- range $i, $e := until (.Values.memcached.replicaCount | int) }}
"{{ $global.Release.Name }}-{{ $i }}.{{ $global.Release.Name }}.{{ $global.Release.Namespace }}.svc.{{ $global.Values.clusterSuffix }}:11211",
{{- end }}
]
}
},
"route": {
"type": "OperationSelectorRoute",
"default_policy": "PoolRoute|A",
"operation_policies": {
"add": "AllFastestRoute|Pool|A",
"delete": "AllFastestRoute|Pool|A",
"get": "LatestRoute|Pool|A",
"set": "AllFastestRoute|Pool|A"
}
}
}
{{- end }}
{{- end }}

I am using the below command, where namespace name is "social-network" and release name is also "social-network"

helm install social-network ./socialnetwork/ -n social-network --set global.mongodb.sharding.enabled=true,global.mongodb.standalone.enabled=false,global.redis.cluster.enabled=true,global.redis.standalone.enabled=false,global.memcached.cluster.enabled=false,global.memcached.standalone.enabled=true,nginx-thrift.replicas=1,mongodb-sharded.shards=1,mongodb-sharded.shardsvr.dataNode.replicaCount=1,mcrouter.memcached.replicaCount=1,global.replicas=1 --timeout 10m0s

@ahalarnkar-ampere
Copy link
Author

Hi @przmk0
Do you have any update? Please let me know if you need more info?
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants