Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flocker 1.15.0 docker plugin issue Error looking up volume plugin flocker: legacy plugin: plugin not found. #2930

Open
digeler opened this issue Oct 23, 2016 · 7 comments
Open

Flocker 1.15.0 docker plugin issue Error looking up volume plugin flocker: legacy plugin: plugin not found. #2930

digeler opened this issue Oct 23, 2016 · 7 comments

Comments

@digeler
Copy link

digeler commented Oct 23, 2016 •
edited by wallrj
Loading

Hi ,
Trying to use flocker with docker plugin get the above error.
looking at plugin logs i see the following:

<E2><97><8F> flocker-docker-plugin.service - Flocker Docker Plugin
   Loaded: loaded (/usr/lib/systemd/system/flocker-docker-plugin.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2016-10-23 09:06:52 UTC; 4min 39s ago
 Main PID: 48157 (flocker-docker-)d/system/flocker-docker-plugin.service; disabl    Tasks: 3reset: enabled)
   Memory: 56.2Me (running) since Sun 2016-10-23 09:06:52 UTC; 4min 39s ago
      CPU: 7.598s(flocker-docker-)
   CGroup: /system.slice/flocker-docker-plugin.service
           <E2><94><94><E2><94><80>48157 /opt/flocker/bin/python /usr/sbin/flocker-docker-plugin --journald
      CPU: 7.598s
Oct 23 09:10:00 slaveswarm1 flocker-docker-plugin[48157]: {"task_uuid": "598f688c-5a74-4a19-90df-797314d576d5", "error": false, "timestamp": 1477213800.22446, "message": "Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7faf74fc0200>", "message_type": "twisted:log", "task_level": [2, 3]}Oct 23 09:10:00 slaveswarm1 flocker-docker-plugin[48157]: {"exception": "twisted.web._newclient.ResponseNeverReceived", "task_level": [2, 4], "action_type": "flocker:apiclient:http_request", "reason": "[<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3**_get_server_certificate', 'certificate verify failed'**)]>]", "timestamp": 1477213800.254202, "task_uuid": "598f688c-5a74-4a19-90df-797314d576d5", "action_status": "failed"}Oct 23 09:10:00 slaveswarm1 flocker-docker-plugin[48157]: {"exception": "twisted.web._newclient.ResponseNeverReceived", "task_level": [3], "action_type": "flocker:common:loop_until", "reason": "[<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]>]", "timestamp": 1477213800.254748, "task_uuid": "598f688c-5a74-4a19-90df-797314d576d5", "action_status": "failed"}
Oct 23 09:10:00 slaveswarm1 flocker-docker-plugin[48157]: {"task_uuid": "a0272fb2-1d05-4ebe-89e6-f11bc19bafa2", "error": false, "timestamp": 1477213800.255262, "message": "Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7faf74fc0200>", "message_type": "twisted:log", "task_level": [1]}Oct 23 09:10:45 slaveswarm1 flocker-docker-plugin[48157]: {"predicate": {"function": "<function <lambda> at 0x7faf74feb230>", "line": 897, "file": "/opt/flocker/local/lib/python2.7/site-packages/flocker/apiclient/_client.py"}, "task_level": [1], "action_type": "flocker:common:loop_until", "timestamp": 1477213845.812243, "task_uuid": "3c53aeda-8e09-4fc3-a326-8ff105c324c2", "action_status": "started"}d": "598f688c-5a74-4a1Oct 23 09:10:45 slaveswarm1 flocker-docker-plugin[48157]: {"request_body": null, "url": "https://control-service:4523/v1/state/nodes/by_era/3e1498bc-c908-4374-9847-7fecd9f3d186", "timestamp": 1477213845.812934, "action_status": "started", "task_uuid": "3c53aeda-8e09-4fc3-a326-8ff105c324c2", "action_type": "flocker:apiclient:http_request", "method": "GET", "task_level": [2, 1]}
Oct 23 09:10:45 slaveswarm1 flocker-docker-plugin[48157]: {"task_uuid": "3c53aeda-8e09-4fc3-a326-8ff105c324c2", "error": false, "timestamp": 1477213845.824979, "message": "Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x7faf75014d88>", "message_type": "twisted:log", "task_level": [2, 3]}
Oct 23 09:10:45 slaveswarm1 flocker-docker-plugin[48157]: {"exception": "twisted.web._newclient.ResponseNeverReceived", "task_level": [2, 4], "action_type": "flocker:apiclient:http_request", "reason": "[<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]>]", "timestamp": 1477213845.855676, "task_uuid": "3c53aeda-8e09-4fc3-a326-8ff105c324c2", "action_status": "failed"}
Oct 23 09:10:45 slaveswarm1 flocker-docker-plugin[48157]: {"exception": "twisted.web._newclient.ResponseNeverReceived", "task_level": [3], "action_type": "flocker:common:loop_until", "reason": "[<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL **routines', 'ssl3_get_server_certificate', 'certificate verify failed')]>]"**, "timestamp": 1477213845.856383, "task_uuid": "3c53aeda-8e09-4fc3-a326-8ff105c324c2", "action_status": "failed"}
Oct 23 09:10:45 slaveswarm1 flocker-docker-plugin[48157]: {"task_uuid": "0c84d616-d4c9-42d2-bb71-9e532f9d69e4", "error": false, "timestamp": 1477213845.856926, "message": "Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x7faf75014d88>", "message_type": "twisted:log", "task_level": [1]}

Am trying to use the azure flocker driver from here : https://github.com/CatalystCode/azure-flocker-driver
this is my agent.yml :

"version": 1
"control-service":
    "hostname": "control-service"
    "port": 4524

dataset:
  backend: "azure_flocker_driver"
  client_id: "f0e3b718-fb05-46a2-9f99-d2beebe72f2b"
  tenant_id: "72f988bf-86f1-41af-91ab-2d7cd011db47"
  client_secret: "********_"
  subscription_id: "928f4e7e-2c28-4063-a56e-6f1e6**_**+"
  storage_account_name: "dmine9**"
  storage_account_key: "*******hF1RjYTfjBY+jWnNBCpqXoiKIOPLV41g6igDUr5a6kQcMZA=="
  storage_account_container: "vhds"
  group_name: "mine-masterswarm-031569"
  location: "North Europe"
  async_timeout: 100000
  debug: "false"

looking on the control service cert :

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            f7:cd:32:84:bc:78:08:ae:3d:70:2e:be:d8:0a:1b:b7
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU=6c70e21d-65c2-4f44-bf2d-9d880c81ccfe, CN=fcluster
        Validity
            Not Before: Oct 22 17:17:39 2016 GMT
            Not After : Oct 17 17:17:39 2036 GMT
        Subject: OU=6c70e21d-65c2-4f44-bf2d-9d880c81ccfe, CN=control-service
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

ca cert :

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            d4:bf:67:b2:7d:98:a9:d7:7d:c2:40:af:64:2e:ed:23
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU=6c70e21d-65c2-4f44-bf2d-9d880c81ccfe, CN=fcluster
        Validity
            Not Before: Oct 22 17:14:21 2016 GMT
            Not After : Oct 17 17:14:21 2036 GMT
        Subject: OU=6c70e21d-65c2-4f44-bf2d-9d880c81ccfe, CN=fcluster
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

ips : 10.0.1.5 control-service
10.0.1.6 -node

flocker control service logs:

<E2><97><8F> flocker-control.service - Flocker Control Service
   Loaded: loaded (/usr/lib/systemd/system/flocker-control.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2016-10-22 19:26:14 UTC; 13h ago
 Main PID: 29125 (flocker-control)
    Tasks: 2
   Memory: 44.1M
      CPU: 20min 43.835s
   CGroup: /system.slice/flocker-control.service
           <E2><94><94><E2><94><80>29125 /opt/flocker/bin/python /usr/sbin/flocker-control --port tcp:4523 --agent-port tcp:4524 --journald

Oct 23 09:24:57 masterswarm flocker-control[29125]: {"timestamp": 1477214697.128875, "task_uuid": "bc905ba2-de76-4bdc-a410-02213b4fb2df", "action_type": "flocker:controlservice:send_cluster_state", "action_status": "started", "task_level": [1]}
Oct 23 09:24:57 masterswarm flocker-control[29125]: {"task_uuid": "bc905ba2-de76-4bdc-a410-02213b4fb2df", "task_level": [2], "action_type": "flocker:controlservice:send_cluster_state", "timestamp": 1477214697.129518, "configuration": "null", "state": "null", "action_status": "succeeded"}
Oct 23 09:24:58 masterswarm flocker-control[29125]: {"task_uuid": "52a61b0d-a95f-4956-a944-242d47206d97", "error": false, "timestamp": 1477214698.865618, "message": "ControlAMP connection established (HOST:IPv4Address(TCP, '10.0.1.5', 4524) PEER:IPv4Address(TCP, '10.0.1.6', 41514))", "message_type": "twisted:log", "task_level": [1]}
Oct 23 09:24:58 masterswarm flocker-control[29125]: {"task_uuid": "270fdba2-c185-414f-b03c-5c5e99dad085", "action_status": "started", "action_type": "flocker:controlservice:agent_connected", "timestamp": 1477214698.86641, "agent": "IPv4Address(TCP, '10.0.1.6', 41514)", "task_level": [1]}
Oct 23 09:24:58 masterswarm flocker-control[29125]: {"timestamp": 1477214698.866884, "task_uuid": "270fdba2-c185-414f-b03c-5c5e99dad085", "action_type": "flocker:controlservice:agent_connected", "action_status": "succeeded", "task_level": [2]}
Oct 23 09:24:58 masterswarm flocker-control[29125]: {"task_uuid": "c297635b-8015-4de0-90ce-9fe7a105e985", "error": false, "timestamp": 1477214698.896355, "message": "ControlAMP connection lost (HOST:IPv4Address(TCP, '10.0.1.5', 4524) PEER:IPv4Address(TCP, '10.0.1.6', 41514))", "message_type": "twisted:log", "task_level": [1]}
Oct 23 09:24:59 masterswarm flocker-control[29125]: {"task_uuid": "4e281a23-111a-4bba-9801-eef243f52baf", "error": false, "timestamp": 1477214699.032024, "message": "ControlAMP connection established (HOST:IPv4Address(TCP, '10.0.1.5', 4524) PEER:IPv4Address(TCP, '10.0.1.6', 41516))", "message_type": "twisted:log", "task_level": [1]}
Oct 23 09:24:59 masterswarm flocker-control[29125]: {"task_uuid": "7ab7962d-9817-45c1-aa2e-e7ee489bb0a6", "action_status": "started", "action_type": "flocker:controlservice:agent_connected", "timestamp": 1477214699.032765, "agent": "IPv4Address(TCP, '10.0.1.6', 41516)", "task_level": [1]}
Oct 23 09:24:59 masterswarm flocker-control[29125]: {"timestamp": 1477214699.033196, "task_uuid": "7ab7962d-9817-45c1-aa2e-e7ee489bb0a6", "action_type": "flocker:controlservice:agent_connected", "action_status": "succeeded", "task_level": [2]}
Oct 23 09:24:59 masterswarm flocker-control[29125]: {"task_uuid": "ef42e26d-c755-4a5c-b3f0-5694ebd2defb", "error": false, "timestamp": 1477214699.059952, "message": "ControlAMP connection lost (HOST:IPv4Address(TCP, '10.0.1.5', 4524) PEER:IPv4Address(TCP, '10.0.1.6', 41516))", "message_type": "twisted:log", "task_level": [1]}

diagnostics here :
clusterhq_flocker_logs_87b1ec28-98ea-11e6-b5a5-000d3ab34f9f.zip

am missing something with the certs ? ,help appreciated
@wallrj
Copy link
Contributor

wallrj commented Oct 31, 2016

Hey @digeler

Sorry for the delayed reply.
And thanks for uploading your logs.

It looks like there's a problem with your flocker-control certificate or with the certificate verification on your system.

You're connecting to the flocker-control service using a hostname "control-service" and that seems to be the common name in your certificate, but also check the "Subject Alternative Name" of the control certificate.
Here's how it looks on one of our acceptance testing nodes:

# openssl x509  -in /etc/flocker/control-service.crt  -text | less
...
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:control-service, IP Address:104.196.XXX.XXX

Also check the output of curl when connecting to the flocker-control REST API port, using the control-service hostname and by IP address. E.g.

curl --verbose --cacert cluster.crt --cert $PWD/user.crt --key $PWD/user.key https://104.196.XXX.XXX:4523/v1/configuration/datasets
...

*   CAfile: cluster.crt
  CApath: none
* NSS: client certificate from file
*       subject: CN=user-allison,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa
*       start date: Oct 31 14:52:23 2016 GMT
*       expire date: Oct 26 14:52:23 2036 GMT
*       common name: user-allison
*       issuer: CN=acceptance-cluster,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa
* ALPN, server accepted to use http/1.1
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=control-service,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa
*       start date: Oct 31 14:52:22 2016 GMT
*       expire date: Oct 26 14:52:22 2036 GMT
*       common name: control-service
*       issuer: CN=acceptance-cluster,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa
> GET /v1/configuration/datasets HTTP/1.1
> Host: 104.196.200.246:4523
> User-Agent: curl/7.47.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Date: Mon, 31 Oct 2016 16:16:45 GMT
< X-Configuration-Tag: 44c0cc03d144555fba239aec12541c63
< Content-Type: application/json
< Server: TwistedWeb/16.4.1
<

@digeler
Copy link
Author

digeler commented Oct 31, 2016

thanks Richard i will update you soon.

On Mon, Oct 31, 2016 at 6:18 PM, Richard Wall [email protected]
wrote:

Hey @digeler https://github.com/digeler

Sorry for the delayed reply.
And thanks for uploading your logs.

It looks like there's a problem with your flocker-control certificate or
with the certificate verification on your system.

You're connecting to the flocker-control service using a hostname
"control-service" and that seems to be the common name in your certificate,
but also check the "Subject Alternative Name" of the control certificate.
Here's how it looks on one of our acceptance testing nodes:

openssl x509 -in /etc/flocker/control-service.crt -text | less

...
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:control-service, IP Address:104.196.XXX.XXX

Also check the output of curl when connecting to the flocker-control REST
API port, using the control-service hostname and by IP address. E.g.

curl --verbose --cacert cluster.crt --cert $PWD/user.crt --key $PWD/user.key https://104.196.XXX.XXX:4523/v1/configuration/datasets
...

  • CAfile: cluster.crt
    CApath: none
  • NSS: client certificate from file
  •   subject: CN=user-allison,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa

  •   start date: Oct 31 14:52:23 2016 GMT

  •   expire date: Oct 26 14:52:23 2036 GMT

  •   common name: user-allison

  •   issuer: CN=acceptance-cluster,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa
  • ALPN, server accepted to use http/1.1
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  •   subject: CN=control-service,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa

  •   start date: Oct 31 14:52:22 2016 GMT

  •   expire date: Oct 26 14:52:22 2036 GMT

  •   common name: control-service

  •   issuer: CN=acceptance-cluster,OU=dddc471f-9407-4f98-0296-aaaaaaaaaaaa

    GET /v1/configuration/datasets HTTP/1.1
    Host: 104.196.200.246:4523
    User-Agent: curl/7.47.1
    Accept: /

    < HTTP/1.1 200 OK
    < Transfer-Encoding: chunked
    < Date: Mon, 31 Oct 2016 16:16:45 GMT
    < X-Configuration-Tag: 44c0cc03d144555fba239aec12541c63
    < Content-Type: application/json
    < Server: TwistedWeb/16.4.1
    <

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2930 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuGGSMMtuA7rHBKRffEW9ioS2TYKQqEks5q5hTpgaJpZM4KeFsH
.

@digeler
Copy link
Author

digeler commented Nov 1, 2016

Hi Richard,

here is the output :
openssl x509 -in /etc/flocker/control-service.crt -text | less

X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:control-service, DNS:controlservice
Signature Algorithm: sha256WithRSAEncryption
17:77:65:c9:94:8a:94:f9:54:ed:ee:3b:d2:26:6a:a2:06:ff:
c3:fd:e2:b2:41:48:5c:a2:1f:a0:78:54:66:3c:f7:2e:d5:a2:
7b:d8:ba:7a:84:45:51:7e:37:2a:e8:0e:2c:07:ca:bc:6a:a3:

to create the certs i used the following :
flocker-ca initialize
flocker-ca create-control-certificate
flocker-ca create-node-certificate

root@masterswarm:/etc/flocker# curl --verbose --cacert cluster.crt --cert $PWD/plugin.crt --key $PWD/plugin.key https://10.0.1.5:4523/v1/configuration/datasets

  • Trying 10.0.1.5...
  • Connected to 10.0.1.5 (10.0.1.5) port 4523 (#0)
  • error reading ca cert file cluster.crt (Error in the certificate.)
  • Closing connection 0
    curl: (77) Problem with the SSL CA cert (path? access rights?)
    root@masterswarm:/etc/flocker#

do i need to use the certificates with fqdn ?
please let me know,am going to try this.

@digeler
Copy link
Author

digeler commented Nov 1, 2016 •
edited
Loading

i have changed the certs name to be fqdn :
i dont see any more cert errors:
but now when i try the plugin :
i get :

red property\n\nFailed validating 'required' in schema:\n {'$schema': 'http://json-schema.org/draft-04/schema#',\n 'properties': {'control-service': {'properties': {'hostname': {'format': 'hostname',\n 'type': 'string'},\n 'port': {'type': 'integer'}},\n 'required': ['hostname'],\n 'type': 'object'},\n 'dataset': {'properties': {'backend': {'type': 'string'}},\n 'required': ['backend'],\n 'type': 'object'},\n 'logging': {'type': 'object'},\n 'version': {'maximum': 1,\n 'minimum': 1,\n 'type': 'number'}},\n 'required': ['version', 'control-service', 'dataset'],\n 'type': 'object'}\n\nOn instance:\n {'control-server': {'hostname': 'flocker.local', 'port': 4524},\n 'dataset': {'async_timeout': 100000,\n 'backend': 'azure_flocker_driver',\n 'client_id': 'f0e3b718-fb05-46a2-9f99-d2beebe72f2b',\n 'client_secret': 'Clo$$$$$$q

',\n 'debug': 'false',\n 'group_name': 'mine-masterswarm-031569',\n 'location': 'North Europe',\n 'storage_account_container': 'vhds',\n 'storage_account_key': '+jWnNBCpqXoiKIOPLV41g6igDUr5a6kQcMZA==',\n 'storage_account_name': 'dmine9633',\n 'subscription_id': '928f4e7e-2c28-4063-a56e-6f1e6f2bb73c',\n 'tenant_id': '72f988bf-86f1-41af-91ab-2d7cd011db47'},\n 'version': 1}\n", "message_type": "twisted:log", "task_level": [1]}
Nov 01 15:36:14 slaveswarm1 flocker-docker-plugin[19240]: {"task_uuid": "ff215c6d-4d56-4c48-accd-ebbd9117dc79", "error": false, "timestamp": 1478014574.783666, "message": "Main loop terminated.", "message_type": "twisted:log", "task_level": [1]}
Nov 01 15:36:15 slaveswarm1 systemd[1]: flocker-docker-plugin.service: Main process exited, code=exited, status=1/FAILURE

files are here :
clusterhq_flocker_logs_d779b6ca-a045-11e6-89a9-000d3ab34f9f.zip
thanks again.

@clifinger
Copy link

Hi @digeler

Did you follow https://flocker-docs.clusterhq.com/en/latest/docker-integration/generate-api-plugin.html ?

Because you never speak about your plugin certificates ...

@wallnerryan
Copy link
Contributor

@digeler I am seeing the following error in your logs.

 File "/opt/flocker/local/lib/python2.7/site-packages/jsonschema/validators.py", line 123, in validate
            raise error
        jsonschema.exceptions.ValidationError: 'control-service' is a required property

What does your agent.yml look like?

@lmeyemezu
Copy link

Hi,
any feedback ?
I encounter the same issue.
I followed flocker installation documentation.
drives me crazy !!!
error reading ca cert file cluster.crt (Error in the certificate.) error
Regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@wallrj @wallnerryan @clifinger @digeler @lmeyemezu