Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REQUEST] Support for Silicon Labs EFR32 (Mighty Gecko family) based Zigbee radio adapters using newer EZSP serial protocol? #243

Closed
Hedda opened this issue Apr 21, 2020 · 228 comments
Closed

[REQUEST] Support for Silicon Labs EFR32 (Mighty Gecko family) based Zigbee radio adapters using newer EZSP serial protocol? #243

Hedda opened this issue Apr 21, 2020 · 228 comments

Comments

@Hedda
Copy link
Contributor

Hedda commented Apr 21, 2020 •
edited
Loading

Can the bellows library be made to support newer EZSP serial protocol such as EZSP v6, v7, and v8 when used with ZNet 6.x on Silicon Labs EFR32 (Mighty Gecko family) based Zigbee radio adapters?

See ex. #242 as that hardware is based on this same family and series of chips from Silicon Labs.

Zigbee EmberZNet version 6.7.0.0 has breaking change (with EZSP Protocol Version 8 and Secure EZSP Protocol Version 2) as both EZSP and Secure EZSP have adopted a new frame format with the following changes: (1) the fields of "Frame Control" and "Frame ID" are now two bytes; (2) no longer use "Legacy Frame ID"; (3) consume two bits of "Frame Control" to indicate the frame format version which is version 1 now.

Perhaps newer modules will dethrone Nortek HUSBZB-1 for ZHA as the must-have Zigbee adapter?
@Hedda Hedda mentioned this issue Apr 21, 2020
Closed
@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020 •
edited
Loading

Maybe someone can test EFR32MG12 based E180-ZG120B module from Ebyte works with bellows?

Ebyte makes an inexpensive development board called "E180-ZG120B-TB" for testing E180-ZG120B

This Zigbee 3.0 capable module has some very powerful MCU and radio specifications for its price:

This development is sold by cdebyte for less than $9 US-dollar on Aliexpress or about twice on eBay:

You can also buy them on bulk from Alibaba for less:

Looks like you can remove some jumpers to disable the USB converter if want to use serial directly.

E180-ZG120B by Chengdu Ebyte (CDEYTE) has 20 dBM powerful Zigbee 3.0 radio for 2.4GHz capable based on an EFR32MG1 Series 1 MCU SoC / chip, specifically EFR32MG1B (IC = EFR32MG1B232F256GM48), from Silicon Labs EFR32 ("Mighty Gecko") family.

EFR32MG1B232F256GM48 includes a 40 MHz ARM Cortex-M4 microcontroller with 256 Flash, 32 RAM and a rich peripheral set in a QFN48 package. With 19.5 dBm maximum output power and receive sensitivity of -101 dBm (250 kbps O-QPSK DSSS). Key Specs: 19.5 Output Power Max (dBm) / 120.5 Total Link Budget (dB).

I understand this also is classified as an Ember based radios using the EZSP (EmberZNet Serial Protocol) serial protocol (UART bus) interface so could therefore be made compatible with the bellows library for zigpy if the E180-ZG120B module is flashed with the right firmware?

I guess that my follow-up question will be which exact firmware to use on the E180-ZG120B module?

Ebyte is now making two EFR32MG1B based modules called E180-ZG120A and E180-ZG120B

E180-ZG120B (and old E180-ZG120A) module is sold by cdebyte on eBay and Aliexpress at low prices.

Example:

@Hedda Hedda changed the title [HARDWARE] Ebyte E180-ZG120B-TB Zigbee 3.0 development board with E180-ZG120B module based on EFR32MG1B Series 1 (Silicon Labs EFR32 family) [HARDWARE] Ebyte E180-ZG120B-TB Zigbee 3.0 development board with E180-ZG120B module based on EFR32MG12 Series 1 (Silicon Labs EFR32 family) Apr 21, 2020
@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020 •
edited
Loading

Silicon Labs also has an official devkit called "EFR32 Mighty Gecko Wireless Starter Kit", but it cost almost $500 US-dollars ($499 to be precise), however, that kit contains three (3) complete dev boards.

@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020

Z-Smart Systems now has driver code supporting latest Silicon Labs Ember dongles at @zsmartsystems

@Adminiuga
Copy link
Collaborator

Does ebyte provide any firmware?
I've ordered one we'll see what firmware it comes with. But in later version of EZSP they changed commands id to 16 bit.

@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020 •
edited
Loading

You could try contacting Ebyte / cdebyte (Chengdu Ebyte), however, I believe that the Ebyte module most likely only comes with standalone bootloader firmware preloaded for boot. Such standalone bootloader firmware usually is just enough to allow you to flash and upgrade the main application firmware via USB (without having to use a GPIO port / JTAG connector / debug adapter and Simplicity Studio or J-Link Downloader).

I would think you have to build/compile Zigbee stack (Zigbee EmberZNet) project with coordinator device type config in their SDK or tools ("Simplicity Studio" and "Simplicity Commander") and then flash application bootloader firmware with Zigbee stack yourself, ...but where to get that Zigbee stack and application bootloader firmware I don't know, maybe from a Silicon Labs SDK? As I understand they call protocol such as their Zigbee stack (Zigbee EmberZNet) for "apps".

Think these might be a couple of easier how-to guide for building EmberZNet for EFR32 and EM35x:

I don't own one myself yet but would be nice if could just install and start their latest SDK or tools ("Simplicity Studio" and "Simplicity Commander") and it would detect the hardware once it is plugged in to then start a new project.

Guessing that you first might need to register a serial number to your account for full SDK access, or at least register a dev kit part-number to your account, like for the official "EFR32 Mighty Gecko Wireless Starter Kit" (that kit part number is "SLWSTK6000B") as the dev board in that dev kit is kind similar to the Ebyte dev board as they both use the same type of chip from Silicon Labs .

Simplicity Studio

Simplicity Commander

Silicon Labs has its community knowledge-base here:

https://www.silabs.com/community/wireless/zigbee-and-thread/knowledge-base

Silicon Labs also have different user forums here:

Silicon Labs IoT also has a community on Slack:

https://silabsiot.slack.com/

Silicon Labs IoT also has a community on Gitter:

https://gitter.im/Silabs-IoT

@cdjackson from @zsmartsystems also mention two main NCP images in their README.md

Perhaps he would be willing to hint where to get the correct Zigbee coordinator firmware for EFR32?

@cdjackson also keep adding new EZSP support from UG100 to com.zsmartsystems.zigbee

@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020 •
edited
Loading

@sprut666666 as mentioned in #242 is also working on similar EFR32MG1 based adapter for @sprut

@Adminiuga
Copy link
Collaborator

So it boils down to getting Silicon Labs SDK which you only get with their dev-kit. But once you get SDK you can also get new image for the same chip as on HUSBZB-1 stick.

@Adminiuga
Copy link
Collaborator

Adminiuga commented Apr 21, 2020 •
edited
Loading

There're three dev boards in the mighty Gecko Silicon labs devkit. Just need to find two more volunteers willing to shell out $166 + taxes for a dev board and access to SDK which would allow to upgrade HUSBZB-1 to the latest emberznet.

@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020 •
edited
Loading

Yes same SDK, but I don't think that you can use the latest EmberZNet Zigbee Stack on older hardware.

That is, you can't upgrade EM35x based dongle with a Zigbee 3.0 stack, only latest Zigbee PRO stack. To run the latest EmberZNet Zigbee Stack with Zigbee 3.0 support you need EFR32 hardware.

Anyway, some posts on their forum hint that you don't need a dev kit serial number for the SDK:

You might only need to enter the kit part number for the official "EFR32 Mighty Gecko Wireless Starter Kit", and that kit part number is "SLWSTK6000B" (as it is kind of a similar development board with the same type of chip) and then just click a checkbox to accept the license agreement.

Then once someone has created hex/bin files with EmberZNet firmware then anyone can flash them themselves (I understand that is already done to get Telegesis ETRX3 adapters to work with bellows).

@Adminiuga
Copy link
Collaborator

not sure if we're reading the same posts, but those do tell you have to have access to dev-kit serial number
https://www.silabs.com/community/wireless/zigbee-and-thread/knowledge-base.entry.html/2017/11/22/access_to_siliconla-jk1S

@Hedda
Copy link
Contributor Author

Hedda commented Apr 21, 2020

not sure if we're reading the same posts, but those do tell you have to have access to dev-kit serial number
https://www.silabs.com/community/wireless/zigbee-and-thread/knowledge-base.entry.html/2017/11/22/access_to_siliconla-jk1S

Note that the original post there was posted more than two years ago so might have changed.

sslin (Employee) teplied Feb 21 2019, 2:40 AM in that thread with this:

Simplicity Studio is free to download and use even if you don't buy any kits.

JTAG driver will be installed automatically when you install the Simplicity Studio.

Some sdks/stack are free to download within the Simplicity Studio like Bluetooth, Flex, MCU, etc ...

With the serial number you got from the Mesh starter kit, you will be granted the access right to Zigbee and Thread for life long.

@cdjackson
Copy link

The HUSBZB does not contain a bootloader so cannot be updated.

@Adminiuga
Copy link
Collaborator

Adminiuga commented Apr 21, 2020 •
edited
Loading

Some sdks/stack are free to download within the Simplicity Studio like Bluetooth, Flex, MCU, etc .

Well, that might be true (for those stacks), but last time I've looked Emberznet stack was not accessible without a kit.

The HUSBZB does not contain a bootloader so cannot be updated

You send it a bootloader frame and it enters into bootload mode and you can upload .ebl image using xmodem.

@Adminiuga
Copy link
Collaborator

from HUSBZB in bootloader

EM3581 Serial Bootloader v5.4.1.0 b962                                       
1. upload ebl                                                                
2. run                                                                       
3. ebl info                                                                  
BL >

@cdjackson
Copy link

You send it a bootloader frame and it enters into bootload mode and you can upload .ebl image using xmodem.

Really? Have you tried this? For the HUSBs that I've looked they have not shown a bootloader being installed.

@Adminiuga
Copy link
Collaborator

send it launchStandaloneBootloader() EZSP command and afterwards connect to it with a serial terminal (might need different baudrate, i don't remember the details) It did enter the boot loader mode

image

@cdjackson
Copy link

cdjackson commented Apr 21, 2020 via email

@Adminiuga
Copy link
Collaborator

Adminiuga commented Apr 21, 2020 •
edited
Loading

I think it is might be up to the vendor to include boot loader into image, but both of Nortek GoControl HUSBZB-1 sticks in my possession respond to launchStandaloneBootloader command and were ready to accept .ebl image.

@cdjackson
Copy link

cdjackson commented Apr 21, 2020 via email

@Adminiuga
Copy link
Collaborator

Adminiuga commented Apr 21, 2020 •
edited
Loading 

debug: Send command getStandaloneBootloaderVersionPlatMicroPhy: ()
debug: Sending: b'2240215754bbef5f7e'
debug: Data frame: b'2340a15754bb05e65d9e49e6697e'
debug: Sending: b'83401b7e'
debug: Application frame 145 (getStandaloneBootloaderVersionPlatMicroPhy) received: b'1054040a03'
[21520, 4, 10, 3]
debug: Closed serial connection

it does.

@cdjackson
Copy link

cdjackson commented Apr 21, 2020 via email

@Adminiuga
Copy link
Collaborator

Adminiuga commented Apr 21, 2020 •
edited
Loading

Just to make it a bit easier #249

@Hedda
Copy link
Contributor Author

Hedda commented Apr 22, 2020 •
edited
Loading

There are otherwise plenty of instructions out there on how to flash Telegesis ETRX357 USB dongles, as you can't use them with bellows until you have first flashed them with other EmberZNet firmware, see:

https://github.com/zigpy/bellows/blob/dev/README.md

  • Telegesis ETRX357USB (Note! This first have to be flashed with other EmberZNet firmware)
  • Telegesis ETRX357USB-LRS (Note! This first have to be flashed with other EmberZNet firmware)
  • Telegesis ETRX357USB-LRS+8M (Note! This first have to be flashed with other EmberZNet firmware)

https://community.home-assistant.io/t/eu-usb-sticks-for-the-new-zigbee-component/16718/10

Quote bellow originally posted by Frank Aug '17 in linked thread:

I can confirm that the ETRX357USB-LRS+8M can be flashed with an appropriate EZSP firmware without the use of any programming hardware.

So, without any guarantees (you might -but probably won’t- brick your device), here’s how:

Download the firmware. I used some random blob someone put on github that at least seemed to have the right name. What could possibly go wrong, right? 😄 https://github.com/yqyunjie/Zigbee-Project/blob/master/firmware/EmberZNet/EM35x-EZSP/build/em35x-ezsp-images/EM357/em357-ncp-uart-xon-xoff-use-with-serial-uart-bl-500.ebl?raw=true 282

(Install USB-to-serial drivers for the device. Linux will load the driver automatically when you plug in; not sure about other OSes.)

Install a serial port communication app that supports X-MODEM. (debian/ubuntu: sudo apt-get install minicom) Run it and configure it (sudo minicom -s) to use the fake serial port (/dev/ttyUSB0) at 19200 baud 8N1. Disable both hardware (RTS/CTS) and software (XON/XOFF) flow control.

Type ‘AT’ , you should get OK in response. Now type ‘AT+BLOAD’ . The device will reboot into the bootloader.

Change the baud rate in the serial port communication app setting to 115200 baud. (exit minicom using CTRL-A Q, run ‘sudo minicom -s’ again)

Pressing enter in the terminal should now show you a three-option boot loader menu. Choose option 1.

‘C’ characters will start showing. Don’t wait for this to finish, but start an X-MODEM upload of the firmware you downloaded earlier (use CTRL-A S in minicom). You have 60 seconds to start the upload.

After the upload finished, you should return to the menu. Now select option 2, to reboot into the new firmware. You’re done.

Enjoy!

@Hedda
Copy link
Contributor Author

Hedda commented Apr 22, 2020 •
edited
Loading

Think these might finally a couple of easier how-to guide for building EmberZNet for EFR32 and EM35x:

It does however do sound as if you need to register EFR32MG development kit with a serial number to get a logon, but once you have you are allowed to reuse it within the same organization or company.

Could be worth it for you to try to contact Silicon Labs Salesforce and explain that you would like a logon for an open-source organization/project without owning a dev kit, as an "organization" is a loose term, zigpy is technically an organization.

After all, projects like zigpy does help Silicon Labs sell chips. Guess that the answer might differ depending on how Silabs feelings are towards helping open-source projects which are not-for-profit.

@Adminiuga Another suggestion could be for you & @dmulcahey to ask if maybe Home Assistant core team would be willing to sponsor you ZHA devs with an official EFR32MG development kit.

Again it sounds as if you would only need one login for "team zigpy" and then several people on "team zigpy" would be allowed to use that same login with Simplicity Studio and other Silicon Labs software.

@Hedda
Copy link
Contributor Author

Hedda commented Apr 22, 2020

I think it is might be up to the vendor to include boot loader into image

Guess they might sometimes recommend disabling bootloader to prevent any firmware upgrades.

Here are otherwise some Silicon Labs recommendations for building firmware images for production:

kpszupin (Employee) originally posted 12/29/2017 but it has probably updated since then:

Some recommendations when building firmware images for production:

  1. Build a single manufacturing hex file for each device with bootloader, application image, flash protection patches, and any tokens.
  2. Enable write protection on the bootloader flash area (0-0x4000) by clearing the lower eight bits of the first byte in the lock bits page (0x0fe04000). Each bit represents 2KB of flash, so 8 bits represents 16KB total for the standard bootloader: --patch 0x0fe04000:0x00
  3. Disable the debug port on the EFR32 by clearing the four LSBs of the DLW register in the lock bits page. This prevents debugger access from reading/writing to the device. Debug access can only be gained again by erasing the device. --patch 0x0fe041FC:0xF0
  4. Program the AES/encryption key (if applicable) and the ECDSA public key tokens. These can be in the same token file as any other tokens you are programming. --tokenfile keyfile.txt
  5. Lock the lock bits page on EFR32 by clearing bit 1 of the ULW register in the lock bits page. This is a requirement to secure the AES and ECDSA tokens which reside in the lock bits page. --patch 0x0fe041F8:0xFD

It is possible to combine these patches into a single hex file for manufacturing with a Simplicity Commander command, which can be put into a batch file or post-build script. For example:

commander.exe convert gecko-bootloader.s37 app_image.s37 --patch 0x0fe04000:0x00 --patch 0x0fe041FC:0xF0 --patch 0x0fe041F8:0xFD —tokengroup znet —tokenfile mfg_tokens.txt -o manufacturing_image.hex -d EFR32MG1P233F256GM48

@Hedda
Copy link
Contributor Author

Hedda commented Apr 22, 2020 •
edited
Loading

Another tip is this "Zigbee Boot Camp Course" wiki for Silicon Labs @MarkDing has put on GitHub :

He has also put together this PDF with Silicon Labs ZigBee Onboarding Roadmap for beginners:

Also see

@Hedda
Copy link
Contributor Author

Hedda commented Apr 22, 2020 •
edited
Loading

Slightly off-topic question; but once have EFR32 based adapter with latest EmberZNet Zigbee stack:

Would it be possible for zigpy to form a Zigbee 3.0 network using latest EmberZNet ZigBee Stack?

Currently, as I understand, zigpy & bellows form a Zigbee HA (Zigbee Home Automation) network?

@Hedda
Copy link
Contributor Author

Hedda commented Apr 28, 2020

FYI @SillyDay don't own a E180-ZG120B but has tried to build a firmware for it as per discussion here:

Koenkk/zigbee-herdsman#168

SillyDay GitHub repo for EFR32 firmware:

https://github.com/SillyDay/EFR32

@MattWestb
Copy link
Contributor

Thanks !!!! :-))

@Adminiuga
Copy link
Collaborator

2020-08-16 11:25:31 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 11 (getMfgToken) received: b'10ffffffffffffffffffffffffffffffff'

You should set the MFG_BOARD_NAME znet token to IKEA Billy EZSP :)

@MattWestb
Copy link
Contributor

How do i do that in HA ?

@MattWestb
Copy link
Contributor

MattWestb commented Aug 16, 2020 •
edited
Loading

zha_custom.execute
command: ezsp_set_keys MFG_BOARD_NAME
command_data: IKEA Billy EZSP

????

@Adminiuga
Copy link
Collaborator

nah, that won't work unless you add code for it. Use the silabs commander to set the manufacturer token

@MattWestb
Copy link
Contributor

The API for EZSP is as you knowing : ezspSetMfgToken(tokenId, tokenDataLength, tokenData)
and offset in userdata is for TOKEN_MFG_CUSTOM_EUI_64 0x0002, TOKEN_MFG_STRING 0x001A and for TOKEN_MFG_BOARD_NAME 0x002A from the manuale :-))
It can be written with commander but i don't like to trying that (have done 2 hard brick with J-Link).
The good thing is that the userdata can being erased (and written) with SWD and then the tokens can being programed but not changed (need flash page erase) from API.
I think i trying putting some info at 0x001A and 0x002A in one bin file and flashing it with SWD to the user data (Ikea have normal model name in user data if more devices sharing the app like the old and new remote control after the token airia at 0x0200).

@Adminiuga
Copy link
Collaborator

(have done 2 hard brick with J-Link)

Hrm, unless you lock out the debug interface, I thought it was possible to unbrick the device, essentially by erasing it. So the "recover bricked device" did not work?

@MattWestb
Copy link
Contributor

Blackmagic probe(BMP) working great but with J-Link and simplicity studio i was erase internal flash, reading with flash map and writing bootloaders and then it was going crazy. Was reading the flash with BMP and the bootloader was written offset and not at the beginning of the flash but working. Booting in bootloader and flashing app = RIP (to much offset for the app and its written in the peripheral aria / lock bits).
Nop and other scripts from sirlabs is not working then the SWD is not responding normal (can being xtstall config and the SWD is working in wrong frequency).
If having one dev kit its little better then the integrated J-Link have special recovery functions for the sirlabs CPU/SOC that not working on the "external" ones.
Some HW config and debug lock combination with command port lock is hard brick.

@MattWestb
Copy link
Contributor

MattWestb commented Aug 17, 2020 •
edited
Loading

I'm missing the token "TOKEN_MFG_MANUF_ID" that i was setting it but not getting it in the log (0x117C) ;-)))

Is this better ??

2020-08-17 09:07:36 DEBUG (MainThread) [bellows.zigbee.application] getting MFG_STRING token
2020-08-17 09:07:36 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getMfgToken: (<EzspMfgTokenId.MFG_STRING: 1>,)
2020-08-17 09:07:36 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 11 (getMfgToken) received: b'10494b4541206f662053776564656effff'
2020-08-17 09:07:36 DEBUG (MainThread) [bellows.zigbee.application] getting MFG_BOARD_NAME token
2020-08-17 09:07:36 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getMfgToken: (<EzspMfgTokenId.MFG_BOARD_NAME: 2>,)
2020-08-17 09:07:36 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 11 (getMfgToken) received: b'1042696c6c7920455a5350206279204d57'
2020-08-17 09:07:36 DEBUG (MainThread) [bellows.ezsp.protocol] Send command getValue: (<EzspValueId.VALUE_VERSION_INFO: 17>,)
2020-08-17 09:07:37 DEBUG (MainThread) [bellows.ezsp.protocol] Application frame 170 (getValue) received: b'0007b40006060400aa'
2020-08-17 09:07:37 INFO (MainThread) [bellows.zigbee.application] EZSP Radio manufacturer: IKEA of Sweden
2020-08-17 09:07:37 INFO (MainThread) [bellows.zigbee.application] EZSP Radio board name: Billy EZSP by MW
2020-08-17 09:07:37 INFO (MainThread) [bellows.zigbee.application] EmberZNet version: 6.6.4.0 build 180

Patched orginal user data file from the E27 WW bulb in "string form".

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿIKEA of SwedenÿÿBilly EZSP by MW�|ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿIKEA of SwedenÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿTRADFRI bulb E27 WW 806lmÿÿÿÿÿÿÿ20181203ÿÿÿÿÿÿÿÿÇ � LED1836G9ÿÿÿÿÿÿÿ  �   ÿÿ½   �X�  d2 è� � F� x�ŠŠŠÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

The data after 0x0200 is the orginal from Ikea and is outside the "token airia" that the EZSP is reading and is used for "custumizing" the "standard" firmware then using the same OTA file for different device types (the new and old remote is doig that and most bulbs).
It was very easy putting the token in the bin file and flashing it to 0x0fe00000.
One thing is that the offset is different in EFR32 1 gen and 2 gen then putting the token in the user data fie.

@Hedda Hedda mentioned this issue Aug 19, 2020
Closed
@MattWestb
Copy link
Contributor

I was yesterday making one deep diving and finding 2 interesting documents first:
AN1233: Zigbee Security.
Then one "clean paring" (the NCP don't have any toking with TC-key for the device) is working the network key handling is working also the rolling of the network key.
Also the TC-link key as “3.3.1 Preconfigured Link Keys” is working if not token is stored for the device in the NCP.
The problem is is showing up in the “3.3.3 Requesting a New Link Key after Joining” then the joining device is requesting one new TC-Link key.
Then paring one device with one toking in the NCP for the device i only getting this is the HA pairing log:

[0x0000:zdo] ZDO request ZDOCmd.Mgmt_Permit_Joining_req: [60, <Bool.false: 0>]
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
[0xfe10] Requesting 'Node Descriptor'
Tries remaining: 2
[0xfe10] Extending timeout for 0x8b request
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23

With time outs of NCP commands to the device and leaving joining until complete time out of the pairing after ca 2 minutes.
With out device token in the in the NCP i getting:

[0x0000:zdo] ZDO request ZDOCmd.Mgmt_Permit_Joining_req: [60, <Bool.false: 0>]
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
[0xfe10] Requesting 'Node Descriptor'
Tries remaining: 2
[0xfe10] Extending timeout for 0x8b request
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
Device 0xfe10 (14:b4:57:ff:fe:53:98:23) joined the network
Skip initialization for existing device 14:b4:57:ff:fe:53:98:23
[0xfe10:zdo] ZDO request ZDOCmd.Device_annce: [0xfe10, 14:b4:57:ff:fe:53:98:23, 142]
[0xfe10:zdo] ZDO request ZDOCmd.Node_Desc_req: [0x0000]
[0xfe10:zdo] Unsupported ZDO request:ZDOCmd.Node_Desc_req
[0xfe10:zdo] ZDO request ZDOCmd.Mgmt_Permit_Joining_req: [180, <Bool.true: 1>]
[0xfe10] Node Descriptor: <Optional byte1=1 byte2=64 mac_capability_flags=142 manufacturer_code=4476 maximum_buffer_size=82 maximum_incoming_transfer_size=82 server_mask=11264 maximum_outgoing_transfer_size=82 descriptor_capability_field=0>
[0xfe10] Discovering endpoints
Tries remaining: 3
[0xfe10] Discovered endpoints: [1, 242]
[0xfe10:1] Discovering endpoint information

And in the end the paring is OK.
One thing is way is the NCP trying to talking to the device and only 3 retries before the TC-Link key is hand shaking is finished (perhaps the order is OK but need longer delay or more retries for the first "Node Description" or waiting for one "Device_annce" (perhaps only for ZB3 devices) before starting the pairing procedure.

The interesting things its the ZDO commands that not being sent / received in the first case.
Probably is the TC-Link key not synced between the the TC and / or the device have not getting the network key so cant sending / receiving messages from the NCP.
In ZB3 the joining device is using the “well known key” for getting the network key from the TC.
Then its requesting one TC-Link key and getting it then leaving the network and connecting with the new TC Lin key.
Then it can that the NCP is not correctly configured for ZB3 and not sending one new TC-Link key or sending the “well known key” then the ZB3 device is leaving the network because to low security.
Or the NCP is sending one new TC-Link key and cant storing it and using the old one then “talking” with the device that have getting the new one = key mismatch.
The “3.5.2 Trust Center Rejoin” is describing the re-joining with TC-Link key but that is only working if the “Network Security Frame Counter” is right that is not then the device is resettled and the counter is zero in the device and the NCP have not resting it in there side.
Or the NCP don't processing the "update device" from the parent router (3.5.2 Trust Center Rejoin)

The thing is how do you doing the updating of the token for the devices in the NCP.
I was doing my second finding in the UG100: EZSP Reference Guide.
“Name: addOrUpdateKeyTableEntry ID: 0x66”
“Description: This function updates an existing entry in the key table or adds a new one. It first searches the table for an existing entry that matches the passed EUI64 address. If no entry is found, it searches for the first free entry. If successful, it updates the key data and resets the associated incoming frame counter. If it fails to find an existing entry and no free one exists, it returns a failure”
Have you possibility to trying it out that the token can being updated with this function ?

2 more important things is the “3.7.2 Trust Center and Source Routing” and “3.7.3 Trust Center Address Cache” but its not so likely the core problem then paring is working then the NCP don’t have any token stored for the device.

I have reading the most of the “AN1233” 2 times and sleeping between without the ZGP things and understand the most but i think you is understanding it better.
I think it good that you taking one deep dive in it 2 and perhaps you is finding the source of the problem in the multiversion bellows.

Regards MW

@Adminiuga
Copy link
Collaborator

I wonder if EMBER_TRUST_CENTER_USES_HASHED_LINK_KEY when forming the network would help with it?

@MattWestb
Copy link
Contributor

Is the trustcenter token updating problem present in all protokoll versions (4 to 8) or is it only in the higher (>=v6) its failing ?
How is it working in the production version of bellows, also problems in the higher protocol versions
?
Is it possible making "super debug" for seeing how the NCP is doing “Name: addOrUpdateKeyTableEntry ID: 0x66” funktion then updating the tokens or have problems and failing of some reasons ?
I think its being possible with the devkit doing deep debug in the NCP with simplicity studio also of one node the its "talking" with the mesh.

Looks like EMBER_TRUST_CENTER_USES_HASHED_LINK_KEY only have good things and saving memory / storage wot i can see.
Its looks like Stefan H is using it in Tasmota EMBER_TRUST_CENTER_USES_HASHED_LINK_KEY = 0x0084,Its looks its defined in xdrv_23_zigbee_0_constants.ino.
So way not testing it !!

I doing one ZB3 rejoining / reparing test with tasmota and seeing if its working (have only testing pairing but not real repairing).

@MattWestb
Copy link
Contributor

I have testing rejoining and force delete / live of device in tasmota and it was not nice :-((
Then deleting one devices it was deleted in tasmotas database but not the device token in the NCP and no leav command was sent to the device.
The result is if repowering the device its silent doing one secure rejoining in the network = false then the device token sud being deleted in the system.
If resetting one device is doing one secure rejoining without the device have one old link key, network key or installation key in the device token store (Its deleted then being reseted) matching one in the NCP token store. Is suld being needed doing one normal paring for letting it in the network.
I have informing Stefan H for fixing the "undocumented future".
Thats its little to low for a real ZB3 network for my !!
And its not helping the bellows problems.

One more detailed description of device commissioning: Base Device Behavior Specification
Version 1.0
The interesting is under "10.2 Node operations" and have more details then the Silabs dokuments.
Its describing commitment process very well.
The "10.3.2 Adding a new node into the network" have good details how its working step by step.
In "Figure 15 – Trust Center link key exchange procedure" and the difference then rejoining one device in "10.3.3 Behavior when a known node joins"
Thats leaving the problem is in the 2 different points (4 and 5) in the 10.3.3. and this is the most interesting point:

5. If bdbJoinUsesInstallCodeKey is equal to FALSE, the Trust Center SHALL
first find the entry in apsDeviceKeyPairSet that corresponds to the joining
node and then overwrite the LinkKey entry with the default global Trust Center
link key and set the KeyAttributes field to PROVISIONAL_KEY. The Trust
Center MAY then set OutgoingFrameCounter to 0 and SHALL set
IncomingFrameCounter to 0.

That is not working in bellows and it must being one policy in EZSP or setting that is doing that is not working in bellows.
Must trying finding wat silabs have for setting that is hanging the behaviour described those funktion.
And you is using the "default global Trust Center link key (“ZigBeeAlliance09”)." so that should not being one problem.

@s-hadinger
Copy link

@MattWestb We're not using EMBER_TRUST_CENTER_USES_HASHED_LINK_KEY in Tasmota.

Our goal is to make Home Automation easy to use for all users, so we're not using advanced security features that would make provisioning difficult. We use only one Link Key (well known key) and only one network key for the entire network.

@Hedda Hedda mentioned this issue Aug 25, 2020
Closed
@MattWestb
Copy link
Contributor

Our bellowed Bellows have in the night moved from "multiversion" to aktive public dev branch and and is working OK then pairing Zigbee 3. HA1.2 and LL devices. Have not so much time for testing but looks very promising.
Here is my screen shot with Billy running on EZSP 6.7.6.0.
ICC-A-1HA6640D
I think it's one "Wolf in sheep's clothing" (deCONZ killer ?).

@Adminiuga
Copy link
Collaborator

I have a few Zigbee-WiFi Bridges available. Latest one is to get price lowest to compete with Sonoff and to use low cost enclosure...

@grobasoz do you have links to your products mentioned in #243 (comment) ? Even if it is a work in progress? I'd rather direct new users toward these in case they are considering HUSBZB-1

@MattWestb
Copy link
Contributor

Thanks for the commit to master branch !!
Bellows 0.20.0 GA is up and running in HA :-)))))

@Hedda
Copy link
Contributor Author

Hedda commented Sep 7, 2020 •
edited
Loading

Our bellowed Bellows have in the night moved from "multiversion" to aktive public dev branch and and is working OK then pairing Zigbee 3. HA1.2 and LL devices. Have not so much time for testing but looks very promising.

It would now be very nice to have a GUI display indication representation like a tag or similar to show in Home Assistant ZHA UI what specific Zigbee protocol profiles and exact Zigbee profile version that each specific device is using.

Perhaps a feature to show another column which state protocol profiles and profile version that each specific device is using?

Example:
ZB3.0
ZHA 1.2
ZLL 1.0
ZLL 2.0

Or is there perhaps some better location or way to get Zigbee protocol profiles and exact Zigbee profile versions into HA ZHA?

PS: Probably not supported by any firmware just yet but Zigbee 3.1 (ZB3.1) devices should also become available very soon now:

https://www.asmag.com/showpost/28387.aspx?name=news

(Zigbee 3.1 will not change any hardware but only focus on interoperability and compatibility between different manufacturers).

@grobasoz
Copy link

grobasoz commented Sep 7, 2020

@Adminiuga - I should have samples available this week so can take some pictures, do some testing and put some information on my github repo. I should be able to get a price estimate as well... Full production should be completed within another 4-6 weeks' time though.

@MattWestb
Copy link
Contributor

@Hedda The R23 is coming very soon an i think the EZSP 6.8.X.X is the last R22 release.
With the R23 all EM35x devices and any devices with 256 kB flash or less [All wireless MCU families] is not longer supported so doing one last "golden" firmware request from our firmware makes that supports the advance functions in R22 is expected then R23 is released.
On the NCP side its the internal flash storage for token storage that is being the braking thing but not for the router and end device.
Then bellows is going from ZHA dev to main release the devs need some time for letting it settle down and fixing problems that is not found yet and then its one long wishing list of futures that can being implanted.

I like more advanced setting and functions that is possible with the EZSP like:
Channel scan and network automatic move if finding one better channel. Automatic network key rolling (with max time age). PAN-ID conflict detect and automatic changing. All with manual override in GUI. GUI for administering install codes. Setting default security in the NCP and make it possible joining with lower / higher settings for next pairing (old LL devices / Install code ZB3 devices then normal is ZB3 with enforced changed Link keys).

The large thing is what is possible implantation in the other zigpy radio platforms without breaking things.
I'm very sure its coming more good things in one or two HA releases that making our system being better and more flexible.

This was referenced Sep 17, 2020
Open
Open
@Hedda
Copy link
Contributor Author

Hedda commented Sep 24, 2020 •
edited
Loading

@grobasoz Any picture of your sample hardware? What storefront will you use to sell your hardware? Shipping only from Australia?

@grobasoz
Copy link

@Hedda - you've been busy :) I've been testing 6.8.0.1 (the minor release "2" is a typo) but am not that impressed with the Multi-Pan option.

No "official" pictures yet as production is still underway - I can post some of the prototypes if you're interested... Some re-design was necessary to make assembly into the enclosures easier...

As I'm predominantly a development engineer with production capability, I tend to stay clear of sales and mainly supply retailers with products. They then on-sell globally where necessary.
With the very low cost price of these products (to compete with the likes of Sonoff), that may need to change that paradigm :(

For EU, USA customers, it doesn't make sense to ship to and from Australia, it may be best to ship products direct from the factory in China to a "local distributor" when they are ready.

@Hedda
Copy link
Contributor Author

Hedda commented Sep 24, 2020 •
edited
Loading

For EU, USA customers, it doesn't make sense to ship to and from Australia, it may be best to ship products direct from the factory in China to a "local distributor" when they are ready.

Sounds great! Yes I think products will probably sell much better globally if local distributors can stock and ship from EU + USA

@Hedda Hedda mentioned this issue Sep 25, 2020
Closed
@Hedda
Copy link
Contributor Author

Hedda commented Sep 28, 2020 •
edited
Loading

FYI, Tasmota project has added a signed version of EmberZNet 6.8.0.1 NCP application firmware for EFR32 in Sonoff ZBBridge:

https://github.com/arendst/Tasmota/tree/development/tools/fw_SonoffZigbeeBridge_ezsp

It is supported in latest Tasmota development Tasmota-zbbridge version, but not yet proven stable with Zigbee2Tasmota (Z2T).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests