-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Screen mirror - app does not show up in bmw & question about waze #26
Comments
Good morning! On a rooted phone with something like Xposed anything can be done with enough work! I have an old branch of code where I managed to get Waze to accept the control connection and output buffer from a hosting simulator. Waze worked well because it drew most of its UI, most of Android Auto is drawn natively by Android Auto itself and I didn't want to redraw all that for an experiment. I would like to extract it out to an Addon at some point, but it was very flaky and the hooked functions were obfuscated and subject to changing with any version upgrade. I typically would test in the car. I did have a headless app that I could watch the logs and confirm initial connections, and recently I started building a fully graphical headunit simulator but it is still pretty basic. |
Hey! Thank you for answering, very nice to see that you are still active on this project! So, I build the screen mirror app myself and got it working, as you say, it was probably the firewall that I accidentally denied it connection to, because it does have a firewall. About the icons, why are some unlabeled? Is this some limitation? Also with the detailed car info, why does the navigation work the way it does? Once I got screen mirror working, I got an app to change my phones resolution to better fit on the car display and found out that if I screen mirror a certain app and then use something like this: https://play.google.com/store/apps/details?id=com.ss.screenoffandplay&hl=en to turn off the display while letting the app run, I can keep the phone "off" while screen mirroring. I have denied all these extra apps network permissions and I have an adblocker so I don't care about the ads really but if this works well I might build a addon that does this for AAIdrive. When using the android auto waze, was the performance any better or is it still limited by the speed of the usb/bt? How come that android auto on other cars can run with more fps and this is limited, is it because it is pictures etc? Because if its not any better I don't see a reason to not just use a screen mirror. My idea is to build a carapp for controlling waze, is the "settings menu" when moving the knob to the right editable? Because my idea would be to add "Report accident" or "Still there" etc. buttons in a menu and then make a "clicker" to execute the user request. However before I can do alot I need to figure out why my AAIdrive keeps crashing, I have disabled battery optimization, what else could cause problems? I have a ram monitor that I try to keep open and it runs around 68-78% and crashes even then. I have IDrive4 from my understanding and when using the scroll wheel it says "not available" or something, however once when I used crDroid (another OS) I got information about the playlist, however my bt sound was bugged so I changed to voltage. If I want to check on the official apps, is it just to download like Twitter and watch the bt connections? Thanks again for your help! |
The car app resources are cryptographically signed by BMW, including the entry button icons and their labels. That circle icon is provided by the Samsung SmartThings app, and while I can't change the icon or text, I can hide them, which I do to try to reduce confusion. I'd be interested to see what you find, perhaps with information about how the screen off feature works. It would seem easy enough to integrate it into the existing addon, if possible. How does touching the screen work while it's off? The Waze hack wasn't any faster, due to limitations of the protocol. The Android Auto protocol sends a video stream to the car over USB or WiFi Direct. BMW apps talk over USB or Bluetooth, and can only send BMP/JPG/PNG pictures to widgets in the car. It is much faster over USB than Bluetooth, but still dependent on the size of the image (including complexity increasing the data size of the JPG). The advantages I was hoping to achieve with native AA app hosting were to gain control to send input events to the app and to run the app without taking over the phone screen. And now, running Android Auto on the phone for Screen Mirroring is less convenient, since Google took the phone interface away again. I suppose it might be possible to recreate the Dev Headunit as an addon, perhaps, tho not very convenient. The side bar on the right side of the screen is not editable from a car app. Are you using the new beta from the Play Store? Someone recently reported an Issue with it seemingly crashing after a few minutes, and I would appreciate if you could find logcat logs showing the problem, since it works for me. The car natively talks AVRCP 1.4 and can view the current queue over Bluetooth if your phone allows it, and can also browse some apps in the Bluetooth screen. However, the hud does not show any playlist from any car app before IDrive5. Your car doesn't talk the apps protocol over Bluetooth, you'd instead want to tcpdump the localhost connections on your phone to monitor the traffic from the apps to the proxy into the car. If you do find s car with Bluetooth Apps, you could hcisnoop the traffic, but my Wireshark plugin is buggy and abandoned since tcpdumping was better. |
In the car info navigation it opens a list with opens which i can click, if I click it I get the info with a button at the top and to get back to the list I have to select which can info I want to see, however clicking on any of the other text in the detailed info, it opens a page without any data. Can attach some pictures soon if you want. That is both sad and weird that they require the buttons to be signed, but I understand now why it looks like it does. I tried the screen off app and waze was a little buggy? The car position did not update but the speedometer did, will need to look into that more, you cant use the touch screen yourself because it uses that to detect when you want to stop using the screen off thing. If the right side bar is not editable, how come the audioplayer has a "Random" button for its sidebar, or is that some default BMW stuff on the mediaplayer? I did manage to get some logs: But I can't really figure out exactly what is doing the crashing. Before I used the app from the play store, non beta, but recently I have built the app myself to try sentry but I do not get any crash reports there. The car does not support bluetooth apps, so I will try to debug my tcp connection sometime. I am guessing everything is unencrypted with the usb connection as well? Edit: here I think I have found a perfect, got this right after the crash |
I know this is more about AAIdrive, should I write there? How exactly can I add a label on for example the music page? Or is the layout predefined and I cant change it? How does this work? Trying to understand the code, I guess the "fits" function has something to do with this? |
So, while the third-party apps (such as Samsung and Spotify) have flexible menus, and the app can add extra handling and decide what screen to open after clicking a button or list item, the Car Info app uses a built-in BMW app (News), and these buttons are hardcoded to open specific screens and can't be changed. What Random button do you see? Do you mean the left toolbar? The toolbar is just a list of buttons, and each button has an image from an icon pack and a string label, so AAIdrive can reuse the shuffle icon and set the label according to the state. I merged the branch so the new API34 code is the main branch, and so now the Github Latest Build has it now. Feel free to run this version for a while and I'll watch for any emails to arrive in my Sentry account. Since you've set up the project to build, you can look at what the car app resources look like: In your project directory, look in |
Hey, sorry for a late response, have been up to a lot lately. What I meant by "settings/options menu" and the random button was this: About the playlist thing, I have tried all AVRCP versions in developer settings yet this is what I get: About the crashes, sadly when it crashes it crashes fully, all apps gone. I did a tcpdump and got this: If you want I can use your build but I tried sentry but did not get any crash reports at all. Right now I am playing around with the menu options and stuff, also I can only see the song title and band, isn't the album supposed to be shown as well? And my last question, is labels a item that can be selectable? Again, thank you for helping me! I am glad I got this working, even if it's sometimes crashes, I would love to look into how, if even possible I can improve it! |
Ah, I haven't used IDrive4 in a while and perhaps have forgotten some of the version differences, apologies. When you say "crashing", does it take the AAIdrive phone interface down too? If there aren't any crashes in the logs or in Sentry, then it may not be AAIdrive crashing but perhaps MyBMW or BMW Connected. Since you say everything disappears, including Screen Mirror, then it would appear to be the MyBMW connection to the car that is disconnecting. IDrive4's multimedia sidebar for global metadata only has two text fields, which AAIdrive fills with artist and title. IDrive5 has a separate system for showing media information which includes cover art and track time or album information. |
I see, will try the file for Wireshark as soon as possible and will enable the debug logs for the MyBMW app as you state in the analysis documentation, to try to figure out if that maybe gives more information. I am not sure if the AAIdrive app crashes, but I don't think so, I think it's rather the MyBMW that crashes, but will look into it! When you say "sidebar" what exactly do you mean? Now when I connect my phone with only bluetooth to the car I get the track time, track name, artist and album name. The track number is always 1/1. However if I use an iPhone the same information is there but the tracks position in its album is also there. I will upload a picture of this later or find some online later, also realized I can try to find the screen I have seen once before online. Will look into it and update here. Also, I have got a accessibility service up to interact with waze to automatically report anything that happens, will now try to figure out to make a iDrive screen for interacting with it! |
As I remember, IDrive4 has a slide-out sidebar on the right that you can toggle on and off, and shows selected information no matter where you are in the system. One of the options to show here is Multimedia Info, and car apps like AAIDrive can set information here. That is what I refer to as a side bar. |
I see, my IDrive is not wide screen, its smaller, only a single page at a time, like this: Something alike what you already have done was my next idea to do but it seams you have already thought of everything :D I also get this in my multimeda view; however when clicking "Spotify" or "Screen Mirroring" here I get to the "Detailed car info" page instead of the correct page. Any idea why this could be happening? Will try to look into the tcpdump, but here they are; these also include a crash at the end. I havn't managed to get the lua plugin working yet as I used wireshark 4, which version would you recommend? My "Audioplayer" looks like this; Using the accessibility service I saw the same thing you saw the the order of the items but will look into your code as well! |
You don't want the Lua plugin, that is just to add support for decoding the btsnoop packets into Etch packets. You only need the ewh file to decode the Etch symbol names. Ah, I see what you mean now, this is the compact Audioplayer interface. The screen technically has both sets of widgets, but conditionally hides them based on whether the screen is in widescreen mode or not. I don't know of any workaround for this either, I'm afraid. |
In the Edit > Preferences window, navigate to Protocols > Etch, and set the Apache Etch symbol folder to the location where you've saved the ewh file. Then you'll start getting symbols in the protocol field of the main interface: That's a really cool visualization! I'm not sure what you mean about "when making lists". The Audioplayer is the official music app from BMW Connected Classic, and a very similar layout is used for the 3rd party music apps in BMW Connected (Spotify, iHeartRadio, Pandora)
|
Thank you, got my Wireshark working, will try to see if I can get any info out of it! I am starting to understand the system now, but I am also starting to have more and more questions, if you don't mind. When it comes to the Car app, each Car app can only have one ui_desc each? I can't use a view from both SmartThings and Media Player in the same App? Can two views use the same state as it is with the view files today as long as they are not opened from each other? Or is it strictly once per app? Weird question maybe but why didn't you use the ids in the ui_desc files, why make the fits function? Wouln't it be easier to say use state with id of x? Also have you compiled a list of what states is available and their components etc? If not I would like to do this to get an overview over what I have available. Also can you get the current property value of an item? If so that could be used on certain states to read if some elements, in the case of the mediaplayer to check if the album image is off to the side or not. When it comes to the models, which ones can have a "value" that can be shown in the UI? For example, Where do you hide the app name and icon in the connecteddrive list? When i mean the list I was wondering if you needed image components to be able to show data in them, but based on this I guess I don't need that?
I will try to look into the instrumentCluster but that is maybe not what I am looking for. Because it does not do anything in my instrumentCluster. What you say about the positioning is very interesting, is it based on the order of the items in the components? Do you know if we can by properties set LayoutBag conditions? Edit: forgot to add; can you change color of text? |
These questions are super fun, I'm really enjoying them :) As far as I know, each RHMI app can only have a single ui_description, yes. The RPC calls use an app handle ID and a numeric model/component ID from this description, which would make it impossible to have multiple ui_description files loaded for the same app handle. When I first started the app, I wasn't sure how different the ui_descriptions would be for different versions of BMW Connected, and so I didn't want to hardcode any IDs. Users provide their own external APKs for the build process, and I didn't want to require specific exact files. The multimedia files have been relatively stable (because the old Spotify/Pandora/iHeartRadio apps needed them to be stable), but the BMW One internal description changes a bit across different versions, and I even had to dynamically find the imageId for one of the icons. There isn't any way to retrieve the model value or property from the car, but the car only knows what is specified in the ui description and any RPC calls. This would not provide access to the LayoutBag's current value, alas. I don't know any way to set any LayoutBag properties, and I just assume the normal Technically all of the models are meant to be shown in the car and are meant to be updated from the app. textIdModel and imageIdModels are integers that point to resources in the texts.zip and images.zip files, which limits their use by AAIdrive. For example, the Music PlaybackState changes the imageIds shown in the toolbar, but further alterations are difficult. raDataModel, raImageModel, and raListModel are more free-form. raListModel stores tables of information, and ByteArrays of images can be put in a cell and will be displayed, but the car limits the table row height to 2 text rows and so any images are likewise limited to ~90px. This is why the custom map's preview map is so small. I do see code in the old Connected Classic app that means that the legacy Audioplayer should support setting the InstrumentCluster playlist. Since I don't have an IDrive4 car to confirm, you should try installing BMW Connected Classic and seeing if the official Audioplayer app shows up in your InstrumentCluster. I believe my code is based on what I tcpdump'd Audioplayer doing, but maybe I missed something. The Richtext models, used by Notifications to view the full notification supports a few formatting things ( Spotify uses a newer multimedia ui_description that includes To reiterate, I am really enjoying these questions :) Feel free to reach out directly! |
Screen Mirroring log 2d49d1a1ed48.txt
Hey!
First of I want to thank you for this awesome project, I LOVE it, this is just what I was looking for. I have some questions and stuff about this project and if you are free to answer some questions I would love to write a bigger question post to get some more info about AAIdrive, because if I find some time I might work on some features and stuff.
Anyways, I am wondering how I can get screen mirroring to work, I have the addon and when I grant screen cast permission it says "Open the app in the car", but no app for screen cast under BMW Apps appear, I only have the ones that was added initially. I tried starting the screen mirroring first and then start the connection, that didnt help. I have added logs for Screen mirror, it says socket failed?
About waze, I read somewhere you said that only certain package names can access it and send commands etc, with a rooted phone, could this be bypassed? Is this anything you have looked into or not?
I am wondering if you have put up some kind of development environment or do you just always test the code directly in the car? And for the reverse engineering, do you have some base app that you have looked into to figure out what you can send/do? About this, this small info-display, is this anything that can be controlled by BMW apps?
Again, thank you for this awesome project. If you have some time, I would love to ask some more questions, I hope this was not too much :)
For context: I have Android 14, VoltageOS, BMW 218i 2016/2015? MV-130.016.001 (Tried to install new update but was not able to, will try another usb maybe later)
Best regards
Philip
Edit:
Added link to log as I forgot to upload first, after some investigation I think I it's the init function of CarApp.kt in screen_mirror
2: Just found; https://bimmergestalt.github.io/BMWConnectedAnalysis/ I have some reading to do, very nice to see that you have already written all of this! I totally missed these pages before, sorry..
The text was updated successfully, but these errors were encountered: