Embedded notecard missing from database.

[sonjamichelle]

My wife is trying to create an info Notecard for an AO. When she tries to embed another notecard and the end user tries to retrieve the embedded note card it opens with a forever "Loading" and an error Toast "Note card missing from database."

I tested this in Firestorm (Latest Release) and Singularity (Latest Release).

Happens on my grid running Halcyon 0.29.30 R6150

This also happens in InWorldz on Halcyon 0.29.30 R6148

[mdickson]

Can you give a bit more information of the steps followed? Was the embedded NC full perm? I'm unable to replicate the issue.

[sonjamichelle]

Created a notecard, full perm. created a second notecard full perm. Drag and dropped second notecard into first. Works for the creator, but not receivers of the notecard.

[mdickson]

Ok confirmed. This is definitely a regression of some sort. I can poke around a bit and see if I can run down whats happening

[sonjamichelle]

Okies, So we're not going crazy. ;-)

[appurist]

Well it's interesting. I haven't commented yet because I haven't been able to reproduce the problem yet. Since both @sonjamichelle and @mdickson haven't had any trouble reproducing it, I've persisted with additional tests.

I've tried passing the notecard while the other user was online, while they were offline and retrieving it later, and tried it with IW2, IW3 and the latest Firestorm viewers. I've tried creating the notecards in subtly different ways. In every test, I had no trouble opening either the outer notecard or the inner notecard.

Can (either of) you provide a bit more environmental and procedural info on your test that does reproduce it? Which OS platform is the viewer running on? Is there anything in the inner notecard other than some text? Did you use a lot of text? Was the embedded notecard at the end of the containing notecard? Did you include more than one embedded notecard or embed anything else with it?

Note, I've been doing my tests on my development test server so that I could debug it once I reproduced the issue, but that should be similar to the main grid's setup (I am configured to use Cloud Files for asset storage like the main grid). But my next step will be to try to reproduce it on the main grid since that is one of the environments reported above. But I thought I'd ask if there was any other small detail left out which might be significant (i.e. different than my simple tests).

[appurist]

Created a notecard, full perm. created a second notecard full perm. Drag and dropped second notecard into first. Works for the creator, but not receivers of the notecard.

So after the second notecard is dragged into the body of the first notecard, that (outer) notecard is then offered to a second user account... and that receiving user is only trying to open the outer notecard and then open the inner notecard (from the contents) see the contents of the inner notecard? Or trying to copy it out, or use it in the scripted AO?

Or more simply, what is that user trying to do with it when that error is generated?

[sonjamichelle]

Not using Cassandra, only whip, aperture, mysql and Halcyon.

All of the Above. We have tried (as a second user) to open the notecards from the first notecard, trying to copy into the inventory then opening it and finally trying to embed them in a scripted AO. No method has worked. Once the notecard is passed to the second user any notecard embedded inside the first becomes unreadable.

[appurist]

I just did a full-path test on the main grid, opening the notecard from the outer notecard, deleting a "Default" AO config file from a HUD and adding it in there, etc. I was thinking maybe the script was getting the error. But even that worked for me, both on my grid and on the main grid.

But since you're saying it starts to fail as soon as the recipient user receives it, that rules out all the script-end issues.

Are you able to reproduce this with a new notecard, or one that was created in the past? I am creating a new notecard for each of my tests.

[sonjamichelle]

Yes, new note cards.

[appurist]

Okay, then let's go over the individual (assumed) steps here in case there's a subtle difference. My process:

1. User A creates a new empty notecard. I do this by right-clicking on my Notecards folder in Inventory and then choosing New Notecard.
2. After step 1, the name is highlighted in Inventory. I type Test1 for the name and press Enter to accept it.
3. In the meantime, the new empty notecard window is still open and I type something into it. I press Save and close the window.
4. I repeat steps 1, 2 and 3 above for notecard Test2.
5. I close Test2 and then drag it from Inventory into the body of Test1.
6. Close Test1, then drag it to the profile of User B.
7. Log in as User B, who has received the notecard. Open notecard Test1 from Inventory.
8. I see the text content of Test1 along with the embedded link to Test2.
9. I click Test2 and it appears in my inventory. (I can also accept it if I have an offer dialog.) It shows the contents of Test2. I can also open it again from the Inventory list.

Do you see any differences from your process and step 9 is when you get the loading error and timeout?

I have checked for the "Note card missing from database." message and it doesn't appear in the server code (in fact "missing from" does not appear). So that must be the viewer generating a message give some error condition (failure to load the notecard). Perhaps there's an error at the server end. If you can reproduce this on your own grid, can you check the region console to see if an error is produced when you try to open it?

[sonjamichelle]

Same process. No Errors being thrown on the console. Only in the viewer, all viewers. FS, IW3, Singularity.

[appurist]

I'm at a bit of a loss at the moment then. I'd normally suggest trying a different region in case it was server region-specific, but you have the same problem on two different grids, and Mike has the same. Oh platform: you are running the viewer on Windows? Maybe @mdickson has some ideas here.

[sonjamichelle]

Yes, running the view on Windows 10.

[appurist]

Me too.

[kf6kjg]

Got a repo!!!

User A and User B are friends, no repo. If they ARE NOT friends, it repos: I see the notecard show up in User B's inventory, but the viewer pops the error.

[appurist]

I think we are (sort of) making good progress on this. We've found some error cases where the symptoms might match, so at first the idea was to try to determine which error condition might be happening. However in testing that, I've found that the viewer makes multiple requests (retries) if it doesn't get a response back quickly, so a region under some load (or perhaps where the viewer is also running on that machine or something of that nature such that the timing changes) may result in a race condition where the viewer has made multiple overlapping notecard copy requests. And when we send the update to the viewer, depending on the order of completion, the viewer may have the wrong (latest) ID for the subsequent notecard open request.

It would be interesting to know, in the case where it fails to open, if you just relog with the viewer does it succeed? And if not, does clearing the viewer cache and relogging succeed?

[sonjamichelle]

No, relog and cache clear does not resolve the issue. Still persists.

[appurist]

Okay, in this case consistency is good. If you reproduced that last test on InWorldz, or if you didn't can you give that a try, then that means you effectively have a broken inventory item.

Can you move that item into a known new subfolder location, such as Notecards / Broken and I'll try to inspect that item in your inventory with our tools here and see what the issue is that is preventing it from opening?

[sonjamichelle]

Sonja Galileo is my username on IW.
In Notecards/Borked
Parent Item: Tester NC in NC Asset ID: a9f99e05-62bd-45bb-9f38-d4e7e8120b0c
Embedded Item: Tester NC in NC 2 Asset ID: f1a63c2e-88ac-b428-5798-7a4fc21bb481

[appurist]

Sonja, is that the item in the receiving user's inventory? I see it in the "borked" folder. 😁 Which one can Sonja Galileo not open successfully?

[sonjamichelle]

The Tester NC in NC 2 can not be opened. And yes I am the receiver.

[appurist]

We're working through possibilities now with a process of elimination. Another question. In that notecard that has the embedded notecard that cannot be opened... if the sender user clears cache and relogs, can the sender open the embedded notecard? (i.e. sender opens tester NC in NC and then clicks on Tester NC in NC2)

[sonjamichelle]

The sender can Open the note card as normal. Only the Receiver has issues.

[appurist]

Quick update: We're still on it (on and off). We've learned that it's not the embedded notecard that is damaged, it's the embedding itself (a problem with the embedded link in the outer notecard). If you've reproduced the problem sending it to another avatar, I think you should actually be able to reproduce it with the one that was sent (i.e. if the original owner opens the sent notecard that has the embedded link) as long as that user clears the viewer (inventory) cache. Or possibly even if the sending user relogs. That is, the link won't open there either.

There seems to be a problem with the creation of the embedded link. I think the viewer is storing a temporary or otherwise bad asset ID in the notecard. It's working for the sender because an asset with that ID it is either in the viewer cache, or it was only in the viewer session's memory. Asset IDs are handed out by the server and a viewer should never embed an asset ID that it assigned (even a temporary placeholder ID) in the body of a notecard, or the server won't be able to provide it (by that ID) later. So progress but I still can't definitively say where the problem is (partly because I'm not familiar enough with how this works in the viewer).

[sonjamichelle]

Not sure if this is related, but my wife did a covenant for an estate. When I try to view the covenant I get the following: Estate covenant notecard is missing from database.

The convenant has a last modified date/time of Wed 31 Dec 1969 22:34

[appurist]

I can reproduce that no problem, 100%. However it's worth noting that I think it's just not being added correctly.

If User A creates the notecard under Firestorm and adds it to the Region/Estate form, they can continue to see it when reopening the form. User B on InWorldz 3 cannot.

But then the two users swap viewers. User A logs on on the same machine with InWorldz 3, but User B logs in with Firestorm. Now User A (the original creator) can no longer see the notecard in the Region/Estate form (it produces the error), while User B who is not the creator and could not see the notecard originally can see the covenant test logging in on the Firestorm viewer that created it (under a different account).

Clearing the viewer inventory cache and relogging both viewers did not change the results at all.

So it seems that the two viewers may have incompatible formats, or an unexpected restriction, or something like that for the covenant. For example, if the covenant text was stored as a link and only one viewer supported that.

I have a strong suspicion that your observation here may be critical in tracking down the embedded notecard case as well. These do sound like the same problem, even if the symptoms don't align 100%. (This case doesn't look like a viewer cache issue since clearing the cache did not change the results.)

[emperorstarfinder]

Just a quick update on this one as I just reproduced this on my beta grid using the Halcyon architecture.

The reason @sonjamichelle saw the Estate covenant notecard is missing from database. error is because the UUID given for the notecard in the estate db tables is incorrect and does not in fact match the UUID provided for the notecard in the inventoryitems db table. I also attempted to change (edit the notecard) and save and appears to get the same error. I wound up having to do a fresh setup for this to go away.

What appears to be occurring is when you drag the notecard from your inventory to the estate covenant window in the region manager a new UUID for the asset is being created and inserted in the estate db tables in the corresponding row. Thus when the estate covenant notecard is added into the estate db tables its given a UUID that in fact does not exist in the asset service itself therefore it cant find the asset because the corresponding asset for the UUID it gives is not actually there.

In reality at least it is probably bad practice to have assetIDs (UUIDS) changing all the time as this clutters the database and asset services with orphaned and essentially non existent assets which could potentially cause other problems in the long run.

Hopefully this provides a bit more of the clue of the causes of this issue.

[appurist]

Thanks, if I see the same thing, that's much more than "a bit more of the clue", that's handing it to me on a silver platter. ;) Thanks again, I'll check and try to reproduce.

[emperorstarfinder]

A quick note on this one as well

I did just do another test on this as it got me to wondering if it was more then just the assetid being missing or if it is just a viewer issue. SO I tested with Singularity (it is old and shouldnt be supported by grids these days), Alchemy (latest Alchemy version), and Firestorm (latest version). The same thing did happen. THe UUIDs in the database are different and the UUID for the estate covenant in the estate tables i have yet to actually find in the asset server itself (directory).

When I have a chance after I get all my team's features ported into our new architecture base code I'll have a further look to see if theres anything else I can find for you here.

[appurist]

I haven't had a chance to do my own testing yet on this, but I hope to soon.

The viewer still shows the problem after a relog, right? Viewers often do assign their own temporary asset IDs when saving a notecard (e.g. Ctrl-S while it's open, or closing one with changes). It then uploads the data and the server saves it and sends back the actual asset ID in an update. This leads to problems where if you close a notecard (or script) A with changes B, and then quickly reopen the notecard before the viewer is aware of the real asset ID, it doesn't have the inventory or Contents item update yet so it requests the original notecard A. If you then make further changes and save that, it will be A+C without the B changes, and those changes will be lost. I'm wondering here if there's a similar problem if this quickly follows notecard changes. It may be applying the original (perhaps empty) notecard asset ID to the covenant. You should be able to reopen the notecard and see the correct contents before applying it as a covenant.

Also, a lot of this may be mixing terms, since if you're getting the IDs from the viewers, the viewers sometimes mislabel inventory item IDs as "asset IDs". (They aren't.) If you give someone a copy of the notecard, or even copy/paste it in your own inventory, they should have the same asset ID, but different inventory IDs. The covenant ID in the database will be the actual server-side asset ID.

[emperorstarfinder]

I had thought the same as you just described myself here. the UUIDs I saw in the estate table in the database might have been assigned by the viewer which could explain why the error message was coming up. However when pasing the notecard between users and following that test the UUID didn't appear to change nor when updating the notecard itself.

As a further experiment I even went as far as changing the assetID in the estate table for the covenant to point to the actual notecard and got the same message even after a full restart (grid and region services).

The problem does appear to persist in any viewer (i tested with Singularity, Alchemy, and Firestorm, I didn't have a chance to test with Kokua). Ideally viewers should respect the assetIDs set by the service and not by the viewer. This is probably a fault of the Lindens when they originally did the viewers as well.

If we find this is going by the assetID in the viewer perhaps it is wise to find a way to link that back to the real assetID until a more complete fix can be done to get rid of orphaned assetIDs and empty assets created by the viewers.

[appurist]

What version of Halcyon are you running?

[emperorstarfinder]

I tested it via the latest version from master branch.