Documentation on parsing the getEntries results

[xMTinkerer]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is your feature request related to a problem? Please describe.
With the soon to be sunset activity log, we are migrating the code way back here to use the new audit log. However, the data seems to be buried in the value which is a Buffer. I'm guessing this is some kind of protobuf, but not really sure which one or where to find out. The type_url is type.googleapis.com/google.cloud.audit.AuditLog, which seems to be the thing documented here, but going from a Buffer to digging out the value we need is a jump we can't seem to make.

Describe the solution you'd like
An example of using getEntries that goes beyond dumping them to the console. For example

 async function printEntryMetadata() {
   const [entries] = await logging.getEntries(options);
   console.log('Logs:');
   entries.forEach(entry => {
     metadata = entry.metadata;
     console.log( `${metadata.timestamp}:`, metadata[metadata.payload] );

     /*
        Do some magic here to pull out the `foo.bar` value
    */

   });
 }

Describe alternatives you've considered
It's not pretty.

            sliced = entries[i].metadata.protoPayload.value.slice(4)
            bufstring = sliced.toString()
            console.log("bufstring: " + bufstring)
            email = bufstring.substring(0, bufstring.search('"'))

Additional context

None?

[xMTinkerer]

With the retirement of the activity logs, we're re-evaluating this again. I managed to track down an npm library, protobufjs that looks like it might help. Then I found the audit_log.proto file and came up with this quick test code:

protobuf.load("audit_log.proto", function(err, root) {
 
    if (err)
        throw err;

 
    // Exemplary payload
 	 var protoUrl = entity.protoPayload.type_url; // "type.googleapis.com/google.cloud.audit.AuditLog"

 	 var protoType = protoUrl.substr( protoUrl.indexOf( '/' ) +1 ); // "google.cloud.audit.AuditLog"

 	 var AuditLogMessage = root.lookupType( protoType );

    // Verify the payload if necessary (i.e. when possibly incomplete or invalid)
    var errMsg = AuditLogMessage.verify( entity );
    if (errMsg)
        throw Error(errMsg);
 
    // Decode an Uint8Array (browser) or Buffer (node) to a message
    var message = AuditLogMessage.decode( entity.protoPayload.value.data );
    // ... do something with message
 	console.log( 'Message2: ' + JSON.stringify( message ) );

});

But, it actually fails on the load with this message:

> { [Error: ENOENT: no such file or directory, open 'google/rpc/context/attribute_context.proto']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'google/rpc/context/attribute_context.proto' }

Which seems to want this attribute_context.proto buried in one of the parent folders of the audit_log.proto file.

So am I to believe I need to include the entire google/rpc folder in my project, just to properly deserialize the buffer?

I really must be missing something.

[mfeyx]

@xMTinkerer I am having the same problems and I could not find any solution either.

But, you can simply call entity.protoPayload.value.data.toString() or even entity.data.toString() for short, and you get at least some information. In my case, I get wrongly decoded data with some chuncks that seem right. Still, the decoding is corrupt, and I tried several encodings.

And I also tried to use the protos module:

const { protos } = require('@google-cloud/logging')
const value = metadata[metadata.payload].value
protos.google.logging.v2.LogEntry.decode(value)

...which results in an error. Error: invalid wire type 7 at offset 569

If I use the following:

const { protos } = require('@google-cloud/logging')
const value = metadata[metadata.payload].value
protos.google.logging.v2.LogEntry.decodeDelimited(value)
// LogEntry { labels: {}, resource: MonitoredResource { labels: {} } }

...I get at least a well formatted object.

Anyway, the Information is emtpy.

[xMTinkerer]

Whew. At least someone else can comiserate. After some poking around, it seems you might have to compile the protocol buffer:
https://developers.google.com/protocol-buffers/docs/pythontutorial#compiling-your-protocol-buffers

Those instructions are for python, digging deeper shows the command comes from here:
https://github.com/protocolbuffers/protobuf

but they have no mention of nodejs, which is suspicious in that it is such a popular and flexible coding framework. Maybe there is a different way for node?

I also feel like this is a step backward from REST. Instead of putting the burden on the API maintainer, it is now on the API consumer. 🤷‍♂️

[xMTinkerer]

Ok, with some help, I tracked this down:
https://github.com/protocolbuffers/protobuf/tree/master/js
Which might be it. I'll see if I can give that a shot this week. I'll report back with my findings.

[mfeyx]

@xMTinkerer thanks for the links, I will have a look at this. Last year I used the python package for some little scripts, and it just worked out of the box. Don't know yet, what's different, but will let you know if I find something.

cheers!

[mfeyx]

@JustinBeckwith and @bcoe any news on this issue?

[mfeyx]

@xMTinkerer so I tried it with python, here is what I got:

ProtobufEntry(
  log_name='SOME_READABLE_VALUE', 
  labels=None, 
  insert_id='SOME_READABLE_VALUE', 
  severity='NOTICE', 
  http_request=None, 
  timestamp=datetime.datetime(2019, 8, 7, 14, 45, 39, 898000, tzinfo=<UTC>), 
  resource=Resource(type='SOME_READABLE_VALUE', 
  labels={'project_id': 'SOME_READABLE_VALUE'}), 
  trace=None, 
  span_id=None, 
  trace_sampled=None, 
  source_location=None, 
  operation=None,
  logger=<google.cloud.logging.logger.Logger object at 0x10729aa90>, 
  payload=type_url: "type.googleapis.com/google.cloud.audit.AuditLog"
  value: "\022\000\032\031\n\027SOME_READABLE_VALUE\"\226\001\n\016SOME_READABLE_VALUE\022\203\001TRUNCATED_VALUE"
)

As you see in the value field, it seems that the encoding is ascii-octal?!
http://defindit.com/ascii.html

What do you think?

[mfeyx]

So at the moment I just assume that the buffer is corrupt... tried it with python and used the class methods: Error.

from google.cloud import logging

client = logging.Client()

client.project = "my-project"
log_filter = '"resource.type="bigquery_resource"'

for entry in client.list_entries(filter_=log_filter):
    print(type(entry))
    entry.to_api_repr()

<class 'google.cloud.logging.entries.ProtobufEntry'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/google/protobuf/json_format.py", line 403, in _CreateMessageFromTypeUrl
    message_descriptor = pool.FindMessageTypeByName(type_name)
KeyError: "Couldn't find message google.cloud.audit.AuditLog"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "audit_logs.py", line 52, in <module>
    entry.to_api_repr()
  File "/usr/local/lib/python3.8/site-packages/google/cloud/logging/entries.py", line 357, in to_api_repr
    info["protoPayload"] = MessageToDict(self.payload)
  File "/usr/local/lib/python3.8/site-packages/google/protobuf/json_format.py", line 175, in MessageToDict
    return printer._MessageToJsonObject(message)
  File "/usr/local/lib/python3.8/site-packages/google/protobuf/json_format.py", line 214, in _MessageToJsonObject
    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File "/usr/local/lib/python3.8/site-packages/google/protobuf/json_format.py", line 331, in _AnyMessageToJsonObject
    sub_message = _CreateMessageFromTypeUrl(type_url, self.descriptor_pool)
  File "/usr/local/lib/python3.8/site-packages/google/protobuf/json_format.py", line 405, in _CreateMessageFromTypeUrl
    raise TypeError(
TypeError: Can not find message descriptor by type_url: type.googleapis.com/google.cloud.audit.AuditLog.

What somehow works is using this:

d = entry._asdict()
payload = d.get('payload')

# binary value
b = getattr(payload, 'value')

# decode the value
b.decode('utf-8')

But decode() throws the following error:

<class 'google.cloud.logging.entries.ProtobufEntry'>
Traceback (most recent call last):
  File "audit_logs.py", line 57, in <module>
    print(b.decode('utf-8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 256: invalid start byte

[xMTinkerer]

Yea, I have no idea at this point. I played with that protoc tool, specifically the js output, but I still can't get it to compile:

tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$  protoc --js_out=library=myproto_libs,binary:. \
>     --proto_path ../the-badger/ \
>     --proto_path ./google/protobuf/any.proto \
>     --proto_path ./google/protobuf/struct.proto \
>     --proto_path ./google/rpc/context/attribute_context.proto \
>     --proto_path ./google/rpc/status.proto \
>     ../the-badger/audit_log.proto 
google/rpc/context/attribute_context.proto: File not found.
google/rpc/status.proto: File not found.
audit_log.proto:21:1: Import "google/rpc/context/attribute_context.proto" was not found or had errors.
audit_log.proto:22:1: Import "google/rpc/status.proto" was not found or had errors.
audit_log.proto:69:3: "google.rpc.Status" is not defined.
audit_log.proto:165:3: "google.rpc.context.AttributeContext.Resource" is not defined.
audit_log.proto:211:3: "google.rpc.context.AttributeContext.Request" is not defined.
audit_log.proto:218:3: "google.rpc.context.AttributeContext.Peer" is not defined.
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ ll google/rpc/context/attribute_context.proto
-rw-r--r--  1 tinkerer  staff    12K Sep 11 16:48 google/rpc/context/attribute_context.proto
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ ll google/rpc/status.proto
-rw-r--r--  1 tinkerer  staff   1.9K Sep 11 16:48 google/rpc/status.proto
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 
tinkerer@appletree:~/xCode/googleapis$ 

While pacing around the living room, I realized a better and more google best practice sort of way is to use a sink. Since I have an AppEngine app anyway, I'll just tweak my code to listen instead of pull. This also has the added bonus of not having to deal with protobufs :D

[mfeyx]

hey @xMTinkerer it's not the ideal solution, but yes, it works.

[minherz]

@xMTinkerer did you find a method to retrieve the info that you need? In your sample you was trying to retrieve it from the metadata field. Did you check what do you get in data field?

[minherz]

I close the issue as this question is no longer require answer. Feel free to reopen the issue if it is still relevant.

[wunderlejohn]

Is there any movement here on a basic example of convert protocol buffers into JSON objects? feel like that's a stumbling block many would run into using this client library. I'm currently struggling to get buffer data of type 'type.googleapis.com/google.cloud.audit.AuditLog' into anything that might be usable. Have tried installing 'proto3-json-serializer', 'protobufjs' and 'google-proto-files' and following this documentation without any luck.

[mlitvinav]

I am able to parse the log entries with the following code. I did not manage to make it work with the inlcuded protos and had to use google-proto-files package to make it work.

The following code is in Bun and TypeScript. I am interested in audit logs only and you may need to extend it in order to parse the messages.
Extending means to parse the messages based on entry.data.type_url value.

  "devDependencies": {
    "@types/bun": "latest"
  },
  "peerDependencies": {
    "typescript": "^5.0.0"
  },
  "dependencies": {
    "@google-cloud/logging": "^11.0.0",
    "google-proto-files": "^4.2.0"
  }

You will need to login into the right google project via gcloud auth application-default login in order to run this.

import { Logging } from '@google-cloud/logging'
import protofiles from 'google-proto-files'

const logging = new Logging()

async function listLogEntries() {
   const filter = `protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"`;
   let pageToken = undefined
   const [entries] = await logging.getEntries({
      autoPaginate: false,
      filter,
      pageSize: 10,
      pageToken: undefined,
      orderBy: 'timestamp desc',
   });

   for (const entry of entries) {
      const payload = (entry as any).metadata.payload as string // typemismatch; value is present but invisible
      if (payload == 'protoPayload' && Buffer.isBuffer(entry.metadata[payload]?.value)) {
         const protopath = protofiles.getProtoPath('../google/cloud/audit/audit_log.proto')
         const root = protofiles.loadSync(protopath)
         const type = root.lookupType('google.cloud.audit.AuditLog')
         const value = type.decode(entry.metadata[payload]?.value as Uint8Array)
         const raw = value as any // typemismatch; values are present but invisible
         console.log('example access', raw.requestMetadata.callerIp)
      }
   }
}
await listLogEntries().catch(console.error)

I'll post how to convert the object into a json in #1421
Since the raw object is untyped converting the value to json via toJSON has no real drawback, as it has to beany anyways but will kick out the empty fields on your entry.