Verbose dump data can cause a loss of queued data in the service listeners

[michael-zock]

Hi

I just encountered a problem where data can be lost because the listener can't write extremely long queue data to the database and restarting the listener will mean the loss of the currently queued reports.

In my case the method processQueue() in the bugLogListenerAsync.cfc went into the catch block, because the _insert() in the dbDataProvider.cfc was unable to insert record due to the extra data.In this case, some HTML from a remote service, whose 55,399 "slightly" exceeded the 5000 character limit in your schema (I'm dealing with a MS SQL Server, where the system wouldn't allow more than 8,000 characters anyway). ;)

Stopping and restarting the listener would mean losing the queued data, which is not exactly desirable.
Wouldn't it be better to do a "left(trim( foo ), 5000)" on the content beforehand to salvage at least some of the reported error report rather than losing all of it and a ton of other stuff caught in the same queue?

[oarevalo]

Hi Michael,
The large field in this case is the exception detail or the HTMLReport? If
it's the HTMLReport what I've done on some installations is replace the
field definition on the db table with a TEXT field. This was done manually
on the DB and the change was transparent to BugLogHQ.

Your sugestion of trimming the data sounds very good, and I actually think
there is some trimming elsewhere.

Thanks,

Oscar

On Mon, Jan 6, 2014 at 9:54 PM, Michael Zock [email protected]:

Hi

I just encountered a problem where data can be lost because the listener
can't write extremely long queue data to the database and restarting the
listener will mean the loss of the currently queued reports.

In my case the method processQueue() in the bugLogListenerAsync.cfc went
into the catch block, because the _insert() in the dbDataProvider.cfc was
unable to insert record due to the extra data.In this case, some HTML from
a remote service, whose 55,399 "slightly" exceeded the 5000 character limit
in your schema (I'm dealing with a MS SQL Server, where the system wouldn't
allow more than 8,000 characters anyway). ;)

Stopping and restarting the listener would mean losing the queued data,
which is not exactly desirable.
Wouldn't it be better to do a "left(trim( foo ), 5000)" on the content
beforehand to salvage at least some of the reported error report rather
than losing all of it and a ton of other stuff caught in the same queue?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/77
.

Oscar Arevalo
http://about.me/oarevalo
http://www.coldbricks.com - Content Management System
http://www.oscararevalo.com - My Blog

[michael-zock]

Hi Oscar

Yes, switching the field type ("bl_Entry.ExceptionDetails") to "ntext" would solve the length restriction, but (at least on an MS SQL Server) such columns are neither indexable nor searchable (unless one casts them to a varchar at query runtime and foregoes any performance advantages an existing index would offer), meaning we'd actually lose the ability to search in those fields, a valuable asset for report filters.

It's just one of those days. Damned if you do, damned if you don't.

[michael-zock]

At a second glance, judging by the column name it seems like the offender is actually the description field rather than the HTML report and therefore a much more likely candidate for a left() operation. Seems a colleague somehow managed to get the HTML output of some malfunctioning web service in there.

The HTML report column already seems to be a "text" type column (at least in the setup I started out with).

[oarevalo]

Hi Michael,

'I've been debugging this issue and I see that before a bug report gets written to the database a left() operation gets applied to truncate the string to the appropriate length. The only exception is the HTMLReport field, which as noted should be defined as a TEXT field on the bl_Entry table.

Have you run into this issue again?

• Oscar

[michael-zock]

Hi Oscar

Yes, I've come across the problem sporadically, even though the left() in the entry.cfc should take care of it.
At this point I'm not quite sure why exactly it happens, but true to Murphy's Law it's usually during weekends or eventful days, often costing me hours worth of debug information.