API speed and timeout issue while using MongoDB Backend #159
Comments
|
Hi All, I am able to reproduce this issue with less objects however. To me it happens when object count in the collection is around 3,000 ~ 4,000. Both taxii server and MongoDB instances have enough resources. It happens on both locally and AWS environments. It takes about 5 minutes after the request for the following log snippet to be thrown:
|
|
Have you tried using mongo db indexing? Here is what we do for testing:
Please tell us if this helps at all. |
Looks like we don't get a proper error response because the error handler has a bug. I think this has been fixed in PR #145 , but the fix hasn't made it into a release. So there are actually two issues at play here: the original error, and the error handler error.
This is a guess, but it looks related to paging, and I have seen paging errors caused by slowness and a smaller session timeout. If session timeout ( Regarding general slowness, I think the mongo backend is inefficient. One misconception people have is that if no filters are given, that no filtering should happen, and queries should be simple and fast. But that's not how TAXII works. TAXII requires that certain default filtering happen. So you can improve performance by adding filters which effectively disable the default filtering. Default filters are defined for |
|
@rpiazza Indexing seems like a good option. Let me try and revert with the response. @chisholm I'd disagree on the inefficiency of mongodb. It can handle millions of records with and without filters. Mongodb takes it quite smoothly. Something is wrong with the query that has been built for it. I'm using for One more observation, after a stix object is pushed to a mongodb collection. Open the collection and check the type of If you plan to change the DB, then TSDB can be a better option for a data which has to be queried using a date range. Elastic Search implementation can also make it super fast. |
That's not what I meant. The medallion architecture includes a separation between the web part (using flask) which handles HTTP requests and responses, and a "backend" which handles the data. There are two backends included, one for mongo (which interfaces with mongodb) and an in-memory backend (doesn't use any database). I meant the mongo backend is inefficient.
If you don't use the "all" version filter, the mongo backend will still go through the process of finding the most recent versions of all the objects. That adds overhead.
As far as I understand, that was done intentionally because the builtin timestamp type didn't support microsecond precision. |
|
@rpiazza I tried adding index to the mongodb. Post that I again tried to poll the data with limits as well as filters.
|
|
@rpiazza thanks for the reply. I used initialization script from test data of this repo which already includes adding indexes to the db. After checking, indexes are indeed already added: `rs0:PRIMARY> db.objects.getIndexes() ] |
Interesting thought, I will test it with our client. However, if this is a root cause of the issue, a fix should be applied to the server because normally there is no control over clients that do not use matching filters. UPDATE: Using both |
|
As of a few weeks ago, we are in the process of transferring maintenance of all of these OASIS Open STIX/TAXII utilities to others. We still monitor these repositories, and would be happy to contribute a fix for a minor bug, but this appears to be a much more involved change. One of the issues is that this is a reference implementation of the TAXII server - a proof of concept - and not intended to be used in production. The new maintainers might be able to do more (@ejratl can you respond?). Also, there are other open-source TAXII server implementations available, which may be more robust. |
|
1. if my understanding is correct, Mitre no longer allows its employees to maintain TAXII, so they should submit a PR to remove their names from the page
iPhone, iTypo, iApologize
…________________________________
From: Emily Ratliff ***@***.***>
Sent: Friday, May 20, 2022 1:08:19 PM
To: oasis-open/cti-taxii-server ***@***.***>
Cc: duncan sfractal.com ***@***.***>; Mention ***@***.***>
Subject: Re: [oasis-open/cti-taxii-server] API speed and timeout issue while using MongoDB Backend (Issue #159)
@rpiazza<https://github.com/rpiazza> Unfortunately, I am not one of the new maintainers of TAXII, perhaps @sparrell<https://github.com/sparrell> can answer instead. (We are also still in need of Co-Chairs for the TAXII WG, in case anyone is interested.)
—
Reply to this email directly, view it on GitHub<#159 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AANEXD7TTFQUAB6PGNIDO2TVK7BIHANCNFSM5V6WDLPQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
2. if my understanding is correct, Mitre no longer doing active development on TAXII, so people shouldn't just assume "other people" will fix stuff. It's a community. Put issues out there. If no one fixes, and you need it - then fix it.
iPhone, iTypo, iApologize
…________________________________
From: duncan sfractal.com ***@***.***>
Sent: Sunday, May 22, 2022 7:25:08 AM
To: oasis-open/cti-taxii-server ***@***.***>; oasis-open/cti-taxii-server ***@***.***>
Cc: Mention ***@***.***>
Subject: Re: [oasis-open/cti-taxii-server] API speed and timeout issue while using MongoDB Backend (Issue #159)
1. if my understanding is correct, Mitre no longer allows its employees to maintain TAXII, so they should submit a PR to remove their names from the page
iPhone, iTypo, iApologize
________________________________
From: Emily Ratliff ***@***.***>
Sent: Friday, May 20, 2022 1:08:19 PM
To: oasis-open/cti-taxii-server ***@***.***>
Cc: duncan sfractal.com ***@***.***>; Mention ***@***.***>
Subject: Re: [oasis-open/cti-taxii-server] API speed and timeout issue while using MongoDB Backend (Issue #159)
@rpiazza<https://github.com/rpiazza> Unfortunately, I am not one of the new maintainers of TAXII, perhaps @sparrell<https://github.com/sparrell> can answer instead. (We are also still in need of Co-Chairs for the TAXII WG, in case anyone is interested.)
—
Reply to this email directly, view it on GitHub<#159 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AANEXD7TTFQUAB6PGNIDO2TVK7BIHANCNFSM5V6WDLPQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
3. I agreed to help with governance of PRs and I'll evaluate PRs meet our guidelines bi did not agree to fix all bugs.
iPhone, iTypo, iApologize
…________________________________
From: duncan sfractal.com ***@***.***>
Sent: Sunday, May 22, 2022 7:27:17 AM
To: oasis-open/cti-taxii-server ***@***.***>; oasis-open/cti-taxii-server ***@***.***>
Cc: Mention ***@***.***>
Subject: Re: [oasis-open/cti-taxii-server] API speed and timeout issue while using MongoDB Backend (Issue #159)
2. if my understanding is correct, Mitre no longer doing active development on TAXII, so people shouldn't just assume "other people" will fix stuff. It's a community. Put issues out there. If no one fixes, and you need it - then fix it.
iPhone, iTypo, iApologize
________________________________
From: duncan sfractal.com ***@***.***>
Sent: Sunday, May 22, 2022 7:25:08 AM
To: oasis-open/cti-taxii-server ***@***.***>; oasis-open/cti-taxii-server ***@***.***>
Cc: Mention ***@***.***>
Subject: Re: [oasis-open/cti-taxii-server] API speed and timeout issue while using MongoDB Backend (Issue #159)
1. if my understanding is correct, Mitre no longer allows its employees to maintain TAXII, so they should submit a PR to remove their names from the page
iPhone, iTypo, iApologize
________________________________
From: Emily Ratliff ***@***.***>
Sent: Friday, May 20, 2022 1:08:19 PM
To: oasis-open/cti-taxii-server ***@***.***>
Cc: duncan sfractal.com ***@***.***>; Mention ***@***.***>
Subject: Re: [oasis-open/cti-taxii-server] API speed and timeout issue while using MongoDB Backend (Issue #159)
@rpiazza<https://github.com/rpiazza> Unfortunately, I am not one of the new maintainers of TAXII, perhaps @sparrell<https://github.com/sparrell> can answer instead. (We are also still in need of Co-Chairs for the TAXII WG, in case anyone is interested.)
—
Reply to this email directly, view it on GitHub<#159 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AANEXD7TTFQUAB6PGNIDO2TVK7BIHANCNFSM5V6WDLPQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
HI @sparrell - "Maintainers" doesn't really have a definition in this context. For the most part, it just means one has permission to merge in a PR. Our sponsor supported MITRE to develop and fix these prototypes, but feels that the community should take the responsibility from now on. As with most things, it is hard to get people to do something they aren't paid to do. That being said, the way I understand it, you have volunteered for exactly what you stated above: governance of PRs and evaluating PRs to meet our guidelines. If you want to take it further then that - it is entirely up to you. |
|
Commenting here too since i have a feeling my pull request #174 might resolve this one too I already raised another issue, since i encountered a hard limit that prevented accessing any objects at all if the collection grew too big, so i felt like it deserved its own issue. I rewrote a bit of the filtering logic of the mongodb backend. These changes increased performance quite alot. Paging is also handled in a way now, that it should not result in 502 errors either due to the Hoping some of the maintainers would get notified this way and the review process could proceed, or at least if still relevant, people in this issue could try out the changes in that pull request if it solves their problems. |
Hi Team,
I've setup the server for MongoDBBackend, and using a taxii2client to test the setup.
The setup is working fine when the count of collection objects is less than few thousands.
However, when the collection objects exceed over 100,000 , then the get_objects api is resulting into a 502 error and timing out even when the limit is restricted to 5 objects.
The text was updated successfully, but these errors were encountered: