Basic stream solution

[jingxihuang411]

Description

Implemented basic functionalities for MonitorStream

Solution

Created MonitorStream and Alertstream. MonitorStream currently support "update" and "alert" operator:

• "update" will send matching tuples returned by topic stream to destinationCollection
• "alert" will send matching tuples returned by topic stream via http request to user-specified URL

Tests

MonitorStream Update Solution

1. Create destinationCollection, collection1
2. Update some documents in the collection1: {"id": "id1", "a_s": "hello", "a_i": "0", "a_f":"1"}
3. Create MonitorStream in collection1:
   monitor(id="percolate1", operator="update", collection1, destinationCollection, q="a_s:hello", fl="id")
4. Update additional documents in collection1:
   {"id": "id2", "a_s": "hello", "a_i": "0", "a_f":"1"}
   {"id": "id3", "a_s": "hello", "a_i": "0", "a_f":"1"}
   {"id": "id4", "a_s": "hellohello", "a_i": "0", "a_f":"1"}
5. Check for matching ids id2 and id3 in destinationCollection
   Note: As per https://lucene.apache.org/solr/guide/8_4/near-real-time-searching.html#commits-and-optimizing, if want to check matches immediately in destinationCollection, need to set true in solrconfig.xml for destinationColletion; Otherwise restart Solr would show the matches

MonitorStream Alert Solution

1. Create collection1
2. Update some documents in the collection1: {"id": "id1", "a_s": "hello", "a_i": "0", "a_f":"1"}
3. Set up a another server running locally to receive the http request. For example, set up a python server running at "http://localhost:8080/abc" with the below code: https://gist.github.com/mdonkers/63e115cc0c79b4f6b8b3a6b797e485c7
4. Create MonitorStream in collection1:
   monitor(id="monitor1", operator="alert", collection1, url="http://localhost:8080/abc", q="a_s:hello", fl="id")
5. Update additional documents in collection1:
   {"id": "id2", "a_s": "hello", "a_i": "0", "a_f":"1"}
   {"id": "id3", "a_s": "hello", "a_i": "0", "a_f":"1"}
   {"id": "id4", "a_s": "hellohello", "a_i": "0", "a_f":"1"}
6. Check that the server running on localhost:8080 get the matching response for the specified field, in this case = {matches={id=13, version=1663375504531521536}}

Note: Currently the only alert function that is implemented is to send a http request to the url specified in the MonitorStream input. Only http requests not https requests are tested and supported now.

[HoustonPutman]

Why are you removing this?

[jingxihuang411]

When changed to useAlias = false since some of the tests cases are skipped becuase when useAlias has a random value of true, some tests with Assume.assumeTrue(!useAlias); will be skipped.

Changed back and remove Assume.assumeTrue(!useAlias); for these test cases

[HoustonPutman]

Is this method necessary? This logic could go into the second constructor, and the first constructor could call the second constructor.

[jingxihuang411]

If calling second inside the first constructor, this(tupleStream, targetURL) should be first line.

[HoustonPutman]

It might be nice if the user could receive the HTTP statuses of each alert request in the tuple.

[jingxihuang411]

Sorry I'm not sure which tuple do you mean. Right now the alert request is sent to the endpoint for each matching tuple so users could retrieve the HTTP status for each request received at the endpoint.

I added the timestamp for sending each request in the tuple.

[HoustonPutman]

when adding the timestamp, you should also add the HTTP status code and maybe other information so that the users know if certain documents were not processed properly

[HoustonPutman]

There looks like a lot of repeated logic in these init methods, might want to have one call the other

[HoustonPutman]

Also similarly to AlertStream, is there a reason to have these init methods, versus just putting the logic into the second two constructors?

[jingxihuang411]

The init method follows the same format as the constructors for other Stream expressions. Also if calling one constructor from another have the problem mentioned above. Moved repeated logic to a separate function instead.