System detecting wrong end date time auctions

[iwalucas]

I have just deployed the ebay-snipe-server, and all the auctions I am watching are returning something like that:

{"ended":true,"endingAt":"1970-01-01T00:00:00.000Z",...}

So I cannot place a snipe b/c it says auction has ended.

[ruippeixotog]

@iwalucas I have just tried the server and I had no problem with the /auction/<id> endpoint. Can you tell me one of the IDs that is failing for you and any custom setting you may have in your application.conf?

[iwalucas]

@ruippeixotog here is one I just picked up: 292496978835

this is the output:
{"ended":true,"endingAt":"1970-01-01T00:00:00.000Z","location":"Riga, default, Letônia","bidCount":0,"shippingCost":"GBP 5.0","currentBid":null,"id":"292496978835","buyNowPrice":"GBP 7.0","defaultCurrency":"GBP","title":"SI TECH LP Scuba Hose BCD Inflator Dry Suit Standart Fitting Dive 80cm/31.5inch | eBay","seller":{"id":"to_deal","feedback":1409,"positivePercentage":100.0},"thumbnailUrl":"https://i.ebayimg.com/images/i/292496978835-0-1/s-l1000.jpg"}

there are no custom settings

[ruippeixotog]

That's strange... I have just tried getting that auction and I got the right date:

{
    "ended": false,
    "endingAt": "2018-04-23T17:21:19.000+01:00",
    "location": "Riga, default, Latvia",
    "bidCount": 0,
    "shippingCost": "GBP 5.99",
    "currentBid": null,
    "id": "292496978835",
    "buyNowPrice": "GBP 7.49",
    "defaultCurrency": "GBP",
    "title": "SI TECH LP Scuba Hose BCD Inflator Dry Suit Standart Fitting Dive 80cm/31.5inch  | eBay",
    "seller": {
        "id": "to_deal",
        "feedback": 1411,
        "positivePercentage": 100
    },
    "thumbnailUrl": "https://i.ebayimg.com/images/i/292496978835-0-1/s-l1000.jpg"
}

It seems that the date is the only field with wrong information. Are you running this on Windows? What's the locale of your computer?

[iwalucas]

I am running on a Mac, with GMT-3 timezone

[ruippeixotog]

Ok, I'm running on Mac too. The fact that you mentioned "Letônia" while I had "Latvia" at the item location points to a difference in localization on the eBay webpage, which can cause this issue if the date is in another place or in another format. It also makes this hard for me to reproduce it... Can you tell me the output of locale in your shell?

[iwalucas]

sure, here it is: LANG="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_CTYPE="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_ALL=

[lcd1232]

I have the same error:

root@server:~# http localhost:3647/auction/202601112701
HTTP/1.1 200 OK
Content-Length: 439
Content-Type: application/json
Date: Mon, 18 Feb 2019 17:38:10 GMT
Server: akka-http/10.0.9

{
    "bidCount": 0,
    "buyNowPrice": null,
    "currentBid": "EUR 600.0",
    "defaultCurrency": "EUR",
    "ended": true,
    "endingAt": "1970-01-01T00:00:00.000Z",
    "id": "202601112701",
    "location": "Cassino, Италия",
    "seller": {
        "feedback": 44,
        "id": "ice_plug",
        "positivePercentage": 100.0
    },
    "shippingCost": "EUR 39.0",
    "thumbnailUrl": "https://i.ebayimg.com/images/i/202601112701-0-1/s-l1000.jpg",
    "title": "APPLE MACBOOK PRO 13\" RETINA DISPLAY 2015 RAM 8GB SSD 128GB   | eBay"
}

[Terr]

Hi,

I am encountering the same issue when requesting auctions from my local eBay (ebay.nl). The date is correctly returned when getting the same auction via ebay.com.

Like in the US, the ending date is mentioned also mentioned in the .vi-tm-left element, under #LeftSummaryPanel, on the Dutch version. But query("ending-at") keeps returning a None when logging the output in EbayClient.scala.

I don't think I'm at the point where date formatting matters, but I have changed it in my ebay-nl.conf to match the Dutch date formats:

auction-info {
  attributes {
    ending-at = { query: ".vi-tm-left, .lable:contains(\"Afgelopen:\") + .u-flL", date-formats: ["(dd MMM yyyy HH:mm:ss zzz)", "dd-MMM HH:mm"] }
  }
}

The locale of the environment the server is running in is set to nl_NL.UTF-8.

The other data is found and returned just fine. Any idea what could cause only .vi-tm-left to not be found?

[Terr]

I still have no idea what causes the issue, but I have learned two things:

• My changes to auction-info.attributes in my ebay-nl.conf were not overriding those of common.conf. So for now I'm editing common.conf directly.
• If I leave out the date-formats attribute in the configuration, query("ending-at") does find the element. But queryType("ending-at") then fails of course.

[ruippeixotog]

Hi @Terr! It sounds plausible that ebay.nl CSS queries have flaws, I didn't test the project with any website other than ebay.com.

Do you mind sending me your current ebay-nl.conf and one or more auctions that are currently failing? Once this is fixed I can include the .nl config in the repo, if you want.

[Terr]

Hi, thanks for the reply!

I have pasted my ebay-nl.conf and common.conf in this Gist.

I had to modify common.conf because I failed to override the auction-info.attributes via ebay-nl.conf but maybe I did it wrong. Should it be possible to override those attributes in a site-specific config file?

The new date format pattern is correct I think, but because that's something that I needed to change I might have made a mistake there.

Here are two random auctions that I tried just now and have a few days remaining:

• https://www.ebay.nl/itm/Lot-Of-10-Used-Micro-SD-Memory-Cards/233202703952?hash=item364bf6de50:g:lywAAOSwTmxcuSu~ (ID: 233202703952)
• https://www.ebay.nl/itm/Aviator-folding-sunglasses-With-case/183773933879?hash=item2ac9c7b137:g:~XgAAOSw02Fciq3b (ID: 183773933879)

I did not translate all the text patterns in ebay-nl.conf yet, but the ones that I did do are enough to log in and retrieve auctions. After I've come across the other texts that need translation I would be happy to share it in a pull request :)

[ruippeixotog]

Hi @Terr! Sorry for taking so long to reply. I finally got the chance to look at your config and the problem seems to be that the date pattern in auction-info.attributes.ending-at.date-formats does not accept dutch month names like "mei".

Unfortunately scala-scraper, the library that extracts content from HTML in the server, does not support localization currently. I just opened ruippeixotog/scala-scraper#65, feel free to contribute if you find the time for it 🙂

[ruippeixotog]

Closing this issue because I'm not officially supporting ebay.nl (even though I'd like to have a config for it here) and I don't have any reproducible example of this failing in ebay.com.

[Terr]

Thanks for taking the time to investigate.

I'm not sure I can contribute code to scala-scraper but I've subscribed to the issue, in case I can help with testing it.

Should I get a complete and working config for ebay.nl together in the future I will be sure to create a merge request :)