Add more downloader tests by recording and replaying

[kyrcha]

This PR includes:

• examples/cmd/testing/recordingscript.go: a script that records requests (the query string) and responses (the data) of the downloader and stores them in a gob file for later replaying. I am not sure if this is the correct place to put it, but I wanted it to run from CLI that is why I added it to the main package.
• github/downloader_test.go: which augments the tests with offline tests that do not need the GH token and also enable pagination to take place.
• 2 .gob files and their json oracles that replay the crawling of the src-d organization and the src-d/gitbase repo. More tests like that can be added.

Signed-off-by: Kyriakos Chatzidimitriou [email protected]

[kyrcha]

Thanks for the comments. I will resolve them ASAP.

[smacker]

But making them lossier (i.e., number of prs > 2 vs. number of prs == 2) is a good idea.

that is exactly what I had in mind.

change big gob files

good point. thanks! Even I don't think 5mb is actually big but anyway, in my opinion, it would be a good idea to compress the fixtures. I don't know how compressible gob files are. But we can always change the underlying format to something else like msgpack if gob doesn't fit us.

[kyrcha]

OK I'll check compression, and if it works (well enough) I'll let you know

[se7entyse7en]

Another concern I have is that when later we need to change big gob files (5MB for example) containing recordings whether this will impact the performance of git due to add/rm large files. WDYT?

Another idea could be to track them separately in another repo, so before the tests, we just download the version we need. But as @smacker said I don't think that 5MB would cause many issues.
The downside of this approach is that we would require having an internet connection also for recorded data.

[se7entyse7en]

Maybe I'd have done something like:

type DownloaderTestSuite struct {
	suite.Suite
	db         *sql.DB
	downloader *Downloader
	jsonTests  testutils.Tests
}

type DownloaderOnlineMemoryTestSuite struct {
	DownloaderTestSuite
}

type DownloaderOfflineMemoryTestSuite struct {
	DownloaderTestSuite
}

type DownloaderOnlineDBTestSuite struct {
	DownloaderTestSuite
}

type DownloaderOfflineDBTestSuite struct {
	DownloaderTestSuite
}

with 2 tests each, one for repo and one for org. Most probably many things for setup could be unified in DownloaderSuite instead of repeating them inside each test. But just an idea.

[kyrcha]

Yeah, that is nicer.

[kyrcha]

Another concern I have is that when later we need to change big gob files (5MB for example) containing recordings whether this will impact the performance of git due to add/rm large files. WDYT?

Another idea could be to track them separately in another repo, so before the tests, we just download the version we need. But as @smacker said I don't think that 5MB would cause many issues.
The downside of this approach is that we would require having an internet connection also for recorded data.

With gzip they are now 80% smaller so perhaps we could keep it like this and if they keep changing polluting the git history we can move them to another repo.

[lwsanty]

@se7entyse7en @kyrcha
moving these files to a separate repo concerns me more than having them in this repo as test data.

If we wanted to use this data for regression scope one day(this day's gonna be quite soon)
it's better to have test data "attached" to the version, like in gitbase project

In the other case we should keep the relations between current repo's version and test data repo's version and it's pretty much painful

am I missing something?

[lwsanty]

sorry for pointing at this each time, but these are internal imports :(

[kyrcha]

Yeah sorry. Visual studio code is rearranging stuff all the time and then I forget to manually rearrange them. I should add some pre-commit hook to handle this stuff. Do you have something to propose?

[lwsanty]

I do it manually tbh, but probably there should be some linter for this purpose

[smacker]

The current size of gob files is 800kb. One JPEG picture may have a size of a few mb. I don't think we should care about this 800kb at all.

[se7entyse7en]

@lwsanty

In the other case we should keep the relations between current repo's version and test data repo's version and it's pretty much painful

True, even if I don't know how much painful would it be, but you guys know surely better than me how to handle these things. Also, in this case, it's not a real problem to think about. 👍

[lwsanty]

@kyrcha just a kind reminder that some minor merge conflicts left

[smacker]

Yeah. Let's solve the conflicts and merge!

[kyrcha]

Tests are failing due to newer methods. Will fix the issues and ping you back

[kyrcha]

• Fix some old method signatures
• Update recordings to incorporate different queries done now
• Update oracles
  @src-d/applications

[smacker]

Thanks! But better use rebase next time.