Add 10k-v2 and stock_simulated for testing and benchmarking Rust library

[alecmocatta]

In the import of the Rust parquet library from http://github.com/sunchao/parquet-rs, the benchmarks and their required files 10k-v2.parquet and stock_simulated.parquet weren't copied across.

This adds them so that updated benchmarks can be included in apache/arrow#3461.

[wesm]

These files are a bit big. Can you generate some data files using pyarrow or similar as part of the benchmarking process?

[alecmocatta]

Good idea. Is it possible though that the generated files could change as pyarrow changes over time? I'd like to ensure the files are bit-for-bit identical over time, for consistent benchmark results.

[wesm]

You could pin the pyarrow version used to generate them

[alecmocatta]

Thanks. I'm not familiar with how to do that but I'll take a look once the aforementioned PR is merged. I removed the benchmarks from it so this PR isn't needed immediately.

[wesm]

I have a lot on my plate. @andygrove or @sunchao could you take ownership of this? I don't like the idea of checking 1MB files into source control, particularly for benchmarking, so please don't do that without checking with me first

[andygrove]

I'm happy to take this on. I am in the final week of a release in my day job so may not have much time until this weekend.

…

On Wed, Feb 13, 2019 at 2:23 PM Wes McKinney ***@***.***> wrote: I have a lot on my plate. @andygrove <https://github.com/andygrove> or @sunchao <https://github.com/sunchao> could you take ownership of this? I don't like the idea of checking 1MB files into source control, particularly for benchmarking, so please don't do that without checking with me first — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA5AxGzpauzu9JvRbqHlg_8NoiXRYh3Mks5vNII-gaJpZM4aS_k5> .

[andygrove]

I'm about to embark on some benchmarking with large files too so I face similar challenges.

Generating test data as part of the build seems like the way to go in order to catch performance regressions in the core libraries.

I am also maintaining my own repo where I have benchmarks against popular public data sets (and I'm just relying on wget or curl to download them).