cannot run algorithm from manually ingested data

[kadereub]

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

• Operating System: Mac Os - Darwin
• Python Version: Python 3.6.4 :: Anaconda, Inc.
• Python Bitness: 64bit
• How did you install Catalyst: conda using the py36 yaml
• Python packages: All the packages in the above yaml file.
• Catalyst Version: catalyst, version 0.5.21

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• I am trying to ingest exchange data from a CSV, and then execute a simple algorithm.
• I referenced the following issue thread to setup my csv for ingestion.
• I do not get any errors when I ingest the exchange data from the csv. However I get the below error when I try to run the simple algorithm.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. Use this as a sample set for the exchange data.
   catalyst_test_btc_usd.csv.zip

2. I then ingest this using the below command:
   catalyst clean-exchange -x bitfinex --csv data/catalyst_test_btc_usd.csv -f minute

Results in the below:

Trying to ingest exchange bundle bitfinex...
[2020-03-07 11:09:52.696229] INFO: exchange_bundle: ingesting csv file: data/catalyst_test_btc_usd.csv

3. Following this I try run the simple example in the docs. I save this fiel locally and run the following command,

catalyst run -f backtest/buy_btc_simple.py -x bitfinex --start 2019-7-2 --end 2019-7-30 -c usd --capital-base 100000 -o backtest/buy_btc_simple_out.pickle --data-frequency minute

This results in the following error:

catalyst.exchange.exchange_errors.PricingDataNotLoadedError: Missing data for bitfinex ['btc_usd'] in date range [2019-07-01 23:59:00+00:00 - 2019-07-01 23:59:00+00:00]
Please run: `catalyst ingest-exchange -x bitfinex -f minute -i btc_usd`. See catalyst documentation for details.

...

What steps have you taken to resolve this already?

I've tried to use the data example on the issue 65 thread. As I thought if I could get a simple test going then might be my data set. This is bat_eth data for bittrex.

catalyst ingest-exchange -x bittrex --csv data/bittrex_bat_eth.csv
...
catalyst run -f backtest/buy_btc_simple.py -x bittrex --start 2017-6-2 --end 2017-6-10 -c eth --capital-base 100000 -o backtest/buy_btc_simple_out.pickle --data-frequency minute

However this results in a separate error,

catalyst.exchange.exchange_errors.NoDataAvailableOnExchange: Requested data for trading pair BAT/ETH is not available on exchange bittrex in `minute` frequency at this time. Check `http://enigma.co/catalyst/status` for market coverage.

I'm simply trying to get the manual ingesting correct using the sample dataset I attached and run the simple example buy_btc in the docs, any help would be great!

Sincerely,
RK

[kadereub]

Update:

I managed to ingest the data, so that it was saved under the ~/.catalyst/data/exchanges/bitfinex/minute_bundles folder. I did this by changing the symbol from btc_usd to btc_usdt, which seemed to solve the ingestion problem.

However oddly I still got the error PricingDataNotLoadedError, after debugging further it seems the reader.get_value line in bundle_utils was breaking. After further investigation for some reason the path was being incorrectly compiled in the reader for the asset.sid.

I manually changed the path to match that of the asset.sid, so for example,

• ~/.catalyst/data/exchanges/bitfinex/minute_bundles/09/88/098871.bcolz

This fixed my problem, however I'm sure this is not the correct way of ingesting the csv data and reading it in for a strat...