Theano/Keras/etc implementations

[braingineer]

Hi @douwekiela

Looking through the code base. I think maybe the fastest way to get something up and running is to pick some core function that could be good to have ported over. Once we have a minimum working example, we can look to flesh it out from there. What do you think that would be?

Brian

[douwekiela]

Hi @braingineer,

Sounds good. The way to do it would be to have implementations in different frameworks, I think. So we'd have cnn/caffe.py, cnn/theano.py, etc. and then load the appropriate module based on availability of the relevant framework. We could then have a CNN parent class that the framework-specific implementations inherit from. The place to start would be here, I guess: https://github.com/douwekiela/mmfeat/blob/master/mmfeat/cnn/__init__.py. Maybe just start with AlexNet?

Cheers!

[braingineer]

Hi Douwe,

Sure! I've looked around and I found various people having implementations with keras. Rather than reimplementing the wheel, I'm going to try to splice things together. Pedagogically, would you want to have the code incorporated in or used as a submodule?

I haven't really used caffe before, so I have a quick question. Where do you handle the preprocessing and stuff, or does caffe automatically do that?

Your fit method just runs the network forward on all of the images, right? and then it caches the resulting layer activations in self.descriptors?

Given the stuff above, I should be able to submit a PR within the next day or so. I'll be traveling for a week, so I'll be pretty dormant until the end of August, I think.

I have some other thoughts about things that I would find super useful in this library:

• how do you feel about also adding h5py support?
• a large part of what I want to do with this is to have it serve data for me in various multimodal circumstances. so, what I mean, is that I am writing a larger model and need it to serve me the visual/whatever features. so what do you think of having a DataServer class that can either be used vanilla or subclassed to serve a multimodal dataset? I usually have something like this implemented: https://github.com/braingineer/ikelos/blob/master/ikelos/data/data_server.py

Best,
Brian

[douwekiela]

Hi Brian,

Sorry for the late reply, I was away for the weekend.

Pedagogically, would you want to have the code incorporated in or used as a submodule?
I think it'd be best to incorporate it in the same way the caffe module is incorporated - which probably means a minimalistic implementation.

Where do you handle the preprocessing and stuff, or does caffe automatically do that?
Yes, caffe handles that, kind of. You need to set transformation params:
https://github.com/douwekiela/mmfeat/blob/master/mmfeat/cnn/__init__.py#L70

Your fit method just runs the network forward on all of the images, right? and then it caches the resulting layer activations in self.descriptors?
Exactly.

I have some other thoughts about things that I would find super useful in this library
h5py: definitely! Data server sounds good too.

Thanks!

Douwe