tensorflow lite operation -> nnapi/armcl IFunction(plan._ops) 변환과정

[timedilation]

tensorflow/contrib/lite/nnapi_delegate.cc

BuildGraph에서 ANeuralNetworksModel 생성 (변수명 nn_model_)
(과정) AddOpsAndParams 에서 tflite model operation을 변환
ANeuralNetworksModel_addOperation을 통해 nn_model_에 operation추가

ANeuralNetworksModel_addOperation Detail

229   auto &graph = model->deref();
230 
231   auto node_param =
232       neurun::graph::operation::Node::InitParam{inputCount, inputs, outputCount, outputs};
234   switch (type)
235   {
236     case ANEURALNETWORKS_CONV_2D:
244       if (inputCount == 7)
245       {
246         using GraphNode = neurun::graph::operation::Conv2D::Implicit::Node;
247 
248         graph.addOperation(nnfw::make_unique<GraphNode>(node_param));

graph.addOperation

 42 operation::Index Graph::addOperation(std::unique_ptr<operation::Node> &&node)
 43 {
 44   assert(_phase == Phase::BUILDING);
 45   return _operations.append(std::move(node));
 46 }

_operations 은 operation들의 set

nn_model_을 이용하여 ANeuralNetworksCompilation (변수명 nn_compiled_model_) 생성
630 CHECK_NN(ANeuralNetworksCompilation_create(nn_model_, &nn_compiled_model_));

 26   ANeuralNetworksCompilation(const std::shared_ptr<neurun::graph::Graph> &model)
 27       : _plan{new neurun::codegen::Plan{model}}

src/codegen/Plan.h

class Plan
32   Plan(const std::shared_ptr<neurun::graph::Graph> &model) : _model(model)

이후 nnapi_delegate에서 연산(Invoke)시
ANeuralNetworksExecution_startCompute 함수 호출, ANeuralNetworksExecution 개체 생성

641   ANeuralNetworksExecution* execution = nullptr;
642   CHECK_NN(ANeuralNetworksExecution_create(nn_compiled_model_, &execution));

neurun/src/frontend/execution.cc

ANeuralNetworksExecution_create 함수에서
compilation(nn_compiled_model) 에서 plan 개체를 받아 Execution개체 생성(publish(plan)).

 38   std::shared_ptr<const neurun::codegen::Plan> plan;
 40   compilation->publish(plan)
 42   *execution = new (std::nothrow) ANeuralNetworksExecution{plan};

neurun/src/frontend/wrapper/compilation.h

publish detail

 36   void publish(std::shared_ptr<const neurun::codegen::Plan> &plan) { plan = _plan; }

neurun/src/frontend/wrapper/execution.h

_plan은 ANeuralNetworksExecution의 member variable

 27   ANeuralNetworksExecution(const std::shared_ptr<const neurun::codegen::Plan> &plan) : _plan{plan}
 28   {
 29     _sources.resize(_plan->model().getInputs().size());
 30     _sinks.resize(_plan->model().getOutputs().size());
 31   }

neurun/src/codegen/Plan.h

 29 class Plan
 30 {
 31 public:
 32   Plan(const std::shared_ptr<neurun::graph::Graph> &model) : _model(model)
 33   {
 34     // DO NOTHING
 35   }
 36 
 49 private:
 50   std::shared_ptr<neurun::graph::Graph> _model;
 51   operand::Context _operands;
 52   operation::Sequence _ops;
 53 };

[timedilation]

Graph::addOperation함수의 return _operations.append(std::move(node)); statement는 모델의 graph 에 operation node들을 추가함. append함수 detail은 다음과 같음.

graph/operation/Set.cc

 35 Index Set::append(std::unique_ptr<Node> &&node)
 36 {
 37   auto index = generateIndex();
 38 
 39   _nodes[index] = std::move(node);
 40   return index;
 41 }

graph/operation/Set.h

 54   std::unordered_map<Index, std::unique_ptr<Node>> _nodes;

[timedilation]

plan._ops, plan._operands 는 nnapi_delegate.cc의 ANeuralNetworksCompilation_finish(nn_compiled_model_)) 를 통해 설정될 것으로 보임.

neurun/src/frontend/wrapper/compilation.cc

 45 int ANeuralNetworksCompilation::finish()
 46 {
 47   auto &plan = this->plan();
 48   const auto &operands = plan.model().operands();
 49 
 50   plan.model().lower();
 51   auto linear = plan.model().linearize();
 52 
 53   // Dump ops
 54   linear->accept(neurun::graph::dumper::Dumper{});
 55 
 56   neurun::codegen::PlanBuilder plan_builder{plan};
 57 
 58   auto tensor_builders = linear->markTensors();
 59 
 60   linear->accept(neurun::codegen::Planner{operands, plan_builder});
 61 
 62   // TODO Add optimization passes
 63   plan_builder.finalize(tensor_builders);
 64 
 65   return ANEURALNETWORKS_NO_ERROR;
 66 }

line 56에서 plan_builder 생성 후, 63에서 plan_builder.finalize(tensor_builders)가 호출됨

neurun/src/codegen/PlainBuilder.cc

 38 void PlanBuilder::finalize(const backend::TensorBuilderSet &tensor_builders)
 46   // Process Stage
 47   ExecutionBuilder execution_builder{_plan};
 48 
 49   for (const auto &stage : _stages)
 50   {
 51     stage(execution_builder);
 52   }

ExecutionBuilder의 append함수가 _plan의 _ops에 IFunction을 추가하기 때문에 stage(execution_builder)에서 _ops관련 작업이 있을 것으로 추측함. (constructor에서는 _plan 포인터
assign만 일어남)

neurun/src/codegen/PlanBuilder.h

 32 public:
 33   ExecutionBuilder(codegen::Plan &plan) : _plan{plan}
 34   {
 35     // DO NOTHING
 36   }
 37 
 38 public:
 39   void append(std::unique_ptr<::arm_compute::IFunction> &&f) override
 40   {
 41     _plan.operations().append(std::move(f));
 42   }

[timedilation]

_stages는 Stage타입의 벡터이며, Stage는 operation Node를 argument로 받는 함수 타입.

neurun/src/backend/IStageGenerator.h

 42 using Stage = std::function<void(IExecutionBuilder &)>;

Stage는 operation dependent한 Planner:visit을 통해 assign됨.
해당 함수가 어디에서 호출되는지는 더 분석 필요.

neurun/src/codegen/Planner.cc

 33 void Planner::visit(const graph::operation::Conv2D::Implicit::Node &node
 34 {
 60   // Generate Stage
 61   auto stage_gen = backend.stage_gen();
 62   _builder.addStage(stage_gen->generate(node));

generate에서 드디어 ExecutionBuilder::append가 불리는데, 이 함수는 operation dependent하며, backend/cpu/StageGenerator.cc 와 backend/acl_cl/StageGenerator.cc 모두가 같은 이름의 함수들을 갖고 있음. 짧은 operation (Reshape)를 예시로 카피함. line 491에서 append가 불림.

neurun/src/backend/acl_cl/StageGenerator.cc

465 Stage StageGenerator::generate(const graph::operation::Reshape::Node &node)
466 {
467   const ::neurun::graph::operand::Index output_index{node.getOutputs().at(0)};
468   const ::neurun::graph::operand::Index input_index{node.getInputs().at(0)};
469 
470   struct Param
471   {
472     int output_index;
473     int input_index;
474   };
475 
476   Param param;
477 
478   param.output_index = output_index.asInt();
479   param.input_index = input_index.asInt();
480 
481   auto tensors = _tensor_builder;
482 
483   return [tensors, param](IExecutionBuilder &builder) {
484     auto output_alloc = tensors->at(::neurun::graph::operand::Index{param.output_index}).get();
485     auto input_alloc = tensors->at(::neurun::graph::operand::Index{param.input_index}).get();
486 
487     auto fn = make_layer<::arm_compute::CLReshapeLayer>();
488 
489     fn->configure(input_alloc, output_alloc);
490 
491     builder.append(std::move(fn));
492   };
493 }

[timedilation]

원래 IFunction을 어떻게 만드는지만 확인하면 되는 문제로 생각하고 _ops가 생성되는 부분을 찾아보았으나, 그 과정에서 operation dependent 한 타입과 함수들이 많이 발견됨. 원래 계획을 폐기하고 같은 과정을 다시 따라가며 operation dependent한 부분들을 정리하는 것으로 계획 수정함.

[timedilation]

#22 함수 콜을 따라가는 대신, dependent한 경우 전부 선언시 graph node 타입에 operation 이름이 들어간다는 사실을 이용하여 검색함.