SDL::modulesBuffer as an ES product #16

slava77 · 2023-10-20T09:02:56Z

This is coupled with SegmentLinking/TrackLooper#355 where SDL::modulesBuffer for its const ES-like part was supplanted by SDL::modulesBufferES, which is not owned by SDL Event.
This allows to create an instance of a modulesBuffer in CMSSW side as an ES product which can be consumed directly in the SDL code.

slava77 · 2023-10-20T09:03:16Z

Note to self: needs to be rebased

…r in the device space with Alpaka

…ducer

VourMa

Not sure whether this push included any additional development and whether things work now. From a first look, the logic makes sense and I mostly have comments to verify that I understand things.
If things still do not work, could you remind me what you see at the output, i.e. which part of the buffers doesn't get properly propagated? In this case, debugging will be needed as I see no smoking gun issue.

RecoTracker/LST/BuildFile.xml

VourMa · 2023-11-17T14:45:51Z

RecoTracker/LST/interface/LSTModulesDev.h

+
+namespace cms::alpakatools {
+  template <>
+  struct CopyToDevice<SDL::modulesBuffer<alpaka_common::DevHost>> {


Just so that I can understand better, what is the alpaka_common::DevHost type used here? A type to be used for alpaka buffers by both host and device?

in alpaka_common it comes as using DevHost = alpaka::DevCpu

VourMa · 2023-11-17T15:00:22Z

RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc

+}
+
+  std::optional<SDL::modulesBuffer<alpaka_common::DevHost>> LSTModulesDevESProducer::produce(const TrackerRecoGeometryRecord &iRecord) {
+    // write directly to SDL : FIXME : SHOULD NOT HAPPEN HERE


What does this comment refer to?

it refers to the lines below L47-48 write to the SDL globals; this should eventually become local

VourMa · 2023-11-17T15:16:50Z

RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc

+    SDL::modulesBuffer<alpaka_common::DevHost> modules(cms::alpakatools::host());
+    alpaka::QueueCpuBlocking queue(cms::alpakatools::host());
+    SDL::LST::loadMaps();
+    SDL::loadModulesFromFile(&modules, SDL::nModules, SDL::nLowerModules, *SDL::pixelMapping, queue, txtFile_.c_str());


What is the reason to treat modules and pixelMapping separately, i.e. one is constructed here, while the other in the SDL::Event constructor?

I didn't get there yet

RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc

slava77 · 2023-11-17T15:30:09Z

Note to self: needs to be rebased

now rebased

slava77 · 2023-11-17T15:31:26Z

If things still do not work, could you remind me what you see at the output, i.e. which part of the buffers doesn't get properly propagated? In this case, debugging will be needed as I see no smoking gun issue.

after rebase I have an error

[10] Calling method for module LSTProducer@alpaka/'lstProducer'
Exception Message:
A std::exception was thrown.
...
returned error  : 'cudaErrorInvalidDevice': 'invalid device ordinal'!

which I don't know yet if it's related to my ES work or some mess up I have in the configuration

slava77 · 2023-11-17T15:46:13Z

RecoTracker/LST/src/alpaka/ES_ModulesDev.cc

@@ -0,0 +1,3 @@
+#include "RecoTracker/LST/interface/LSTModulesDev.h"
+#include "HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h"
+TYPELOOKUP_ALPAKA_TEMPLATED_DATA_REG(SDL::modulesBuffer);


Hi @makortel
Is there a way to avoid having an explicit host-specific RecoTracker/LST/src/ES_ModulesDev.cc ?
For some reason this TEMPLATED_DATA_REG leads to appropriate symbols in libRecoTrackerLSTCudaAsync.so, but nothing in the libRecoTrackerLSTSerialSync.so

This behavior doesn't change if I remove RecoTracker/LST/src/ES_ModulesDev.cc.
Can LTO be at play in some way?
I tried to add <flags CXXFLAGS="-fno-lto"/>, but it didn't make the CPU ES edm::typelookup::className symbols show up in the CPU/serial so file.

Is there a way to avoid having an explicit host-specific RecoTracker/LST/src/ES_ModulesDev.cc ?

No, the behavior is intentional (at least for now). We want the non-Alpaka modules to be able to consume the host-side EventSetup data products (analogously to host-side Event data products). From there it follows that the symbols of the host-side data products must be in the non-Alpaka library of RecoTracker/LST, and the .cc file containing the TYPELOOKUP_DATA_REG() for the host-specific data product must reside there.

slava77 · 2023-12-06T16:14:31Z

@makortel

I'm trying to debug a setup with dual external build (lst_gpu and lst_cuda). The job crashes in some not very descriptive place in the framework (Tracer says it's right after ++ starting: begin job and igtrace backtrace is below.
Do you have a suggestion how to debug further? I'm guessing I need some finer tracer details.
This is 13_3_0_pre3 with LST extras

06-Dec-2023 07:37:09.45 PST  ++ starting: begin job
A fatal system signal has occurred: segmentation violation
#5  0x00007fa67c475332 in __strcmp_avx2 () from /lib64/libc.so.6
#6  0x00007fa67ef031ed in __gnu_cxx::__normal_iterator<edm::eventsetup::EventSetupRecordKey const*, ...
#7  0x00007fa67ede4f1c in edm::eventsetup::ESRecordsToProductResolverIndices::indexInRecord
#8  0x00007fa67edde357 in edm::EDConsumerBase::updateLookup
#9  0x00007fa67eef2246 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::updateLookup
#10 0x00007fa67eece8b6 in edm::WorkerManager::beginJob

the commit range that introduces the dual cpu/cuda build and also the ES details are in a0caa0f...cbfc1aa

makortel · 2023-12-06T19:57:10Z

@slava77 Does the crash happen when running on CPU, or when running on GPU?

makortel · 2023-12-06T20:00:31Z

I also assume that lst_cpu and lst_cuda contain same symbols, i.e. only one of them can be loaded into a process. Is this correct?

Is lst_headers just providing the headers (that have the same content in both lst_cpu and lst_cuda)?

slava77 · 2023-12-07T06:27:37Z

@slava77 Does the crash happen when running on CPU, or when running on GPU?

it's a default alpaka for LST modules, so I guess it's GPU.

slava77 · 2023-12-07T06:29:45Z

I also assume that lst_cpu and lst_cuda contain same symbols, i.e. only one of them can be loaded into a process. Is this correct?

Is lst_headers just providing the headers (that have the same content in both lst_cpu and lst_cuda)?

lst_headers package does not provide a library

<tool name="lst_headers" version="1.0">
  <client>
    <environment name="LSTBASE" default="$CMSSW_BASE/external/lst/TrackLooper"/>
    <environment name="INCLUDE" default="$LSTBASE"/>
  </client>
  <runtime name="LST_BASE" value="$LSTBASE"/>
</tool>

slava77 · 2023-12-07T06:34:20Z

I also assume that lst_cpu and lst_cuda contain same symbols, i.e. only one of them can be loaded into a process. Is this correct?

I didn't check. The alpaka kernels will be different symbols, but yes, there can be the same symbols for non-dev-templated code

Recall that the code without my ES-related commits (up to 72edb80 in the diff runs

slava77 · 2023-12-07T16:26:36Z

with debug symbols added I found that the crash is in SeedCreatorFromRegionHitsEDProducerT<SeedFromConsecutiveHitsCreator> during calling edm::EDConsumerBase::updateLookup
In it, the crash is m_esTokenInfo in it's first element

m_record = {type_ = {
<edm::TypeIDBase> = {t_ = 0x7fff1edb2fa0}, 
m_name = 0x8e44 <error: Cannot access memory at address 0x8e44>}}, 
m_key = {type_ = {<edm::TypeIDBase> = {
        t_ = 0x7fff94c048a8 <typeinfo for TrackerGeometry>}, 
m_name = 0x7fff94c00066 "TrackerGeometry"}

So, oddly enough the ES record detail gets corrupted and it's in a non-LST module.
I could guess though that it's in the TrackerRecoGeometryRecord (the new ES consumed in the LSTProducer.cc).
Where is the place upstream to check the origin of corruption?

makortel · 2023-12-07T17:34:59Z

Thanks @slava77. Going with the hypothesis the cause would be an ODR violation, I'd suggest the following in order to move forward (it's a little bit nasty, bit could be technically ok for now, given that the SDL::modulesBuffer<T> ES products are an implementation detail of the LST plugins, and as long as no-one tries to run the CPU and GPU versions simultaneously in one job)

Replace RecoTracker/LST/src/ES_ModulesDev.cc and RecoTracker/LST/src/alpaka/ES_ModulesDev.cc with RecoTracker/LST/plugins/alpaka/ES_ModulesDev.cc that contains

#include <SDL/Module.h>
#include "HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h"

// Temporary hack: The DevHost instantiation is needed in both CPU and GPU plugins,
// whereas the (non-host-)Device instantiation only in the GPU plugin
TYPELOOKUP_DATA_REG(SDL::modulesBuffer<alpaka_common::DevHost>);
TYPELOOKUP_ALPAKA_TEMPLATED_DATA_REG(SDL::modulesBuffer);

Move the specialization of cms::alpakatools::CopyToDevice<SDL::modulesBuffer<alpaka_common::DevHost>> to RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc and remove RecoTracker/LST/interface/LSTModulesDev.h (or move the header to ../plugins/alpaka directory).

Remove <use name="lst_cpu"/> from RecoTrackerLSTPlugins in RecoTracker/LST/plugins/BuildFile.xml.

Then a higher level question: are you using ESProducer only for the purpose to copy the data to all devices, or is it foreseen for the ES product to be used (shared) by many EDModules?

makortel · 2023-12-07T19:09:48Z

RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc

+LSTModulesDevESProducer::LSTModulesDevESProducer(const edm::ParameterSet &iConfig)
+  : ESProducer(iConfig), txtFile_{iConfig.getParameter<edm::FileInPath>("txt").fullPath()}
+{
+  setWhatProduced(this, iConfig.getParameter<std::string>("ComponentName"));


The ComponentName is strictly speaking redundant, because the framework already provides the same functionality out of the box with appendToDataLabel parameter.

makortel · 2023-12-07T19:11:05Z

RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc

+    SDL::modulesBuffer<alpaka_common::DevHost> modules(cms::alpakatools::host());
+    alpaka::QueueCpuBlocking queue(cms::alpakatools::host());
+    SDL::LST::loadMaps();
+    SDL::loadModulesFromFile(&modules, SDL::nModules, SDL::nLowerModules, *SDL::pixelMapping, queue, txtFile_.c_str());


(possibly only just curious) For what is the CPU queue used here?

The queue is used in e.g. alpaka::memcpy(queue, modulesBuf->nModules_buf, src_view_nModules);

Guessing from the variable names, is this a copy from non-Alpaka-buffer-CPU-memory (src_view_nModules) to a Alpaka-buffer-CPU-memory (modulesBuf->nModules_buf)? If this is the case, could the copy be done with std::memcpy() (or with an assignment if the thing being copied is one number)?

In general we have struct modulesBuffer<TAcc>* modulesBuf and the method is used to fill the GPU modules as well (in the standalone version).
In this specific case in the ESProducer, things are explicitly on host/CPU.

So, I'm not sure how to interpret your suggestion to use std::memcpy in this combined context.

Ok, I understand the code is more general towards the target device of that memcpy(), and the host target is just a special case. Thanks for the clarification. Would a comment be worth it?

Would a comment be worth it?

sure, I can add

slava77 · 2023-12-08T12:41:37Z

Going with the hypothesis the cause would be an ODR violation, I'd suggest the following ... and as long as no-one tries to run the CPU and GPU versions simultaneously in one job

Don't we have a more substantive problem though:

pluginRecoTrackerLSTPlugins.so includes LSTOutputConverter.cc, LSTPixelSeedInputProducer.cc, LSTPhase2OTHitsInputProducer.cc and depends on libsdl_cpu.so
pluginRecoTrackerLSTPluginsPortableCudaCudaAsync.so includes LSTModulesDevESProducer.cc, LSTProducer.cc and depends on libsdl_cuda.so

libsdl_cpu.so and libsdl_cuda.so contain overlapping sets of symbols.
I can confirm by running a version before my changes that gdb info sharedlib shows both libsdl_cpu.so and libsdl_cuda.so are loaded.
Or is this safe in case library dependencies for plugins are somehow isolated?

makortel · 2023-12-08T15:01:16Z

Going with the hypothesis the cause would be an ODR violation, I'd suggest the following ... and as long as no-one tries to run the CPU and GPU versions simultaneously in one job

Don't we have a more substantive problem though:

pluginRecoTrackerLSTPlugins.so includes LSTOutputConverter.cc, LSTPixelSeedInputProducer.cc, LSTPhase2OTHitsInputProducer.cc and depends on libsdl_cpu.so

pluginRecoTrackerLSTPluginsPortableCudaCudaAsync.so includes LSTModulesDevESProducer.cc, LSTProducer.cc and depends on libsdl_cuda.so

Good point, already this setup should be a problem. For time being (in order to make progress, but to be undoed in the future) it would be ok to move the LSTOutputConverter.cc, LSTPixelSeedInputProducer.cc, and LSTPhase2OTHitsInputProducer.cc from plugins/ to plugins/alpaka/.

libsdl_cpu.so and libsdl_cuda.so contain overlapping sets of symbols. I can confirm by running a version before my changes that gdb info sharedlib shows both libsdl_cpu.so and libsdl_cuda.so are loaded.

I forgot to ask earlier if these "same symbols" are

same symbols (functions) containing the same code, or
same symbols (functions) containing different code (e.g. CPU-specific code in case of libsdl_cpu.so and CUDA-specific code in case of libsdl_cuda.so)

I have been assuming the latter (and now I can't remember if it is just my assumption, or if we discussed about this detail in the hackathon).

Or is this safe in case library dependencies for plugins are somehow isolated?

No, library dependencies via plugins do not get any "extra safety". The library setup is somewhat different in that case though (like the loading order), so if the problem is really in ODR violation (which is undefined behavior), I could believe one setup to look like it works, and the other to fail in strange way.

slava77 · 2023-12-08T15:26:55Z

I forgot to ask earlier if these "same symbols" are

* same symbols (functions) containing the same code, or

* same symbols (functions) containing different code (e.g. CPU-specific code in case of `libsdl_cpu.so` and CUDA-specific code in case of `libsdl_cuda.so`)

both kind; there are quite a few helper methods that are independent of the accelerator type

slava77 · 2023-12-08T16:04:30Z

@makortel
I see that ASAN_OPTIONS=detect_odr_violation=2 (or 1) could be used.
Even though now it's obvious that the problem is there, it could be something useful to weed out the issues.
Is there a way to run it with scram in the regular build or do I need to rebuild in the ASAN build?

slava77 · 2023-12-08T16:22:16Z

Replace RecoTracker/LST/src/ES_ModulesDev.cc and RecoTracker/LST/src/alpaka/ES_ModulesDev.cc with RecoTracker/LST/plugins/alpaka/ES_ModulesDev.cc that contains

#include <SDL/Module.h>
#include "HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h"

// Temporary hack: The DevHost instantiation is needed in both CPU and GPU plugins,
// whereas the (non-host-)Device instantiation only in the GPU plugin
TYPELOOKUP_DATA_REG(SDL::modulesBuffer<alpaka_common::DevHost>);
TYPELOOKUP_ALPAKA_TEMPLATED_DATA_REG(SDL::modulesBuffer);

this does not compile:

In file included from /cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaCore/interface/alpaka/ESDeviceProduct.h:8,
                 from /cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h:5,
                 from $CMSSW_BASE/src/RecoTracker/LST/src/alpaka/ES_ModulesDev.cc:2:
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaInterface/interface/config.h:103:45: error: template argument 2 is invalid
  103 |   using Acc = alpaka::AccCpuSerial<TDim, Idx>;
      |                                             ^

makortel · 2023-12-08T16:32:43Z

@makortel I see that ASAN_OPTIONS=detect_odr_violation=2 (or 1) could be used. Even though now it's obvious that the problem is there, it could be something useful to weed out the issues. Is there a way to run it with scram in the regular build or do I need to rebuild in the ASAN build?

I don't really know, but I'd guess the full ASAN build would be safer. I wonder if we already use detect_odr_violation in the ASAN IB.

makortel · 2023-12-08T16:46:15Z

Replace RecoTracker/LST/src/ES_ModulesDev.cc and RecoTracker/LST/src/alpaka/ES_ModulesDev.cc with RecoTracker/LST/plugins/alpaka/ES_ModulesDev.cc that contains

#include <SDL/Module.h>
#include "HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h"

// Temporary hack: The DevHost instantiation is needed in both CPU and GPU plugins,
// whereas the (non-host-)Device instantiation only in the GPU plugin
TYPELOOKUP_DATA_REG(SDL::modulesBuffer<alpaka_common::DevHost>);
TYPELOOKUP_ALPAKA_TEMPLATED_DATA_REG(SDL::modulesBuffer);

this does not compile:

In file included from /cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaCore/interface/alpaka/ESDeviceProduct.h:8,
                 from /cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h:5,
                 from $CMSSW_BASE/src/RecoTracker/LST/src/alpaka/ES_ModulesDev.cc:2:
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaInterface/interface/config.h:103:45: error: template argument 2 is invalid
  103 |   using Acc = alpaka::AccCpuSerial<TDim, Idx>;
      |                                             ^

Mmkay, the error looks strange, as the Idx should be defined in the config.h. Does the compilation log have any other failures?

slava77 · 2023-12-08T18:15:51Z

I don't really know, but I'd guess the full ASAN build would be safer. I wonder if we already use detect_odr_violation in the ASAN IB.

https://github.com/cms-sw/cmssw-config/blob/7061c60a2606087a9f8ff456d3afc4bbc78a0fca/Projects/CMSSW/Self.xml#L57
detect_odr_violation=0

slava77 · 2023-12-08T18:20:16Z

Mmkay, the error looks strange, as the Idx should be defined in the config.h. Does the compilation log have any other failures?

sorry, I stopped on the first part of the error message:

/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaInterface/interface/config.h:103:45: error: template argument 2 is invalid
  103 |   using Acc = alpaka::AccCpuSerial<TDim, Idx>;
      |                                             ^
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaInterface/interface/config.h:104:17: error: 'Acc' {aka 'alpaka::AccCpuSerial<std::integral_constant<long unsigned int, 3>, long unsigned int>'} is not a template
  104 |   using Acc1D = Acc<Dim1D>;
      |                 ^~~
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaInterface/interface/config.h:105:17: error: 'Acc' {aka 'alpaka::AccCpuSerial<std::integral_constant<long unsigned int, 3>, long unsigned int>'} is not a template
  105 |   using Acc2D = Acc<Dim2D>;
      |                 ^~~
/cvmfs/cms.cern.ch/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_0_pre3/src/HeterogeneousCore/AlpakaInterface/interface/config.h:106:17: error: 'Acc' {aka 'alpaka::AccCpuSerial<std::integral_constant<long unsigned int, 3>, long unsigned int>'} is not a template
  106 |   using Acc3D = Acc<Dim3D>;
      |                 ^~~

there is also a GPU variant of the same set (not included above)

makortel · 2023-12-08T21:14:39Z

Mmkay, the error looks strange, as the Idx should be defined in the config.h. Does the compilation log have any other failures?

sorry, I stopped on the first part of the error message:

Thanks, although the rest didn't really help. I tested the following

#include "HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h"

template <typename Dev>
struct FooTest {
  int foo;
};

TYPELOOKUP_DATA_REG(FooTest<alpaka_common::DevHost>);
TYPELOOKUP_ALPAKA_TEMPLATED_DATA_REG(FooTest);

as a HeterogeneousCore/AlpakaTest/plugins/alpaka/foo.cc and that compiled fine in 13_3_0_pre3. Does SDL::modulesBuffer<T> do something special?

makortel · 2023-12-09T02:09:12Z

I was able to reproduce the compilation error by adding using Idx = std::size_t; before the #include ".../typelookup.h (which is what is done in https://github.com/SegmentLinking/TrackLooper/blob/master/SDL/Constants.h#L25). The setup is also easy to reproduce in compiler explorer https://godbolt.org/z/heMbdTaqr.

I'd suggest to enclose those type aliases in SDL into a namespace. Or open an issue in CMSSW GitHub that the setup in config.h is somewhat brittle. Or do both. Or reorganize the code in SDL such that only the modulesBuffer gets exposed to CMSSW. Or reorder the includes so that SDL/Module.h comes last.

slava77 · 2023-12-28T23:09:18Z

I'd suggest to enclose those type aliases in SDL into a namespace.

I moved to namespace SDL in 10fe919

slava77 · 2024-01-03T23:21:57Z

I updated the PR description now that this seems to run

…setup

VourMa

This side of the ES producer developments looks ok to me, so I am approving. I will push the button to merge at the same time that the Tracklooper part is also ready.

slava77 mentioned this pull request Nov 14, 2023

updates to get module maps used as ES products jointly with CMSSW SegmentLinking/TrackLooper#355

Merged

slava77devel added 4 commits November 16, 2023 18:10

skeleton implementation for SDL::modulesBuffer ES product and produce…

5aa6223

…r in the device space with Alpaka

an example hook to use SDL::modulesBuffer as ES product in the LSTPro…

efec09f

…ducer

fill modules data in the buffer as an ES product

7942f33

fixup to dual lst setup: add lst_headers

cbfc1aa

slava77 force-pushed the from-CMSSW_13_3_0_pre3_LST_X/modules-dev branch from 9d57ff7 to cbfc1aa Compare November 17, 2023 03:41

VourMa reviewed Nov 17, 2023

View reviewed changes

slava77 commented Nov 17, 2023

View reviewed changes

makortel reviewed Dec 7, 2023

View reviewed changes

decouple SDL dependencies on CPU and GPU by moving more to /alpaka

7b50c1b

set SDL::modulesBuffersES

dd468e3

slava77 marked this pull request as ready for review January 3, 2024 23:10

VourMa mentioned this pull request Jan 16, 2024

module map ES producers #14

Closed

slava77devel added 5 commits January 19, 2024 07:55

add lstProducer_cff to include era modifiers and also the related ES …

f3bbe84

…setup

specify the centroid file in fillDescriptions

473d09e

drop outdated/invalid parameters

55c3a80

update to new LST call interface to fill ES data

c4e0dac

comment cleanup

ebb3001

VourMa approved these changes Jan 30, 2024

View reviewed changes

VourMa merged commit 9c06cdb into CMSSW_13_3_0_pre3_LST_X Feb 2, 2024

ariostas mentioned this pull request Mar 6, 2024

Move Maps to CSV Based Method SegmentLinking/TrackLooper#374

Merged

SDL::modulesBuffer as an ES product #16

SDL::modulesBuffer as an ES product #16

Conversation

slava77 commented Oct 20, 2023 • edited Loading

slava77 commented Oct 20, 2023

VourMa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slava77 commented Nov 17, 2023

slava77 commented Nov 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slava77 commented Dec 6, 2023

makortel commented Dec 6, 2023

makortel commented Dec 6, 2023

slava77 commented Dec 7, 2023

slava77 commented Dec 7, 2023

slava77 commented Dec 7, 2023

slava77 commented Dec 7, 2023

makortel commented Dec 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slava77 commented Dec 8, 2023

makortel commented Dec 8, 2023

slava77 commented Dec 8, 2023

slava77 commented Dec 8, 2023

slava77 commented Dec 8, 2023

makortel commented Dec 8, 2023

makortel commented Dec 8, 2023

slava77 commented Dec 8, 2023

slava77 commented Dec 8, 2023

makortel commented Dec 8, 2023

makortel commented Dec 9, 2023 • edited Loading

slava77 commented Dec 28, 2023

slava77 commented Jan 3, 2024

VourMa left a comment

Choose a reason for hiding this comment

slava77 commented Oct 20, 2023 •

edited

Loading

makortel commented Dec 9, 2023 •

edited

Loading