Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SDL::modulesBuffer as an ES product #16

Merged
merged 11 commits into from
Feb 2, 2024
Merged

SDL::modulesBuffer as an ES product #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5aa6223
skeleton implementation for SDL::modulesBuffer ES product and produce…
slava77devel Oct 20, 2023
efec09f
an example hook to use SDL::modulesBuffer as ES product in the LSTPro…
slava77devel Oct 20, 2023
7942f33
fill modules data in the buffer as an ES product
slava77devel Nov 14, 2023
cbfc1aa
fixup to dual lst setup: add lst_headers
slava77devel Nov 17, 2023
7b50c1b
decouple SDL dependencies on CPU and GPU by moving more to /alpaka
slava77devel Dec 28, 2023
dd468e3
set SDL::modulesBuffersES
slava77devel Jan 3, 2024
f3bbe84
add lstProducer_cff to include era modifiers and also the related ES …
slava77devel Jan 19, 2024
473d09e
specify the centroid file in fillDescriptions
slava77devel Jan 19, 2024
55c3a80
drop outdated/invalid parameters
slava77devel Jan 20, 2024
c4e0dac
update to new LST call interface to fill ES data
slava77devel Jan 25, 2024
ebb3001
comment cleanup
slava77devel Jan 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions RecoTracker/IterativeTracking/python/HighPtTripletStep_cff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -387,14 +387,9 @@
from RecoTracker.LST.lstSeedTracks_cfi import lstInitialStepSeedTracks,lstHighPtTripletStepSeedTracks
from RecoTracker.LST.lstPixelSeedInputProducer_cfi import lstPixelSeedInputProducer
from RecoTracker.LST.lstPhase2OTHitsInputProducer_cfi import lstPhase2OTHitsInputProducer
from RecoTracker.LST.lstProducer_cfi import lstProducer as _lstProducer
# Define lstProducer like this to be able to run the serial version explicitly
lstProducer = _lstProducer.clone(
alpaka = cms.untracked.PSet(backend = cms.untracked.string('serial_sync'))
)
from Configuration.ProcessModifiers.gpu_cff import gpu
gpu.toReplaceWith(lstProducer,_lstProducer.clone())
_HighPtTripletStepTask_LST.add(siPhase2RecHits, lstInitialStepSeedTracks, lstHighPtTripletStepSeedTracks, lstPixelSeedInputProducer, lstPhase2OTHitsInputProducer, lstProducer)
from RecoTracker.LST.lstProducer_cff import *
_HighPtTripletStepTask_LST.add(siPhase2RecHits, lstInitialStepSeedTracks, lstHighPtTripletStepSeedTracks, lstPixelSeedInputProducer, lstPhase2OTHitsInputProducer,
lstProducer, lstModulesDevESProducer)
(trackingPhase2PU140 & trackingLST).toReplaceWith(HighPtTripletStepTask, _HighPtTripletStepTask_LST)

# fast tracking mask producer
Expand Down
6 changes: 5 additions & 1 deletion RecoTracker/LST/BuildFile.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
<use name="DataFormats/Common"/>
<use name="alpaka"/>
<use name="lst_headers"/>
VourMa marked this conversation as resolved.
Show resolved Hide resolved
<use name="DataFormats/TrackerRecHit2D"/>
<use name="HeterogeneousCore/AlpakaInterface"/>
<flags ALPAKA_BACKENDS="cuda serial"/>
<export>
<lib name="RecoTrackerLST"/>
<lib name="1"/>
</export>
4 changes: 4 additions & 0 deletions RecoTracker/LST/plugins/BuildFile.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
<use name="DataFormats/TrackerRecHit2D"/>
<use name="DataFormats/TrajectorySeed"/>
<use name="FWCore/Framework"/>
<use name="FWCore/MessageLogger"/>
<use name="FWCore/ParameterSet"/>
<use name="FWCore/Utilities"/>
<use name="Geometry/TrackerGeometryBuilder"/>
Expand All @@ -13,6 +14,7 @@
<use name="RecoTracker/LST"/>
<use name="RecoTracker/TkSeedingLayers"/>
<use name="RecoTracker/TkSeedGenerator"/>
<use name="RecoTracker/Record"/>
<use name="TrackingTools/GeomPropagators"/>
<use name="TrackingTools/Records"/>
<use name="TrackingTools/TrajectoryState"/>
Expand All @@ -31,6 +33,7 @@
<use name="HeterogeneousCore/AlpakaCore"/>
<use name="HeterogeneousCore/AlpakaInterface"/>
<use name="RecoTracker/LST"/>
<use name="RecoTracker/Record"/>
<flags ALPAKA_BACKENDS="serial"/>
<flags EDM_PLUGIN="1"/>
</library>
Expand All @@ -44,6 +47,7 @@
<use name="HeterogeneousCore/AlpakaCore"/>
<use name="HeterogeneousCore/AlpakaInterface"/>
<use name="RecoTracker/LST"/>
<use name="RecoTracker/Record"/>
<flags ALPAKA_BACKENDS="cuda"/>
<flags EDM_PLUGIN="1"/>
</library>
57 changes: 57 additions & 0 deletions RecoTracker/LST/plugins/alpaka/LSTModulesDevESProducer.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#include "FWCore/ParameterSet/interface/ParameterSet.h"

#include "HeterogeneousCore/AlpakaCore/interface/alpaka/ESProducer.h"
#include "HeterogeneousCore/AlpakaCore/interface/alpaka/ModuleFactory.h"
#include "HeterogeneousCore/AlpakaInterface/interface/config.h"
#include "HeterogeneousCore/AlpakaInterface/interface/memory.h"

#include "RecoTracker/Record/interface/TrackerRecoGeometryRecord.h"

// LST includes
#include <SDL/Module.h>
#include <SDL/ModuleMethods.h>
#include <SDL/LST.h>
VourMa marked this conversation as resolved.
Show resolved Hide resolved

namespace cms::alpakatools {
template <>
struct CopyToDevice<SDL::modulesBuffer<alpaka_common::DevHost>> {
template <typename TQueue>
static auto copyAsync(TQueue& queue, SDL::modulesBuffer<alpaka_common::DevHost> const& srcData) {
return SDL::modulesBuffer<alpaka::Dev<TQueue>>(queue, srcData);
}
};
} //namespace cms::alpakatools
namespace ALPAKA_ACCELERATOR_NAMESPACE {

class LSTModulesDevESProducer : public ESProducer {
public:
LSTModulesDevESProducer(const edm::ParameterSet &iConfig);

static void fillDescriptions(edm::ConfigurationDescriptions &descriptions);

std::optional<SDL::modulesBuffer<alpaka_common::DevHost>> produce(const TrackerRecoGeometryRecord &iRecord);

};

LSTModulesDevESProducer::LSTModulesDevESProducer(const edm::ParameterSet &iConfig)
: ESProducer(iConfig)
{
setWhatProduced(this, iConfig.getParameter<std::string>("ComponentName"));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ComponentName is strictly speaking redundant, because the framework already provides the same functionality out of the box with appendToDataLabel parameter.

}

void LSTModulesDevESProducer::fillDescriptions(edm::ConfigurationDescriptions &descriptions) {
edm::ParameterSetDescription desc;
desc.add<std::string>("ComponentName", "")->setComment("Product label");
descriptions.addWithDefaultLabel(desc);
}

std::optional<SDL::modulesBuffer<alpaka_common::DevHost>> LSTModulesDevESProducer::produce(const TrackerRecoGeometryRecord &iRecord) {
SDL::modulesBuffer<alpaka_common::DevHost> modules(cms::alpakatools::host());
alpaka::QueueCpuBlocking queue(cms::alpakatools::host());
SDL::LST::loadAndFillES(queue, &modules);
return modules;
}

} // namespace ALPAKA_ACCELERATOR_NAMESPACE

DEFINE_FWK_EVENTSETUP_ALPAKA_MODULE(LSTModulesDevESProducer);
9 changes: 8 additions & 1 deletion RecoTracker/LST/plugins/alpaka/LSTProducer.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#include <alpaka/alpaka.hpp>

#include "FWCore/MessageLogger/interface/MessageLogger.h"
#include "FWCore/ParameterSet/interface/ConfigurationDescriptions.h"
#include "FWCore/ParameterSet/interface/ParameterSet.h"
#include "FWCore/ParameterSet/interface/ParameterSetDescription.h"
Expand All @@ -16,6 +17,8 @@
#include "RecoTracker/LST/interface/LSTPhase2OTHitsInput.h"
#include "RecoTracker/LST/interface/LSTPixelSeedInput.h"

#include "RecoTracker/Record/interface/TrackerRecoGeometryRecord.h"

#include "SDL/LST.h"

namespace ALPAKA_ACCELERATOR_NAMESPACE {
Expand All @@ -26,6 +29,7 @@ namespace ALPAKA_ACCELERATOR_NAMESPACE {
: lstPixelSeedInputToken_{consumes<LSTPixelSeedInput>(config.getParameter<edm::InputTag>("pixelSeedInput"))},
lstPhase2OTHitsInputToken_{
consumes<LSTPhase2OTHitsInput>(config.getParameter<edm::InputTag>("phase2OTHitsInput"))},
modulesESToken_{esConsumes()},
verbose_(config.getParameter<int>("verbose")),
lstOutputToken_{produces()} {}

Expand All @@ -35,7 +39,9 @@ namespace ALPAKA_ACCELERATOR_NAMESPACE {
auto const& pixelSeeds = event.get(lstPixelSeedInputToken_);
auto const& phase2OTHits = event.get(lstPhase2OTHitsInputToken_);

lst_.eventSetup();
auto const& modulesData = setup.getData(modulesESToken_);
SDL::modulesBuffersES = &modulesData;
SDL::modulesInGPU->setData(modulesData);
lst_.run(event.queue(),
verbose_,
pixelSeeds.px(),
Expand Down Expand Up @@ -79,6 +85,7 @@ namespace ALPAKA_ACCELERATOR_NAMESPACE {
private:
edm::EDGetTokenT<LSTPixelSeedInput> lstPixelSeedInputToken_;
edm::EDGetTokenT<LSTPhase2OTHitsInput> lstPhase2OTHitsInputToken_;
device::ESGetToken<SDL::modulesBuffer<Device>, TrackerRecoGeometryRecord> modulesESToken_;
const int verbose_;
edm::EDPutTokenT<LSTOutput> lstOutputToken_;

Expand Down
9 changes: 9 additions & 0 deletions RecoTracker/LST/python/lstProducer_cff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
import FWCore.ParameterSet.Config as cms

from RecoTracker.LST.lstProducer_cfi import lstProducer

from Configuration.ProcessModifiers.gpu_cff import gpu
(~gpu).toModify(lstProducer.alpaka, backend = 'serial_sync')

from RecoTracker.LST.lstModulesDevESProducer_cfi import lstModulesDevESProducer
(~gpu).toModify(lstModulesDevESProducer.alpaka, backend = 'serial_sync')
7 changes: 7 additions & 0 deletions RecoTracker/LST/src/alpaka/ES_ModulesDev.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#include <SDL/Module.h>
#include "HeterogeneousCore/AlpakaCore/interface/alpaka/typelookup.h"

// Temporary hack: The DevHost instantiation is needed in both CPU and GPU plugins,
// whereas the (non-host-)Device instantiation only in the GPU plugin
TYPELOOKUP_DATA_REG(SDL::modulesBuffer<alpaka_common::DevHost>);
TYPELOOKUP_ALPAKA_TEMPLATED_DATA_REG(SDL::modulesBuffer);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @makortel
Is there a way to avoid having an explicit host-specific RecoTracker/LST/src/ES_ModulesDev.cc ?
For some reason this TEMPLATED_DATA_REG leads to appropriate symbols in libRecoTrackerLSTCudaAsync.so, but nothing in the libRecoTrackerLSTSerialSync.so

This behavior doesn't change if I remove RecoTracker/LST/src/ES_ModulesDev.cc.
Can LTO be at play in some way?
I tried to add <flags CXXFLAGS="-fno-lto"/>, but it didn't make the CPU ES edm::typelookup::className symbols show up in the CPU/serial so file.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to avoid having an explicit host-specific RecoTracker/LST/src/ES_ModulesDev.cc ?

No, the behavior is intentional (at least for now). We want the non-Alpaka modules to be able to consume the host-side EventSetup data products (analogously to host-side Event data products). From there it follows that the symbols of the host-side data products must be in the non-Alpaka library of RecoTracker/LST, and the .cc file containing the TYPELOOKUP_DATA_REG() for the host-specific data product must reside there.

5 changes: 1 addition & 4 deletions RecoTracker/LST/test/LSTAlpakaTester.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,8 @@
)

process.options = cms.untracked.PSet(
FailPath = cms.untracked.vstring(),
IgnoreCompletely = cms.untracked.vstring(),
Rethrow = cms.untracked.vstring(),
SkipEvent = cms.untracked.vstring(),
accelerators = cms.untracked.vstring('*'),
allowUnscheduled = cms.obsolete.untracked.bool,
canDeleteEarly = cms.untracked.vstring(),
Expand Down Expand Up @@ -91,8 +89,7 @@
# Main LST Producer
process.load('Configuration.StandardSequences.Accelerators_cff')
process.load("HeterogeneousCore.AlpakaCore.ProcessAcceleratorAlpaka_cfi")
from RecoTracker.LST.lstProducer_cfi import lstProducer as _lstProducer
process.lstProducer = _lstProducer.clone()
process.load("RecoTracker.LST.lstProducer_cff")

# Track Fitting
import RecoTracker.TrackProducer.TrackProducer_cfi
Expand Down