Skip to content

Adding possibility to add a list of files to TRestDataSet #514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
juanangp wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Adding possibility to add a list of files to TRestDataSet #514

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7b15447
Adding possibility to add a list of files to TRestDataSet instead of …
juanangp Apr 10, 2024
c612c9c
Increasing ClassDef version on TRestDataSet
juanangp Apr 10, 2024
0b81d37
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 10, 2024
b788e50
Moving file pattern from string to a vector of string and adding the …
juanangp Apr 12, 2024
6dd9735
Merge branch 'fileListDF' of github.com:rest-for-physics/framework in…
juanangp Apr 12, 2024
8124fa4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Apr 12, 2024
8314255
Addressing error while pushing
juanangp Apr 12, 2024
34f56bf
Merge branch 'fileListDF' of github.com:rest-for-physics/framework in…
juanangp Apr 12, 2024
497dbae
Merge branch 'master' into fileListDF
jgalan Jun 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion source/framework/core/inc/TRestDataSet.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ class TRestDataSet : public TRestMetadata {
/// A list of new columns together with its corresponding expressions added to the dataset
std::vector<std::pair<std::string, std::string>> fColumnNameExpressions; //<

/// List of files to generate the dataSet
std::vector<std::string> fFileList; //!

/// A flag to enable Multithreading during dataframe generation
Bool_t fMT = false; //<

Expand Down Expand Up @@ -181,6 +184,7 @@ class TRestDataSet : public TRestMetadata {
inline auto GetCut() const { return fCut; }
inline auto IsMergedDataSet() const { return fMergedDataset; }

inline void SetFileList(const std::vector<std::string>& fileList) { fFileList = fileList; }
inline void SetObservablesList(const std::vector<std::string>& obsList) { fObservablesList = obsList; }
inline void SetFilePattern(const std::string& pattern) { fFilePattern = pattern; }
inline void SetQuantity(const std::map<std::string, RelevantQuantity>& quantity) { fQuantity = quantity; }
Expand Down Expand Up @@ -209,6 +213,6 @@ class TRestDataSet : public TRestMetadata {
TRestDataSet(const char* cfgFileName, const std::string& name = "");
~TRestDataSet();

ClassDefOverride(TRestDataSet, 7);
ClassDefOverride(TRestDataSet, 8);
};
#endif
6 changes: 3 additions & 3 deletions source/framework/core/src/TRestDataSet.cxx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,16 +389,16 @@ std::vector<std::string> TRestDataSet::FileSelection() {
return fFileSelection;
}

std::vector<std::string> fileNames = TRestTools::GetFilesMatchingPattern(fFilePattern);
if (fFileList.empty()) fFileList = TRestTools::GetFilesMatchingPattern(fFilePattern);

RESTInfo << "TRestDataSet::FileSelection. Starting file selection." << RESTendl;
RESTInfo << "Total files : " << fileNames.size() << RESTendl;
RESTInfo << "Total files : " << fFileList.size() << RESTendl;
RESTInfo << "This process may take long computation time in case there are many files." << RESTendl;

fTotalDuration = 0;
std::cout << "Processing file selection.";
int cnt = 1;
for (const auto& file : fileNames) {
for (const auto& file : fFileList) {
if (cnt % 100 == 0) {
std::cout << std::endl;
std::cout << "Files processed: " << cnt << " ." << std::flush;
Expand Down