Skip to content

Commit 560ef02

Browse files
authored
GH-46600: [C++][CI] Add job with ARROW_LARGE_MEMORY_TESTS enabled (#49490)
### Rationale for this change Now that we have self-hosted runners with AWS we should test the `ARROW_LARGE_MEMORY_TESTS` on CI. ### What changes are included in this PR? Added new runner for ARROW_LARGE_MEMORY_TESTS. Fix `parquet-writer-test` to generate huge expected page using huge `max_rows_per_page` instead of default. ### Are these changes tested? Yes via CI ### Are there any user-facing changes? No * GitHub Issue: #46600 Authored-by: Raúl Cumplido <[email protected]> Signed-off-by: Raúl Cumplido <[email protected]>
1 parent bbfb242 commit 560ef02

2 files changed

Lines changed: 18 additions & 4 deletions

File tree

.github/workflows/cpp_extra.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,16 @@ jobs:
107107
- image: alpine-linux-cpp
108108
runs-on: ubuntu-latest
109109
title: AMD64 Alpine Linux
110+
- image: ubuntu-cpp
111+
run-options: >-
112+
-e ARROW_CTEST_TIMEOUT=2000
113+
-e ARROW_C_FLAGS_DEBUG="-O1"
114+
-e ARROW_CXX_FLAGS_DEBUG="-O1"
115+
-e ARROW_GANDIVA=OFF
116+
-e ARROW_LARGE_MEMORY_TESTS=ON
117+
-e BUILD_WARNING_LEVEL=PRODUCTION
118+
runs-on: "runs-on=${{ github.run_id }}/family=x8i.2xlarge/volume=80gb/spot=capacity-optimized"
119+
title: AMD64 Ubuntu Large Memory Tests
110120
- image: conda-cpp
111121
run-options: >-
112122
-e ARROW_USE_MESON=ON

cpp/src/parquet/column_writer_test.cc

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1053,8 +1053,10 @@ TEST(TestColumnWriter, LARGE_MEMORY_TEST(WriteLargeDictEncodedPage)) {
10531053
{
10541054
PrimitiveNode::Make("item", Repetition::REQUIRED, Type::INT32),
10551055
}));
1056-
auto properties =
1057-
WriterProperties::Builder().data_pagesize(1024 * 1024 * 1024)->build();
1056+
auto properties = WriterProperties::Builder()
1057+
.data_pagesize(1024 * 1024 * 1024)
1058+
->max_rows_per_page(std::numeric_limits<int64_t>::max())
1059+
->build();
10581060
auto file_writer = ParquetFileWriter::Open(sink, schema, properties);
10591061
auto rg_writer = file_writer->AppendRowGroup();
10601062

@@ -1124,8 +1126,10 @@ TEST(TestColumnWriter, LARGE_MEMORY_TEST(ThrowsOnDictIndicesTooLarge)) {
11241126
{
11251127
PrimitiveNode::Make("item", Repetition::REQUIRED, Type::INT32),
11261128
}));
1127-
auto properties =
1128-
WriterProperties::Builder().data_pagesize(4 * 1024LL * 1024 * 1024)->build();
1129+
auto properties = WriterProperties::Builder()
1130+
.data_pagesize(4 * 1024LL * 1024 * 1024)
1131+
->max_rows_per_page(std::numeric_limits<int64_t>::max())
1132+
->build();
11291133
auto file_writer = ParquetFileWriter::Open(sink, schema, properties);
11301134
auto rg_writer = file_writer->AppendRowGroup();
11311135

0 commit comments

Comments
 (0)