Skip to content

Replace std::uc namespace with std::ranges and std::views #93

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ednolan merged 1 commit into from
Oct 3, 2025
Merged

Replace std::uc namespace with std::ranges and std::views #93

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion include/beman/utf_view/code_unit_view.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

namespace beman::utf_view {

/* PAPER: namespace std::uc { */
/* PAPER: namespace std::ranges::views { */

namespace detail {

Expand Down
2 changes: 1 addition & 1 deletion include/beman/utf_view/detail/concepts.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

namespace beman::utf_view {

/* PAPER: namespace std::uc { */
/* PAPER: namespace std::ranges { */

template <class T>
using exposition_only_with_reference = T&; // exposition only
Expand Down
18 changes: 11 additions & 7 deletions include/beman/utf_view/to_utf_view.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ namespace detail {

} // namespace detail

/* PAPER: namespace std::uc { */
/* PAPER: namespace std::ranges { */

/* PAPER */

Expand Down Expand Up @@ -1160,6 +1160,8 @@ inline constexpr detail::to_utf_impl<char16_t> to_utf16;

inline constexpr detail::to_utf_impl<char32_t> to_utf32;

/* PAPER: namespace views { */
/* PAPER: */
/* PAPER: template<@*code-unit-to*@ ToType> */
/* PAPER: inline constexpr @*unspecified*@ to_utf; */
/* PAPER: */
Expand All @@ -1168,11 +1170,13 @@ inline constexpr detail::to_utf_impl<char32_t> to_utf32;
/* PAPER: inline constexpr @*unspecified*@ to_utf16; */
/* PAPER: */
/* PAPER: inline constexpr @*unspecified*@ to_utf32; */
/* PAPER: */
/* PAPER: } */
/* PAPER: */
/* PAPER: } */

} // namespace beman::utf_view

/* PAPER: } */

template <class ToType, class V>
inline constexpr bool std::ranges::enable_borrowed_range<
beman::utf_view::exposition_only_to_utf_view_impl<ToType, V>> =
Expand All @@ -1194,16 +1198,16 @@ inline constexpr bool std::ranges::enable_borrowed_range<beman::utf_view::to_utf
/* PAPER: */
/* PAPER: template <class ToType, class V> */
/* PAPER: inline constexpr bool enable_borrowed_range< */
/* PAPER: std::uc::exposition_only_to_utf_view_impl<ToType, V>> = enable_borrowed_range<V>; */
/* PAPER: std::exposition_only_to_utf_view_impl<ToType, V>> = enable_borrowed_range<V>; */

@jwakely jwakely Oct 3, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be std::ranges?

/* PAPER: */
/* PAPER: template <class V> */
/* PAPER: inline constexpr bool enable_borrowed_range<std::uc::to_utf8_view<V>> = enable_borrowed_range<V>; */
/* PAPER: inline constexpr bool enable_borrowed_range<std::to_utf8_view<V>> = enable_borrowed_range<V>; */

@jwakely jwakely Oct 3, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ranges again here

@jwakely jwakely Oct 3, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And two more places below

@ednolan ednolan Oct 24, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, thanks for catching these. It's now moot though, because in the latest revision these lines have been removed since the view is no longer borrowed.

/* PAPER: */
/* PAPER: template <class V> */
/* PAPER: inline constexpr bool enable_borrowed_range<std::uc::to_utf16_view<V>> = enable_borrowed_range<V>; */
/* PAPER: inline constexpr bool enable_borrowed_range<std::to_utf16_view<V>> = enable_borrowed_range<V>; */
/* PAPER: */
/* PAPER: template <class V> */
/* PAPER: inline constexpr bool enable_borrowed_range<std::uc::to_utf32_view<V>> = enable_borrowed_range<V>; */
/* PAPER: inline constexpr bool enable_borrowed_range<std::to_utf32_view<V>> = enable_borrowed_range<V>; */
/* PAPER: */
/* PAPER: } */

Expand Down
78 changes: 42 additions & 36 deletions paper/P2728.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ monofont: "DejaVu Sans Mono"
- Replace code unit views with range adaptor closure objects that are
expression-equivalent to P3117 `transform_view`
- Move `null_sentinel` and `null_term` into P3705
- Remove `std::uc` namespace and replace it with `std::ranges` and `std::ranges::views`

# Motivation

Expand Down Expand Up @@ -169,7 +170,7 @@ code units in sequence may encode a particular code point.

```cpp
std::u32string hello_world =
u8"こんにちは世界" | std::uc::to_utf32 | std::ranges::to<std::u32string>();
u8"こんにちは世界" | std::views::to_utf32 | std::ranges::to<std::u32string>();
```

## Sanitizing potentially invalid Unicode
Expand All @@ -181,15 +182,15 @@ Substitution of Maximal Subparts:
```cpp
template <typename CharT>
std::basic_string<CharT> sanitize(CharT const* str) {
return std::null_term(str) | std::uc::to_utf<CharT> | std::ranges::to<std::basic_string<CharT>>();
return std::null_term(str) | std::views::to_utf<CharT> | std::ranges::to<std::basic_string<CharT>>();
}
```

## Returning the final non-ASCII code point in a string, transcoding backwards lazily:

```cpp
std::optional<char32_t> last_nonascii(std::ranges::view auto str) {
for (auto c : str | std::uc::to_utf32 | std::views::reverse
for (auto c : str | std::views::to_utf32 | std::views::reverse
| std::views::filter([](char32_t c) { return c > 0x7f; })
| std::views::take(1)) {
return c;
Expand Down Expand Up @@ -259,14 +260,14 @@ void change_playing_card_suits() {
Let's say that we want to take code points that we got from ICU, and transcode
them to UTF-8. The problem is that ICU's code point type is `int`. Since `int`
is not a character type, it's not deduced by `to_utf8` to be UTF-32 data. We
can address this by using the `std::uc::as_char32_t` to cast the `int`s to
can address this by using the `std::views::as_char32_t` to cast the `int`s to
`char32_t`:

```cpp
std::vector<int> input = get_icu_code_points();
// This is ill-formed without the as_char32_t adaptation.
auto input_utf8 =
input | std::uc::as_char32_t | std::uc::to_utf8 | std::ranges::to<std::u8string>();
input | std::views::as_char32_t | std::views::to_utf8 | std::ranges::to<std::u8string>();
```

# Proposed design
Expand Down Expand Up @@ -299,7 +300,7 @@ implementation-dependent.

### Rejecting ranges of `char` and `wchar_t`
```c++
using namespace std::uc;
using namespace std::views;

auto v1 = u8"text" | to_utf32; // Ok.
auto v2 = u"text" | to_utf8; // Ok.
Expand All @@ -318,7 +319,7 @@ auto v10 = std::wstring | as_charN_t | to_utf8; // Ok.

### Accepting ranges of `char` and `wchar_t`
```c++
using namespace std::uc;
using namespace std::views;

auto v1 = u8"text" | to_utf32; // Ok.
auto v2 = u"text" | to_utf8; // Ok.
Expand Down Expand Up @@ -420,7 +421,7 @@ practice.
Rejecting `char` and `wchar_t` for UTF transcoding will therefore have
limited benefits. On the other hand, rejecting these types will send users
over to Stack Overflow to discover they need to copy boilerplate called
`| std::uc::as_char8_t` for reasons that will seem academic to most of them.
`| std::views::as_char8_t` for reasons that will seem academic to most of them.

## Error handling mechanism

Expand All @@ -441,8 +442,8 @@ The UTF transcoding views in this paper provide such a basis operation by
adding an `success()` member function to the iterator of the transcoding view,
which informs users whether the current code point is a U+FFFD that was
inserted in response to an invalid code unit sequence. The `success()` member
function returns a `std::expected<void, std::uc::transcoding_error>`, where
`std::uc::transcoding_error` is a new enum class containing enumerators for
function returns a `std::expected<void, std::transcoding_error>`, where
`std::transcoding_error` is a new enum class containing enumerators for
every category of transcoding error.

Users who choose not to implement error handling will simply sanitize any
Expand All @@ -456,12 +457,12 @@ by wrapping the iterator or by iterating with a traditional for loop:
- Producing an error log message
- Collecting statistics on transcoding errors
- Implementing a custom transcoding view whose `value_type` is
`std::expected<charN_t, std::uc::transcoding_error>`
`std::expected<charN_t, std::transcoding_error>`

### Why `std::expected<void, E>`?

The main alternative to consider here would be to specify that
default-constructed `std::uc::transcoding_error` values represent success, or
default-constructed `std::transcoding_error` values represent success, or
add a `success` enumerator whose value is zero. There is precedent for doing
this in the standard in the error handling approach of `std::from_chars`,
which returns a `std::from_chars_result` containing a `std::errc` that has an
Expand Down Expand Up @@ -520,7 +521,7 @@ but not vice versa. See [Appendix: Implementing Existing Practice for Error
Handling](#appendix-implementing-existing-practice-for-error-handling) for
code examples which demonstrate this.

### `std::uc::transcoding_error` enumerators
### `std::transcoding_error` enumerators

- `truncated_utf8_sequence`
- An ill-formed subsequence that matches the beginning of some well-formed
Expand Down Expand Up @@ -4765,7 +4766,7 @@ we can always provide `base()`, we have no trouble returning a
```
template<input_iterator I, sentinel_for<I> S, output_iterator<char8_t> O>
transcode_result<I, O> transcode_to_utf32(I first, S last, O out) {
auto r = ranges::subrange(first, last) | uc::as_utf32;
auto r = ranges::subrange(first, last) | views::as_utf32;

auto copy_result = ranges::copy(r, out);

Expand All @@ -4786,7 +4787,7 @@ standard.

The transcoding views are lazy, as you'd expect. They also compose with the
standard view adaptors, so just transcoding at most 10 UTF-16 code units out
of some UTF can be done with `foo | std::uc::to_utf16 |
of some UTF can be done with `foo | std::views::to_utf16 |
std::ranges::views::take(10)`.

Error handling strategies of the user's choosing can be implemented by the
Expand All @@ -4806,7 +4807,7 @@ produce a replacement character; there is no danger of UB.
## Exposition-only concepts and traits

```c++
namespace std::uc {
namespace std::ranges {

template<class T>
constexpr bool @*is-empty-view*@ = false;
Expand Down Expand Up @@ -4860,7 +4861,7 @@ namespace std::uc {
## Transcoding views

```c++
namespace std::uc {
namespace std::ranges {

enum class transcoding_error {
truncated_utf8_sequence,
Expand Down Expand Up @@ -5276,6 +5277,8 @@ namespace std::uc {
template<class R>
to_utf32_view(R&&) -> to_utf32_view<views::all_t<R>>;

namespace views {

template<@*code-unit-to*@ ToType>
inline constexpr @*unspecified*@ to_utf;

Expand All @@ -5284,22 +5287,25 @@ namespace std::uc {
inline constexpr @*unspecified*@ to_utf16;

inline constexpr @*unspecified*@ to_utf32;

}

}

namespace std::ranges {

template <class ToType, class V>
inline constexpr bool enable_borrowed_range<
std::uc::@*to-utf-view-impl*@<ToType, V>> = enable_borrowed_range<V>;
std::@*to-utf-view-impl*@<ToType, V>> = enable_borrowed_range<V>;

template<class V>
inline constexpr bool enable_borrowed_range<std::uc::to_utf8_view<V>> = enable_borrowed_range<V>;
template <class V>
inline constexpr bool enable_borrowed_range<std::to_utf8_view<V>> = enable_borrowed_range<V>;

template<class V>
inline constexpr bool enable_borrowed_range<std::uc::to_utf16_view<V>> = enable_borrowed_range<V>;
template <class V>
inline constexpr bool enable_borrowed_range<std::to_utf16_view<V>> = enable_borrowed_range<V>;

template<class V>
inline constexpr bool enable_borrowed_range<std::uc::to_utf32_view<V>> = enable_borrowed_range<V>;
template <class V>
inline constexpr bool enable_borrowed_range<std::to_utf32_view<V>> = enable_borrowed_range<V>;

}
```
Expand Down Expand Up @@ -5494,17 +5500,17 @@ since a `utf_view` is empty if and only if its underlying range is empty.
## Add code unit adaptors

```
namespace std::uc {
namespace std::ranges::views {

template<class T>
struct @*implicit-cast-to*@ {
constexpr T operator()(auto x) const noexcept { return x; }
};

inline constexpr @*unspecified*@ as_char8_t;
inline constexpr @*unspecified*@ as_char8_t;

inline constexpr @*unspecified*@ as_char16_t;

inline constexpr @*unspecified*@ as_char32_t;
}
```
Expand Down Expand Up @@ -5723,7 +5729,7 @@ size_t iconv(iconv_t cd, const char** inbuf, size_t* inbytesleft, char** outbuf,
assert(outbuf);
assert(*outbuf);
assert(outbytesleft);
auto view = std::ranges::subrange(*inbuf, *inbuf + *inbytesleft) | std::uc::to_utf32;
auto view = std::ranges::subrange(*inbuf, *inbuf + *inbytesleft) | std::views::to_utf32;
for (auto it = std::ranges::begin(view), end = std::ranges::end(view); it != end;) {
if (it.success()) {
if (*outbytesleft < sizeof(char32_t)) {
Expand Down Expand Up @@ -5826,7 +5832,7 @@ constexpr char16_t* u_strFromUTF8WithSub(
++*pNumSubstitutions;
if (subchar > 0xFFFF) {
std::array<char16_t, 2> subchar_utf16{};
std::ranges::copy(std::array{subchar} | std::uc::to_utf16, subchar_utf16.data());
std::ranges::copy(std::array{subchar} | std::views::to_utf16, subchar_utf16.data());
write(subchar_utf16[0]);
if (destCapacity == 0) {
return dest;
Expand All @@ -5845,9 +5851,9 @@ constexpr char16_t* u_strFromUTF8WithSub(
};

if (srcLength == -1) {
return impl(std::null_term(src) | std::uc::to_utf16);
return impl(std::null_term(src) | std::views::to_utf16);
} else {
return impl(std::ranges::subrange(src, src + srcLength) | std::uc::to_utf16);
return impl(std::ranges::subrange(src, src + srcLength) | std::views::to_utf16);
}
}
```
Expand Down Expand Up @@ -5912,17 +5918,17 @@ constexpr int MultiByteToWideChar(unsigned int CodePage, unsigned long dwFlags,
};
if (cbMultiByte == -1) {
if constexpr (sizeof(wchar_t) == 2) {
return impl(std::null_term(lpMultiByteStr) | std::uc::to_utf16);
return impl(std::null_term(lpMultiByteStr) | std::views::to_utf16);
} else {
return impl(std::null_term(lpMultiByteStr) | std::uc::to_utf32);
return impl(std::null_term(lpMultiByteStr) | std::views::to_utf32);
}
} else {
if constexpr (sizeof(wchar_t) == 2) {
return impl(std::ranges::subrange(lpMultiByteStr, lpMultiByteStr + cbMultiByte) |
std::uc::to_utf16);
std::views::to_utf16);
} else {
return impl(std::ranges::subrange(lpMultiByteStr, lpMultiByteStr + cbMultiByte) |
std::uc::to_utf32);
std::views::to_utf32);
}
}
}
Expand Down