char8_t backward compatibility remediation

• Introduction
• Examples
• Proposal
• Remediation strategies
  • Shut this off, shut these all off
  • New keyword and std members
  • Changed return type for std::filesystem::path member functions
  • UTF-8 literal remediation
    • Simple remediation for common scenarios
    • Complex remediation for uncommon scenarios
    • Array initialization
• Formal wording
  • Library wording
  • Annex C Compatibility wording
  • Annex D Compatibility features wording
• References

Introduction

The support for char8_t as adopted for C++20 via P0482R6 ^[P0482R6] affects backward compatibility for existing programs in at least the following ways:

• Introduction of a new char8_t keyword.
• Change of type for u8 character and string literals.
• Introduction of new std::u8string, std::u8string_view, std::u8streampos type aliases and std::mbrtoc8 and std::c8rtomb functions; these names may conflict with existing uses of these names due to ADL or use of using namespace std.
• Change of return type for std::filesystem::path member functions u8string and generic_u8string.

This paper presents a set of remediation strategies for addressing backward compatibility issues as well as a few minor changes to the C++ standard to better facilitate migration to C++20.

Examples

Code	C++17	C++20 with P0482R6	C++20 with this proposal
std::cout << u8'x'; std::cout << u8"text";	Writes a sequence of UTF-8 code units as characters. (mojibake if the execution character encoding is not UTF-8)	Writes an integer or pointer value. (consistent with handling of char16_t and char32_t literals)	Ill-formed. (for all of char8_t, char16_t, and char32_t literals)
std::string s(u8"text");	Constructs a string object with UTF-8 encoded data.	Ill-formed.	Ill-formed.
std::filesystem::path p = ...; std::string s = p.u8string();	Constructs a string object with UTF-8 encoded data.	Ill-formed.	Ill-formed.
std::filesystem::path(u8"filename");	Constructs a path object with a UTF-8 filename.	Ill-formed.	Constructs a path object with a UTF-8 filename.

Proposal

• Add deleted overloads of basic_ostream<char, ...>::operator<< for char8_t character and string types. This avoids the silent and surprising behavior change introduced by P0482R6 ^[P0482R6] that resulted in UTF-8 character being formatted as numeric values and UTF-8 strings being formatted as pointers.
• Add deleted overloads of basic_ostream<char, ...>::operator<< for char16_t and char32_t character and string types. This removes surprising behavior that has been present since C++11; that characters are formatted as numeric values and that strings are formatted as pointers.
• Modify std::filesystem::u8path to accept ranges and iterators with char8_t value types. This allows existing code that passes UTF-8 string literals to remain well-formed.
  u8path(u8"filename"); // Ok; ill-formed following P0482R6 ^[P0482R6].
• Update the __cpp_lib_char8_t feature test macro to reflect proposed changes in library behavior.

Remediation strategies

A single aproach to addressing backward compatibility impact is unlikely to be the best approach for all projects. This section presents a number of options to address various types of backward compatibility impact. In some cases, the best solution may involve a mix of these options.

Shut this off, shut these all off

The simplest possible solution in the short term is to simply disable the new features completely. Clang and gcc will allow disabling char8_t features in both the language and standard library, via a -fno-char8_t option. It is expected that Microsoft and EDG based compilers will offer a similar option.

This option should be considered a short-term solution to enable testing existing C++17 code compiled as C++20 with minimal effort. This isn't a viable long-term option as continued use would complicate interoperation with code that depends on the new features.

New keyword and std members

The lack of a standard char8_t type has prompted some projects to define their own char8_t type alias and corresponding u8string type. For open source projects reviewed by the author, switching to the new standard features is straight forward at the source level, though binary compatibility may be affected. Such projects can retain binary compatibility by continuing to use a type alias, but with a name other than char8_t.

Changed return type for std::filesystem::path member functions

FIXME

UTF-8 literals remediation

Each of these strategies assumes a requirement for continued use of UTF-8 encoded literals with char based types. For most projects, such a requirement is expected to be temporary while the project is fully migrated to C++20. However, some projects may retain a sustained need for such literals. For those projects, the complex remediation approach provides a long-term solution.

Simple remediation for common scenarios

Common uses of u8 literals can be handled in a backward compatible manner through use of reinterpret_cast or by adding new function overloads. Note that use of reinterpret_cast is ok in these situations since lvalues of type char may be used to access values of other types.

This approach may suffice when there are just a few uses of UTF-8 literals that need to be addressed. In general, sprinkling reinterpret_cast all over a code base is not desirable.

const char &r = u8’x';

const char &r = reinterpret_cast<const char &>(u8’x');

const char *p = u8"text";

const char *p = reinterpret_cast<const char *>(u8"text");

template<int N>
int ft(const char(&)[N]);



ft(u8"text");

template<int N>
int ft(const char(&)[N]);
template<int N>
int ft(const char8_t(&)[N]);

ft(u8"text");

int operator ""_udl(const char*, unsigned long);


int v = u8"text"_udl;

int operator ""_udl(const char*, unsigned long);
int operator ""_udl(const char8_t*, unsigned long);

int v = u8"text"_udl;

Complex remediation for uncommon scenarios

The techniques applied here also apply to the common scenerios discussed in the prior section. This approach makes use of P0732 to enable constexpr UTF-8 encoded char based literals using a user defined literal. The example code below defines overloaded character and string UDL operators named _as_char. These UDLs can then be used in place of existing UTF-8 character and string literals.

#include <utility>

template<std::size_t N>
struct char8_t_string_literal {
  static constexpr inline std::size_t size = N;
  template<std::size_t... I>
  constexpr char8_t_string_literal(
    const char8_t (&r)[N],
    std::index_sequence<I...>)
  :
    s{r[I]...}
  {}
  constexpr char8_t_string_literal(
    const char8_t (&r)[N])
  :
    char8_t_string_literal(r, std::make_index_sequence<N>())
  {}
  auto operator <=>(const char8_t_string_literal&) = default;
  char8_t s[N];
};

template<char8_t_string_literal L, std::size_t... I>
constexpr inline const char as_char_buffer[sizeof...(I)] =
  { static_cast<char>(L.s[I])... };

template<char8_t_string_literal L, std::size_t... I>
constexpr auto& make_as_char_buffer(std::index_sequence<I...>) {
  return as_char_buffer<L, I...>;
}

constexpr char operator ""_as_char(char8_t c) {
  return c;
}

template<char8_t_string_literal L>
constexpr auto& operator""_as_char() {
  return make_as_char_buffer<L>(std::make_index_sequence<decltype(L)::size>());
}

constexpr const char &r = u8’x';

constexpr const char &r = u8’x'_as_char;

constexpr const char *p = u8"text";

constexpr const char *p = u8"text"_as_char;

// gcc extension in C++17; standard C++ doesn't permit conversion
// to arrays of unknown bound.
constexpr const char (&r)[] = u8"text";

// Ok in C++20 with P0388R2 [P0388R2]

constexpr const char (&r)[] = u8"text"_as_char;

When wrapped in macros, the above UDL can be used to retain source compatibility across C++17 and C++20 for all known scenarios except for array initialization.

#include <utility>

#if defined(__cpp_char8_t)
#define U8(x) u8##x##_as_char
#else
#define U8(x) u8##x
#endif

constexpr char c = U8('x');
constexpr const char &rc = U8('x');
constexpr const char *ps = U8("text");
constexpr const char (&rac)[] = U8("text"); // Ok (with gcc extension or P0388).

char a[] = u8"text";

FIXME

constexpr char a[] = u8"text";

FIXME