ICU 64.2
64.2
|
C API: New API for Unicode Normalization. More...
#include "unicode/utypes.h"
#include "unicode/localpointer.h"
#include "unicode/stringoptions.h"
#include "unicode/uset.h"
Go to the source code of this file.
Namespaces | |
icu | |
File coll.h. | |
Typedefs | |
typedef enum UNormalizationCheckResult | UNormalizationCheckResult |
Result values for normalization quick check functions. More... | |
typedef struct UNormalizer2 | UNormalizer2 |
C typedef for struct UNormalizer2. More... | |
Enumerations | |
enum | UNormalization2Mode { UNORM2_COMPOSE, UNORM2_DECOMPOSE, UNORM2_FCD, UNORM2_COMPOSE_CONTIGUOUS } |
Constants for normalization modes. More... | |
enum | UNormalizationCheckResult { UNORM_NO, UNORM_YES, UNORM_MAYBE } |
Result values for normalization quick check functions. More... | |
Functions | |
const UNormalizer2 * | unorm2_getNFCInstance (UErrorCode *pErrorCode) |
Returns a UNormalizer2 instance for Unicode NFC normalization. More... | |
const UNormalizer2 * | unorm2_getNFDInstance (UErrorCode *pErrorCode) |
Returns a UNormalizer2 instance for Unicode NFD normalization. More... | |
const UNormalizer2 * | unorm2_getNFKCInstance (UErrorCode *pErrorCode) |
Returns a UNormalizer2 instance for Unicode NFKC normalization. More... | |
const UNormalizer2 * | unorm2_getNFKDInstance (UErrorCode *pErrorCode) |
Returns a UNormalizer2 instance for Unicode NFKD normalization. More... | |
const UNormalizer2 * | unorm2_getNFKCCasefoldInstance (UErrorCode *pErrorCode) |
Returns a UNormalizer2 instance for Unicode NFKC_Casefold normalization. More... | |
const UNormalizer2 * | unorm2_getInstance (const char *packageName, const char *name, UNormalization2Mode mode, UErrorCode *pErrorCode) |
Returns a UNormalizer2 instance which uses the specified data file (packageName/name similar to ucnv_openPackage() and ures_open()/ResourceBundle) and which composes or decomposes text according to the specified mode. More... | |
UNormalizer2 * | unorm2_openFiltered (const UNormalizer2 *norm2, const USet *filterSet, UErrorCode *pErrorCode) |
Constructs a filtered normalizer wrapping any UNormalizer2 instance and a filter set. More... | |
void | unorm2_close (UNormalizer2 *norm2) |
Closes a UNormalizer2 instance from unorm2_openFiltered(). More... | |
int32_t | unorm2_normalize (const UNormalizer2 *norm2, const UChar *src, int32_t length, UChar *dest, int32_t capacity, UErrorCode *pErrorCode) |
Writes the normalized form of the source string to the destination string (replacing its contents) and returns the length of the destination string. More... | |
int32_t | unorm2_normalizeSecondAndAppend (const UNormalizer2 *norm2, UChar *first, int32_t firstLength, int32_t firstCapacity, const UChar *second, int32_t secondLength, UErrorCode *pErrorCode) |
Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the length of the first string. More... | |
int32_t | unorm2_append (const UNormalizer2 *norm2, UChar *first, int32_t firstLength, int32_t firstCapacity, const UChar *second, int32_t secondLength, UErrorCode *pErrorCode) |
Appends the second string to the first string (merging them at the boundary) and returns the length of the first string. More... | |
int32_t | unorm2_getDecomposition (const UNormalizer2 *norm2, UChar32 c, UChar *decomposition, int32_t capacity, UErrorCode *pErrorCode) |
Gets the decomposition mapping of c. More... | |
int32_t | unorm2_getRawDecomposition (const UNormalizer2 *norm2, UChar32 c, UChar *decomposition, int32_t capacity, UErrorCode *pErrorCode) |
Gets the raw decomposition mapping of c. More... | |
UChar32 | unorm2_composePair (const UNormalizer2 *norm2, UChar32 a, UChar32 b) |
Performs pairwise composition of a & b and returns the composite if there is one. More... | |
uint8_t | unorm2_getCombiningClass (const UNormalizer2 *norm2, UChar32 c) |
Gets the combining class of c. More... | |
UBool | unorm2_isNormalized (const UNormalizer2 *norm2, const UChar *s, int32_t length, UErrorCode *pErrorCode) |
Tests if the string is normalized. More... | |
UNormalizationCheckResult | unorm2_quickCheck (const UNormalizer2 *norm2, const UChar *s, int32_t length, UErrorCode *pErrorCode) |
Tests if the string is normalized. More... | |
int32_t | unorm2_spanQuickCheckYes (const UNormalizer2 *norm2, const UChar *s, int32_t length, UErrorCode *pErrorCode) |
Returns the end of the normalized substring of the input string. More... | |
UBool | unorm2_hasBoundaryBefore (const UNormalizer2 *norm2, UChar32 c) |
Tests if the character always has a normalization boundary before it, regardless of context. More... | |
UBool | unorm2_hasBoundaryAfter (const UNormalizer2 *norm2, UChar32 c) |
Tests if the character always has a normalization boundary after it, regardless of context. More... | |
UBool | unorm2_isInert (const UNormalizer2 *norm2, UChar32 c) |
Tests if the character is normalization-inert. More... | |
int32_t | unorm_compare (const UChar *s1, int32_t length1, const UChar *s2, int32_t length2, uint32_t options, UErrorCode *pErrorCode) |
Compares two strings for canonical equivalence. More... | |
C API: New API for Unicode Normalization.
Unicode normalization functionality for standard Unicode normalization or for using custom mapping tables. All instances of UNormalizer2 are unmodifiable/immutable. Instances returned by unorm2_getInstance() are singletons that must not be deleted by the caller. For more details see the Normalizer2 C++ class.
Definition in file unorm2.h.
typedef enum UNormalizationCheckResult UNormalizationCheckResult |
Result values for normalization quick check functions.
For details see http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms
typedef struct UNormalizer2 UNormalizer2 |
enum UNormalization2Mode |
Constants for normalization modes.
For details about standard Unicode normalization forms and about the algorithms which are also used with custom mapping tables see http://www.unicode.org/unicode/reports/tr15/
Enumerator | |
---|---|
UNORM2_COMPOSE | Decomposition followed by composition. Same as standard NFC when using an "nfc" instance. Same as standard NFKC when using an "nfkc" instance. For details about standard Unicode normalization forms see http://www.unicode.org/unicode/reports/tr15/
|
UNORM2_DECOMPOSE | Map, and reorder canonically. Same as standard NFD when using an "nfc" instance. Same as standard NFKD when using an "nfkc" instance. For details about standard Unicode normalization forms see http://www.unicode.org/unicode/reports/tr15/
|
UNORM2_FCD | "Fast C or D" form. If a string is in this form, then further decomposition without reordering would yield the same form as DECOMPOSE. Text in "Fast C or D" form can be processed efficiently with data tables that are "canonically closed", that is, that provide equivalent data for equivalent text, without having to be fully normalized. Not a standard Unicode normalization form. Not a unique form: Different FCD strings can be canonically equivalent. For details see http://www.unicode.org/notes/tn5/#FCD
|
UNORM2_COMPOSE_CONTIGUOUS | Compose only contiguously. Also known as "FCC" or "Fast C Contiguous". The result will often but not always be in NFC. The result will conform to FCD which is useful for processing. Not a standard Unicode normalization form. For details see http://www.unicode.org/notes/tn5/#FCC
|
Result values for normalization quick check functions.
For details see http://www.unicode.org/reports/tr15/#Detecting_Normalization_Forms
Enumerator | |
---|---|
UNORM_NO | The input string is not in the normalization form.
|
UNORM_YES | The input string is in the normalization form.
|
UNORM_MAYBE | The input string may or may not be in the normalization form. This value is only returned for composition forms like NFC and FCC, when a backward-combining character is found for which the surrounding text would have to be analyzed further.
|
int32_t unorm2_append | ( | const UNormalizer2 * | norm2, |
UChar * | first, | ||
int32_t | firstLength, | ||
int32_t | firstCapacity, | ||
const UChar * | second, | ||
int32_t | secondLength, | ||
UErrorCode * | pErrorCode | ||
) |
Appends the second string to the first string (merging them at the boundary) and returns the length of the first string.
The result is normalized if both the strings were normalized. The first and second strings must be different buffers.
norm2 | UNormalizer2 instance |
first | string, should be normalized |
firstLength | length of the first string, or -1 if NUL-terminated |
firstCapacity | number of UChars that can be written to first |
second | string, should be normalized |
secondLength | length of the source string, or -1 if NUL-terminated |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
void unorm2_close | ( | UNormalizer2 * | norm2 | ) |
Closes a UNormalizer2 instance from unorm2_openFiltered().
Do not close instances from unorm2_getInstance()!
norm2 | UNormalizer2 instance to be closed |
UChar32 unorm2_composePair | ( | const UNormalizer2 * | norm2, |
UChar32 | a, | ||
UChar32 | b | ||
) |
Performs pairwise composition of a & b and returns the composite if there is one.
Returns a composite code point c only if c has a two-way mapping to a+b. In standard Unicode normalization, this means that c has a canonical decomposition to a+b and c does not have the Full_Composition_Exclusion property.
This function is independent of the mode of the UNormalizer2.
norm2 | UNormalizer2 instance |
a | A (normalization starter) code point. |
b | Another code point. |
uint8_t unorm2_getCombiningClass | ( | const UNormalizer2 * | norm2, |
UChar32 | c | ||
) |
Gets the combining class of c.
The default implementation returns 0 but all standard implementations return the Unicode Canonical_Combining_Class value.
norm2 | UNormalizer2 instance |
c | code point |
int32_t unorm2_getDecomposition | ( | const UNormalizer2 * | norm2, |
UChar32 | c, | ||
UChar * | decomposition, | ||
int32_t | capacity, | ||
UErrorCode * | pErrorCode | ||
) |
Gets the decomposition mapping of c.
Roughly equivalent to normalizing the String form of c on a UNORM2_DECOMPOSE UNormalizer2 instance, but much faster, and except that this function returns a negative value and does not write a string if c does not have a decomposition mapping in this instance's data. This function is independent of the mode of the UNormalizer2.
norm2 | UNormalizer2 instance |
c | code point |
decomposition | String buffer which will be set to c's decomposition mapping, if there is one. |
capacity | number of UChars that can be written to decomposition |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
const UNormalizer2* unorm2_getInstance | ( | const char * | packageName, |
const char * | name, | ||
UNormalization2Mode | mode, | ||
UErrorCode * | pErrorCode | ||
) |
Returns a UNormalizer2 instance which uses the specified data file (packageName/name similar to ucnv_openPackage() and ures_open()/ResourceBundle) and which composes or decomposes text according to the specified mode.
Returns an unmodifiable singleton instance. Do not delete it.
Use packageName=NULL for data files that are part of ICU's own data. Use name="nfc" and UNORM2_COMPOSE/UNORM2_DECOMPOSE for Unicode standard NFC/NFD. Use name="nfkc" and UNORM2_COMPOSE/UNORM2_DECOMPOSE for Unicode standard NFKC/NFKD. Use name="nfkc_cf" and UNORM2_COMPOSE for Unicode standard NFKC_CF=NFKC_Casefold.
packageName | NULL for ICU built-in data, otherwise application data package name |
name | "nfc" or "nfkc" or "nfkc_cf" or name of custom data file |
mode | normalization mode (compose or decompose etc.) |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
const UNormalizer2* unorm2_getNFCInstance | ( | UErrorCode * | pErrorCode | ) |
Returns a UNormalizer2 instance for Unicode NFC normalization.
Same as unorm2_getInstance(NULL, "nfc", UNORM2_COMPOSE, pErrorCode). Returns an unmodifiable singleton instance. Do not delete it.
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
const UNormalizer2* unorm2_getNFDInstance | ( | UErrorCode * | pErrorCode | ) |
Returns a UNormalizer2 instance for Unicode NFD normalization.
Same as unorm2_getInstance(NULL, "nfc", UNORM2_DECOMPOSE, pErrorCode). Returns an unmodifiable singleton instance. Do not delete it.
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
const UNormalizer2* unorm2_getNFKCCasefoldInstance | ( | UErrorCode * | pErrorCode | ) |
Returns a UNormalizer2 instance for Unicode NFKC_Casefold normalization.
Same as unorm2_getInstance(NULL, "nfkc_cf", UNORM2_COMPOSE, pErrorCode). Returns an unmodifiable singleton instance. Do not delete it.
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
const UNormalizer2* unorm2_getNFKCInstance | ( | UErrorCode * | pErrorCode | ) |
Returns a UNormalizer2 instance for Unicode NFKC normalization.
Same as unorm2_getInstance(NULL, "nfkc", UNORM2_COMPOSE, pErrorCode). Returns an unmodifiable singleton instance. Do not delete it.
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
const UNormalizer2* unorm2_getNFKDInstance | ( | UErrorCode * | pErrorCode | ) |
Returns a UNormalizer2 instance for Unicode NFKD normalization.
Same as unorm2_getInstance(NULL, "nfkc", UNORM2_DECOMPOSE, pErrorCode). Returns an unmodifiable singleton instance. Do not delete it.
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
int32_t unorm2_getRawDecomposition | ( | const UNormalizer2 * | norm2, |
UChar32 | c, | ||
UChar * | decomposition, | ||
int32_t | capacity, | ||
UErrorCode * | pErrorCode | ||
) |
Gets the raw decomposition mapping of c.
This is similar to the unorm2_getDecomposition() function but returns the raw decomposition mapping as specified in UnicodeData.txt or (for custom data) in the mapping files processed by the gennorm2 tool. By contrast, unorm2_getDecomposition() returns the processed, recursively-decomposed version of this mapping.
When used on a standard NFKC Normalizer2 instance, unorm2_getRawDecomposition() returns the Unicode Decomposition_Mapping (dm) property.
When used on a standard NFC Normalizer2 instance, it returns the Decomposition_Mapping only if the Decomposition_Type (dt) is Canonical (Can); in this case, the result contains either one or two code points (=1..4 UChars).
This function is independent of the mode of the UNormalizer2.
norm2 | UNormalizer2 instance |
c | code point |
decomposition | String buffer which will be set to c's raw decomposition mapping, if there is one. |
capacity | number of UChars that can be written to decomposition |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
UBool unorm2_hasBoundaryAfter | ( | const UNormalizer2 * | norm2, |
UChar32 | c | ||
) |
Tests if the character always has a normalization boundary after it, regardless of context.
For details see the Normalizer2 base class documentation.
norm2 | UNormalizer2 instance |
c | character to test |
UBool unorm2_hasBoundaryBefore | ( | const UNormalizer2 * | norm2, |
UChar32 | c | ||
) |
Tests if the character always has a normalization boundary before it, regardless of context.
For details see the Normalizer2 base class documentation.
norm2 | UNormalizer2 instance |
c | character to test |
UBool unorm2_isInert | ( | const UNormalizer2 * | norm2, |
UChar32 | c | ||
) |
Tests if the character is normalization-inert.
For details see the Normalizer2 base class documentation.
norm2 | UNormalizer2 instance |
c | character to test |
UBool unorm2_isNormalized | ( | const UNormalizer2 * | norm2, |
const UChar * | s, | ||
int32_t | length, | ||
UErrorCode * | pErrorCode | ||
) |
Tests if the string is normalized.
Internally, in cases where the quickCheck() method would return "maybe" (which is only possible for the two COMPOSE modes) this method resolves to "yes" or "no" to provide a definitive result, at the cost of doing more work in those cases.
norm2 | UNormalizer2 instance |
s | input string |
length | length of the string, or -1 if NUL-terminated |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
int32_t unorm2_normalize | ( | const UNormalizer2 * | norm2, |
const UChar * | src, | ||
int32_t | length, | ||
UChar * | dest, | ||
int32_t | capacity, | ||
UErrorCode * | pErrorCode | ||
) |
Writes the normalized form of the source string to the destination string (replacing its contents) and returns the length of the destination string.
The source and destination strings must be different buffers.
norm2 | UNormalizer2 instance |
src | source string |
length | length of the source string, or -1 if NUL-terminated |
dest | destination string; its contents is replaced with normalized src |
capacity | number of UChars that can be written to dest |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
int32_t unorm2_normalizeSecondAndAppend | ( | const UNormalizer2 * | norm2, |
UChar * | first, | ||
int32_t | firstLength, | ||
int32_t | firstCapacity, | ||
const UChar * | second, | ||
int32_t | secondLength, | ||
UErrorCode * | pErrorCode | ||
) |
Appends the normalized form of the second string to the first string (merging them at the boundary) and returns the length of the first string.
The result is normalized if the first string was normalized. The first and second strings must be different buffers.
norm2 | UNormalizer2 instance |
first | string, should be normalized |
firstLength | length of the first string, or -1 if NUL-terminated |
firstCapacity | number of UChars that can be written to first |
second | string, will be normalized |
secondLength | length of the source string, or -1 if NUL-terminated |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
UNormalizer2* unorm2_openFiltered | ( | const UNormalizer2 * | norm2, |
const USet * | filterSet, | ||
UErrorCode * | pErrorCode | ||
) |
Constructs a filtered normalizer wrapping any UNormalizer2 instance and a filter set.
Both are aliased and must not be modified or deleted while this object is used. The filter set should be frozen; otherwise the performance will suffer greatly.
norm2 | wrapped UNormalizer2 instance |
filterSet | USet which determines the characters to be normalized |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
UNormalizationCheckResult unorm2_quickCheck | ( | const UNormalizer2 * | norm2, |
const UChar * | s, | ||
int32_t | length, | ||
UErrorCode * | pErrorCode | ||
) |
Tests if the string is normalized.
For the two COMPOSE modes, the result could be "maybe" in cases that would take a little more work to resolve definitively. Use spanQuickCheckYes() and normalizeSecondAndAppend() for a faster combination of quick check + normalization, to avoid re-checking the "yes" prefix.
norm2 | UNormalizer2 instance |
s | input string |
length | length of the string, or -1 if NUL-terminated |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
int32_t unorm2_spanQuickCheckYes | ( | const UNormalizer2 * | norm2, |
const UChar * | s, | ||
int32_t | length, | ||
UErrorCode * | pErrorCode | ||
) |
Returns the end of the normalized substring of the input string.
In other words, with end=spanQuickCheckYes(s, ec);
the substring UnicodeString(s, 0, end)
will pass the quick check with a "yes" result.
The returned end index is usually one or more characters before the "no" or "maybe" character: The end index is at a normalization boundary. (See the class documentation for more about normalization boundaries.)
When the goal is a normalized string and most input strings are expected to be normalized already, then call this method, and if it returns a prefix shorter than the input string, copy that prefix and use normalizeSecondAndAppend() for the remainder.
norm2 | UNormalizer2 instance |
s | input string |
length | length of the string, or -1 if NUL-terminated |
pErrorCode | Standard ICU error code. Its input value must pass the U_SUCCESS() test, or else the function returns immediately. Check for U_FAILURE() on output or use with function chaining. (See User Guide for details.) |
int32_t unorm_compare | ( | const UChar * | s1, |
int32_t | length1, | ||
const UChar * | s2, | ||
int32_t | length2, | ||
uint32_t | options, | ||
UErrorCode * | pErrorCode | ||
) |
Compares two strings for canonical equivalence.
Further options include case-insensitive comparison and code point order (as opposed to code unit order).
Canonical equivalence between two strings is defined as their normalized forms (NFD or NFC) being identical. This function compares strings incrementally instead of normalizing (and optionally case-folding) both strings entirely, improving performance significantly.
Bulk normalization is only necessary if the strings do not fulfill the FCD conditions. Only in this case, and only if the strings are relatively long, is memory allocated temporarily. For FCD strings and short non-FCD strings there is no memory allocation.
Semantically, this is equivalent to strcmp[CodePointOrder](NFD(foldCase(NFD(s1))), NFD(foldCase(NFD(s2)))) where code point order and foldCase are all optional.
UAX 21 2.5 Caseless Matching specifies that for a canonical caseless match the case folding must be performed first, then the normalization.
s1 | First source string. |
length1 | Length of first source string, or -1 if NUL-terminated. |
s2 | Second source string. |
length2 | Length of second source string, or -1 if NUL-terminated. |
options | A bit set of options:
|
pErrorCode | ICU error code in/out parameter. Must fulfill U_SUCCESS before the function call. |
Referenced by icu::Normalizer::compare().