我需要解析UTF-8通过输入字符(UTF-8码点,而不是Java的字符)流角色。什么是最好的方法呢?
问题的更新,以使其更清晰(感谢@skomisa):所以下面的非流:
private static String replaceNonBPMWithUnknownCharacter(final String input) {
StringBuilder result = new StringBuilder(input.length());
input.codePoints().forEach((codePoint) -> {
if (isBmpCodePoint(codePoint)) {
result.append('\ufffd');
} else {
result.append(isBmpCodePoint(codePoint) ? toChars(codePoint) : REPLACEMENT_CHAR);
}
});
return result.toString();
}
String result = replaceNonBPMWithUnknownCharacter("\uD83D\uDE0E? X")
我想有一个流版本,e.g:
InputStream stream = replaceNonBPMWithUnknownCharacter(new ByteArrayInputStream("\uD83D\uDE0E? Y".getBytes(UTF_8)))
它使用尽可能少的CPU和内存。下面的问题是类似的,但是非流:qazxsw POI。
最重要的是:我如何从流读取代码点? (所以我怎样可以转换从我知道他们是UTF-8编码来编码点的一个流的字节流)。
首先需要注意的是:
所以总体做法是:
String
以获得其代码点。String.codePointAt()
。List<Integer>
作为码点流。下面的代码,使用用于数据的可变字节长度的一些随机Unicode字符:
List<Integer>
下面是从运行它的输出。它只是列出了返回import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Stream;
import static java.nio.charset.StandardCharsets.UTF_8;
public class Main {
public static void main(String[] args) {
String text = "¢\uD841\uDF31\u30918औWあش";
Stream<Integer> codePoints = Main.processByteStream(new ByteArrayInputStream(text.getBytes(UTF_8)));
codePoints.forEach(System.out::println);
}
/**
* Processes a stream of bytes, and returns a Stream of Unicode codepoints
* associated with the characters derived from that byte stream.
*
* @param bais ByteArrayInputStream to be processed.
* @return A stream of Unicode codepoints derived from UTF-8 characters in the supplied stream.
*/
private static Stream<Integer> processByteStream(ByteArrayInputStream bais) {
int nextByte = 0;
byte b = 0;
byte[] utf8Bytes = null;
int byteCount = 0;
List<Integer> codePoints = new ArrayList<>();
while ((nextByte = bais.read()) != -1) {
b = (byte) nextByte;
byteCount = Main.getByteCount(b);
utf8Bytes = new byte[byteCount];
utf8Bytes[0] = (byte) nextByte;
for (int i = 1; i < byteCount; i++) { // Get any subsequent bytes for this UTF-8 character.
nextByte = bais.read();
utf8Bytes[i] = (byte) nextByte;
}
int codePoint = new String(utf8Bytes, StandardCharsets.UTF_8).codePointAt(0);
codePoints.add(codePoint);
}
return codePoints.stream();
}
/**
* Returns the number of bytes in a UTF-8 character based on the bit pattern
* of the supplied byte. The only valid values are 1, 2 3 or 4. If the
* byte has an invalid bit pattern an IllegalArgumentException is thrown.
*
* @param b The first byte of a UTF-8 character.
* @return The number of bytes for this UTF-* character.
* @throws IllegalArgumentException if the bit pattern is invalid.
*/
private static int getByteCount(byte b) throws IllegalArgumentException {
if ((b >= 0)) return 1; // Pattern is 0xxxxxxx.
if ((b >= (byte) 0b11000000) && (b <= (byte) 0b11011111)) return 2; // Pattern is 110xxxxx.
if ((b >= (byte) 0b11100000) && (b <= (byte) 0b11101111)) return 3; // Pattern is 1110xxxx.
if ((b >= (byte) 0b11110000) && (b <= (byte) 0b11110111)) return 4; // Pattern is 11110xxx.
throw new IllegalArgumentException(); // Invalid first byte for UTF-8 character.
}
}
码点:
Stream<Integer>
笔记:
C:\Java\openJDK\jdk-11.0.2\bin\java.exe -javaagent:C:\Users\johndoe\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\191.4738.6\lib\idea_rt.jar=60544:C:\Users\johndoe\AppData\Local\JetBrains\Toolbox\apps\IDEA-U\ch-0\191.4738.6\bin -Dfile.encoding=UTF-8 -classpath C:\Users\johndoe\IdeaProjects\Codepoint\out\production\Codepoint Main
162
132913
12433
56
2324
87
12354
1588
Process finished with exit code 0
的字节得到一个代码点。调用it is a convoluted process是效率较低,但更清洁的替代方法。String.codePointAt()
,因此引发the replacement character可能是没有必要的。