aboutsummaryrefslogtreecommitdiff
path: root/src/utfstring.h (follow)
AgeCommit message (Collapse)Author
2011-04-08Rearranged the API a bit.Mike Buland
2011-04-07Pretty sure all utf encoders and decoders are complete and tested.Mike Buland
2011-04-04UtfString is going really well. It can now parse Utf8, Utf16 (le,be), andMike Buland
Utf32 (le,be). The internal storage seems to be working fine, although we do have a problem with random access, but at least we can tell which half of a surrogate pair we're on, so we can always rapidly determine the entire code point from any utf16 index that we're on. The only optomization that I'm not doing yet is reading in entire 16bit or 32bit words at a time and converting them from their byte order to native. There are a few potential issues with that, so we'll see. I added a couple of testing datafiles and a test program, I'll delete them all just as soon as it's verified to write correctly.
2011-04-04I made some awesome progress on the UtfString system, it stores in native utf16Mike Buland
encoding to make things easier (little endian in our case). It can currently read utf8 and utf16be, but not BOM. It will give you full unicode code points instead of the raw utf16 values, which is pretty slick.
2011-04-04Really just made some decisions about the overall functionality of the UtfStringMike Buland
and now I'm ready to put some more of the basics into action.
2011-03-22We now have a UTF-8 test parser, I'm going to move it into a functor, I think.Mike Buland
2011-03-18Wow, a lot has changed. String is not a template class, and it can do it's ownMike Buland
formatting ala QString.
2011-01-20Made (very) basic progress towards defining UtfString. It's actually going toMike Buland
use a Bu::String as it's backend storage, so we'll get all the great out of that...
2011-01-20Wow, got the Stream changes propegated, all tests build with string instead ofMike Buland
fstring, and updated the copyright notice to extend to 2011
2011-01-20Bu::FString is now String, and there's a shell script to fix any other programsMike Buland
that were using fstring, I hope.
2010-11-19I now think that this may not work out at all. It looks like if we want properMike Buland
Unicode handling we'll need to implement a series of codecs and converters as well as tables of codepages and lookups. It'll be interesting, I guess, but it makes me care a lot less about proper encoding. Anyway, UtfString uses shorts instead of chars, so it's a step in the right direction, but still not enough to be able to handle proper UTF-16 encoding, maybe UCS-2 encoding, but... ...that's lame. Bu::FBasicString has been generalized a bit with optimizations from libc for char based strings. It also, unfortunately, still uses char-only functions in several places, those all rely on char casting strings at the moment just to get the thing to compile. Basically, it's not a good UTF-16 solution yet, and it may never be and remain compatible with char based strings.