ByteBufferOutputStream class has a public static variable named
HIGH_PRECISION. By default, it is set to
false. You can set it to
true so that you get 6 decimal places when rounding a number. In Java it will be like:
ByteBufferOutputStream.HIGH_PRECISION = true;
That will cost you some performance (but maybe you'll hardly notice that) and the resulting file will have more bytes (but measurements will be more accurate).
Though my example is in Java, you can easily do the same in C#.
Click this link if you want to see how to answer this question in iText 5.