 |
Welcome to the Natami / Amiga ForumThis forum is for AMIGA fans interested in the new NATAMI platform.
Please read the forum usage manual.
|
 Do you have ideas and feature wishes? Post them here and discuss your ideas. |
| Stream Decompression | page 1 2 3 4 5 6 7 8
|
|---|
|
|---|
Matt Hey
USA
| |
Posts 733
06 Feb 2011 17:07
| Wojtek P wrote:
|
Cesare Di Mauro wrote:
|
I think it'll be better to have and HAM8/YUV mode instead of the classic HAM8/RGB, since with a YUV color space a pixel transition will be much soften.
Eyes are more sensible to light (gradient) changes, so with an HAM/YUV mode if the first "change command" is related to the Y channel, it'll make quite less artifacts.
May be a test program will prove it.
|
It's better to have 3-plane 8-bit YUV with U,V planes resolution halved.
already proved.
|
"Narrow Channel" YAB sounds similar to what you guys are talking about. It was a fairly simple and early compression method used for textures on the Voodoo 3 for example. Maybe a decompressor would be small enough for our fpga?
"The Voodoo Graphics system provides a form of narrow channel compression that uses a Yab color space based on intensity/chrominance information. The compression is based on an algorithm that compresses a 24-bit RGB value to an 8-bit Yab format with little loss in precision. This Yab compression algorithm is especially suited to texture mapping, as textures typically contain very similar color components. The algorithm is performed by the host CPU, and Yab compressed textures are passed to SST-1. The advantages of using compressed textures are increased effective texture storage space and lower bandwidth requirements to perform texture filtering.
The Yab color space is represented with eight bits per pixel, and, like the GR_TEXTFMT_RGB_332 representation (see ), it allocates specific fields in those eight bits to specific components: four bits for Y and two bits each for a and b. For example, if the mapping from RGB to Yab is accomplished by the following linear matrix transformation,
Y = 0.299*red + 0.587*green + 0.114*blue
a = 0.596*red + 0.275*green + 0.321*blue
b = 0.212*red + 0.523*green + 0.311*blue Equation Set 1
it is called Yiq compression. Two Glide texture formats utilize Yiq compression: GR_TEXTFMT_YIQ_422 and GR_TEXTFMT_AYIQ_8422.
Compression is achieved by quantizing the Y, a, and b space more coarsely than the RGB space (by allocating fewer bits to each channel in Yab space) without degrading the quality of the image substantially. Also, instead of allocating the same number of bits to each channel (as is done when compressing RGB values directly), we can allocate more bits to channels carrying more information, and fewer bits otherwise. For example, when the image is represented in Yiq space with the equations above, it is possib le to allocate only 16 distinct values to Y, which carries the intensity variations in the image, and only 4 distinct values for the i and q channels, which carry the hue information. Hence, the original 24-bit RGB image can be represented in Yiq space wit h only eight bits of information, reducing the space requirements for the texture by a factor of three.
The decompression from Yiq to RGB is the inverse of the compression equations above. The RGB values can be recovered as follows:
red = Y + 0.95*a + 0.62*b
blue = Y – 0.28*a – 0.64*b
green = Y – 1.11*a + 1.73*b Equation Set 2
Implementing these equations in hardware as formulated above is expensive: the Yab components must be scaled and two multipliers per component are needed. In addition, when compressed textures are used in conjunction with bilinear filtering, 24 multiplier s are needed, since four texels must be made available simultaneously. But, by rewriting the equations as vectors (shown below) and building a small lookup table with pre-computed RGB values, the need for multipliers is eliminated, at least in the decompression circuitry.
(red, green, blue) = (Y, Y, Y) + (0.95*a, –0.28*a, –1.11*a) + (0.62*b, –0.64*b, –1.73*b) Equation 3
The four entries in the lookup table for a, then, represent the values of red, green, and blue calculated for four distinct values of a: –256, –85, 85, and 255. And the four entries in the lookup table for b represent the RGB values calculated for four distinct values of b. Y is implemented with a lookup table as well, but with sixteen distinct entries. Note that the quantized values of Y, a, and b can be any four values and don’t necessarily have to be evenly spaced or cover the full range of values.
Note that the Voodoo Graphics hardware will work with any set of similar compression/decompression equations: the constants are contained in the table entries and the mechanics of the decompression are independent of them. The constants in the equations above are the ones used in Yiq space and were chosen to optimize the compression of flesh tones and backgrounds in photographs and videos. Most computer graphics textures, like terrain, sky, building facades, and so on, are not necessarily aligned along the orange-blue and purple-green axes of Yiq space and benefit from a different set of constants. The 3Dfx Interactive TexUS texture utility software provides routines for generating compressed textures using the Yiq equations shown above. It also provides a neural net program that can optimize the choice of factors in the equation for a given texture."
I think gfx/texture compression would be better than a generic disk compression format.
+ Most of the data that needs to be compressed is graphics
+ The data could stay compressed in memory and decompress (if fast enough) as it's copied to a display buffer (saves memory)
+ Less chance of file system problems and less confusing to users and programmers than generic file compression
- Only compresses graphics
- No canned fpga code to use
All the gfx board manufacturers are using texture compression with hardware assist. Generic file compression in hardware is less common.Edit: The forum ate my post again. The link describing how "narrow band" YAB is done is...
EXTERNAL LINK Search for YAB.
| |
SID Hervé
France
| |
Posts 663
06 Feb 2011 18:10
| Thomas Richter wrote:
|
You basically set one specific compression algorithm in stone. However, compression is always source dependent, that is, you can either create a universal, but not well-performing algorithm, or a specialized, but great-for-its-source algorithm. By implementing the algorithm in hardware, you basically freeze the development of better algorithms.
|
Since there are different algorithms more or less specialized, then the type of compression/decompression chosen should be independent of the data to be stored, something generic, whose qualities will be inexorably linked to the speed and simplicity of implementation.Anyway, the use of one (hardware decompression) should not exclude the use of the other (software decompression). Obviously, those are just speculations.
| |
Matt Hey
USA
| |
Posts 733
06 Feb 2011 21:31
| If this NOVA hardware decompression is real time, and the data encoded is only accessed sequentially, perhaps this would be possible...
1) Encode an identification header with compression type and raw data encoded at the beginning. A .nova file extension name would be used too. This would be compressed on disk.
2) Read the file from disk into a generic object like a New_BitMap or New_SoundSample but do NOT decompress. The object would contain the information on the compression type and format of the raw uncompressed data.
3) When the object is used to load, play, etc, the data is decompressed as needed.
As newer compression formats become possible in real time, new compression types would be put in hardware and the older formats would be decoded in software (xad/xfd style). By real time, I mean that the smaller compressed data can be read and decoded in nearly the same time as a larger uncompressed version of the data is read. If not possible or unpractical, then don't do hardware compression at all IMHO.
| |
Thomas Richter
Germany
| | (MX-Board Owner)
Posts 1425
06 Feb 2011 23:33
| Wojtek P wrote:
|
Thomas Richter wrote:
|
compression. This is mostly bitjuggling, searching and table-lookup type of instructions, low-level entropy coding stuff (huffman, arithmetic coding) and then leave it to software to use this type of hardware support to implement specific codecs.
|
Unless i miss something 68k instruction set is rather good in it.
|
"You miss something", then. (-: I guess we had a couple of bit-insert/bit-extract methods discussed in previous threads which would be useful for that and which are not exactly equivalent to the bitfield instructions.Wojtek P wrote:
|
As for bzip2 it's BAD unless you have megabytes of cache.
grzip is too but much less bad and compresses actully far better than bzip2.
|
bzip2 is at least popular enough to make it worth considering. There are many universal compression programs, seeEXTERNAL LINK
for a list of performance tests. Is it worth implementing them all in hardware? If not, why not? If some should be implemented, which, and why? Wojtek P wrote:
|
i REALLY recommend it.
It packs always faster than bzip2, unpack with similar speed, and always packs better.
|
Provably not. In fact, it is easy to see that there is no universal compression program that compresses all of its inputs, and given that the two compression programs are different, it would be easy to construct a stream that can be compressed very well by one of the programs by exactly matching its data model. Probably you mean "on average for typical input", which is a different statement that the one you've made.Anyhow, just from the size of the list Matt compiled one should be very suspicious implementing any of these algorithms in hardware. Implement in hardware what's common to all of them to easy efficient implementations, but that's it. So long, Thomas
| |
Thierry Atheist
Canada
| |
Posts 1828
07 Feb 2011 07:14
| Claudio Wieland wrote:
| | In Natami, we must find ALL possible ways to accelerate the overall system speed, at the lowest possible price in logic resources. Adding an experimental tiny decompressor, doesn't hurt. Even if someone finds it useless, then this is one opinion of many many others. If the majority finds it actually not worthwhile, then we can remove this experimental feature again. As long as it is marked as experimental and people know it could possibly be removed again in the future, they know what they are dealing with. Natami is one great experimenting platform, for a major part. If we don't do experimental things here and there, |
Yeeeeeesssssss!!!!!!
Claudio Wieland wrote:
| we can stop right now and buy some Intel chips.
|
Nooooooooooooooooooo!!!!!!!!:-DDDD
| |
Wojtek P
Poland
| |
Posts 1597
07 Feb 2011 15:53
| Thomas Richter wrote:
|
"You miss something", then. (-: I guess we had a couple of bit-insert/bit-extract methods discussed in previous threads which would be useful for that and which are not exactly equivalent to the bitfield instructions.
|
how it will help huffman decoder. All you need is normal shifting and table lookup with table of few kB (fits L1 cache)
huffman encoder is even more simple.
bzip2 is at least popular enough to make it worth considering. There are many universal compression programs, see
|
Do you choose programs because of popularity or quality?
At least strange for amiga user.
| |
Thomas Richter
Germany
| | (MX-Board Owner)
Posts 1425
07 Feb 2011 16:30
| Wojtek P wrote:
|
how it will help huffman decoder. All you need is normal shifting and table lookup with table of few kB (fits L1 cache)
huffman encoder is even more simple.
|
Still, you need to do this by hand. Ditto for AC coding. Given that this is a heavy-duty operation required by almost any encoder, it is worth considering adding this to the hardware. Instead of adding some proprietary algorithm.
Wojtek P wrote:
|
Do you choose programs because of popularity or quality?
At least strange for amiga user.
|
I choose a compression program, or a specific format in general, by availibility and portability. It doesn't help me to have a super compressor if all the people I want to sent packaged data do not have a decompressor installed. Whether it compresses a few percent better or worse doesn't matter anymore.
Remember, for example, that most programs on Aminet are lha, even though lha is more or less obsolete these days. Same goes for its successor, lzx. These formats are (were) prominent because they are wide-spread and available, not because they compress so good.
So long,
Thomas
| |
Wojtek P
Poland
| |
Posts 1597
07 Feb 2011 16:37
| Thomas Richter wrote:
| | Still, you need to do this by hand. Ditto for AC coding. Given that this is a heavy-duty operation required by almost any encoder, it is worth considering adding this to the hardware. Instead of adding some proprietary algorithm.
|
Do you want
decode.huffman from,to,howmuchinstruction? As for huffman decoding there isn't much need. As for general it would be fetchits instruction that fetches X bits from memory address addressed bit-precise and increment address
and writebits. anyway - too little gain and too huge complexity IMHO.
I choose a compression program, or a specific format in general, by availibility and portability. It doesn't help me to have a super
|
grzip is fully available and completely portable. I just needed to run windows-targeted gcc to make grzip.exe for someone using windows for example. I didn't even test it - it just worked.
| |
JeeBee McJibble
United Kingdom
| |
Posts 85
07 Feb 2011 18:39
| Two things that would be useful to have hardware acceleration for - crypto (especially SSL) and GZip. Both are used during web browsing, so it would benefit lots of users.
| |
Thomas Richter
Germany
| | (MX-Board Owner)
Posts 1425
07 Feb 2011 18:53
| Wojtek P wrote:
|
Thomas Richter wrote:
|
Still, you need to do this by hand. Ditto for AC coding. Given that this is a heavy-duty operation required by almost any encoder, it is worth considering adding this to the hardware. Instead of adding some proprietary algorithm.
|
Do you want
decode.huffman from,to,howmuch
instruction?
|
Something along those lines. I would need bitstream "I/O" instructions.Wojtek P wrote:
|
As for huffman decoding there isn't much need.
As for general it would be fetchits instruction that fetches X bits from memory address addressed bit-precise and increment address
and writebits.
anyway - too little gain and too huge complexity IMHO.
|
How do you know? Actually, for many codecs that's the place where the encoder or decoder spends most of its time. Thus, a heavy duty part of compressors whose improvement would be beneficial for many users.Wojtek P wrote:
|
grzip is fully available and completely portable. I just needed to run windows-targeted gcc to make grzip.exe for someone using windows for example. I didn't even test it - it just worked.
|
And it's completely unused by anyone but you. It's not integrated in tar, it's not part of major distributions, it is not used by the community. How many archives are that are compressed by zip,rar,gzip and bzip2 compared to grzip or other exotic archive types?Greetings, Thomas
| |
Wojtek P
Poland
| |
Posts 1597
07 Feb 2011 18:56
| Thomas Richter wrote:
|
And it's completely unused by anyone but you. It's not integrated in tar, it's not part of major distributions, it is not used by the community. How many archives are that are compressed by zip,rar,gzip and bzip2 compared to grzip or other exotic archive types?
|
Great future natami user. Warranty of making it useless platform.
EOT from my side.
| |
Thomas Richter
Germany
| | (MX-Board Owner)
Posts 1425
07 Feb 2011 19:34
| Wojtek P wrote:
|
Great future natami user.
|
Indeed. gzip or bzip2 support is incredibly more useful than any other exotic archive formats.So long,
Thomas
| |
Cesare Di Mauro
Italy
| |
Posts 526
07 Feb 2011 19:41
| Even 7-Zip ( EXTERNAL LINK ) is more supported.
| |
SID Hervé
France
| |
Posts 663
07 Feb 2011 21:27
| Matt Hey wrote:
|
If this NOVA hardware decompression is real time, and the data encoded is only accessed sequentially, perhaps this would be possible...
1) Encode an identification header with compression type and raw data encoded at the beginning. A .nova file extension name would be used too. This would be compressed on disk.
2) Read the file from disk into a generic object like a New_BitMap or New_SoundSample but do NOT decompress. The object would contain the information on the compression type and format of the raw uncompressed data.
3) When the object is used to load, play, etc, the data is decompressed as needed.
As newer compression formats become possible in real time, new compression types would be put in hardware and the older formats would be decoded in software (xad/xfd style). By real time, I mean that the smaller compressed data can be read and decoded in nearly the same time as a larger uncompressed version of the data is read. If not possible or unpractical, then don't do hardware compression at all IMHO.
|
That is a possibility but I thought about something less elaborate, ie without distinguishing sign (space saving optimized). Something like only the user data blocks are subject to compression. This would likely have the advantage of preserving how the media is arranged and therefore, authorize any repairs or recovery.
| |
Wojtek P
Poland
| |
Posts 1597
08 Feb 2011 00:51
| Cesare Di Mauro wrote:
|
Even 7-Zip ( EXTERNAL LINK ) is more supported.
|
And even OS/2, not to mention unix and finally windows, is more supported than AmigaOS. So why natami.What a dumb arguments. What more support do you need from compressor than compression program and clearly separated compression library. Integrating with anything takes 15 minutes of coding. with tar - if you would ever use unix tools, you would know to tar cf - |grzip >output.tar.grz
grzcat input.tar.grz|tar xpf -
| |
Cesare Di Mauro
Italy
| |
Posts 526
08 Feb 2011 05:52
| You tell "dumb" to your sister, ok? It seams that you don't understand the meaning of "support" in Thomas words.
| |
Wojtek P
Poland
| |
Posts 1597
08 Feb 2011 07:58
| Cesare Di Mauro wrote:
|
You tell "dumb" to your sister, ok?
It seams that you don't understand the meaning of "support" in Thomas words.
|
Seems i completely don't understand you all. Most importantly why you want Natami.
| |
Thierry Atheist
Canada
| |
Posts 1828
08 Feb 2011 08:08
| Wojtek P wrote:
|
Cesare Di Mauro wrote:
|
You tell "dumb" to your sister, ok?It seams that you don't understand the meaning of "support" in Thomas words.
|
Seems i completely don't understand you all. Most importantly why you want Natami.
|
CLICK HERE
| |
Megol .
| |
Posts 675
08 Feb 2011 08:41
| Wojtek P wrote:
|
Cesare Di Mauro wrote:
|
You tell "dumb" to your sister, ok?
It seams that you don't understand the meaning of "support" in Thomas words.
|
Seems i completely don't understand you all. Most importantly why you want Natami.
|
Your motivation seems to make it as incompatible as possible. There are easier ways to make that: design a three stack processor with 9 bit bytes, Huffman compressed instructions and bit-banged sound and video. Hex floats and middle-endian format are obviously the way to go. Harvard style memory with only indirect access via the storage controller, the later is of course record oriented with built-in parsing of text control characters. The character encoding can be Huffman encoded using the frequency table of the rarest language possible.Like it or not Amiga isn't a universe in itself.
| |
Cesare Di Mauro
Italy
| |
Posts 526
08 Feb 2011 13:02
| Wojtek P wrote:
|
Cesare Di Mauro wrote:
|
You tell "dumb" to your sister, ok?
It seams that you don't understand the meaning of "support" in Thomas words.
|
Seems i completely don't understand you all. |
That's a your problem.
| Most importantly why you want Natami. |
No, I just want to know what does it mean regards what are we talking about.P.S. I know Natami's goals and I understand why Gunnar & co. want to implement some new things.
| |
|
|
|
|
