Welcome to the Natami / Amiga ForumThis forum is for AMIGA fans interested in the new NATAMI platform.
Please read the forum usage manual.
|
 Do you have ideas and feature wishes? Post them here and discuss your ideas. |
| Stream Decompression | page 1 2 3 4 5 6 7 8
|
|---|
|
|---|
Team Chaos Leader
USA
| | (Moderator)
Posts 2094
05 Feb 2011 11:57
| Well if you have agreed to let the compression of files be 100% optional then could you please divert 10 minutes away from insulting everyone and please explain how a programmer, such as myself, would use the hardware compressor/decompressor in my own exe? It would be really really helpful if the hardware compressor was compatible to an existing compressor such as either z.library, lha, or zip. Will you make it compatible to one of those? I use compression/decompression in my AMIGA games. I use an audio compressor for audio, z.library for some stuff and lha for the autosaves. So I know what I am talking about and my comments have value, whether you like it or not ;p
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 13:22
| Team Chaos Leader wrote:
|
my comments have value, whether you like it or not ;p
|
No offence but I wondered of you aim to beat Thierry BS hiscore lately. I'm really looking forward to see again valuable posts from you.
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 13:25
| Team Chaos Leader wrote:
|
It would be really really helpful if the hardware compressor was compatible to an existing compressor such as either z.library, lha, or zip. Will you make it compatible to one of those?
|
A compatible one would best be done in SW.
For this using a maidcore would be useful. But a non-compatible could be chosen to be a lot simpler to do in HW than zip and therefore be real cheap in a FPGA. For sich an algorithm we could choose to use LE for it.
| |
Deep Sub Micron
Germany
| | (MX-Board Owner)
Posts 567
05 Feb 2011 13:51
| Gunnar von Boehn wrote:
|
Wojtek P wrote:
|
And it will be a ballast for future versions that have to be kept as is.
|
Thanks for your opinions.
But none of you has a clue what he talks about.
|
Is it really necessary to write such sentences?
That way you will probably soon get similar responses.Gunnar von Boehn wrote:
|
None of you knows how much space this would take in an FPGA.
So what value have your comment? None!
|
The same is true for your response. But there was a valid point.It is known fact that if a feature is added to a system then some software will use it and rely on it. Once that happened all following systems need a way to provide the same feature.
I think at least for the beginning a decompression unit must be marked in the programmer reference as experimental and optional.
Problem solved. Next!
| |
Deep Sub Micron
Germany
| | (MX-Board Owner)
Posts 567
05 Feb 2011 13:59
| Gunnar, can you explain the difference between the HW decompression you propose and the most similar standard (de-)compression algorithm.
And don't forget to show which difference is necessary to simplify hardware implementation. Remember ZIP is not a single algorithm it is a set of different compression methods. Maybe a subset or even just one method is similar enough. Please also compare to deflate: EXTERNAL LINK
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 14:09
| deep sub micron wrote:
|
Gunnar von Boehn wrote:
|
Wojtek P wrote:
|
And it will be a ballast for future versions that have to be kept as is.
|
Thanks for your opinions.
But none of you has a clue what he talks about.
|
Is it really necessary to write such sentences?
|
We had some good topic here in the forum lately - but some got ruined by some people posting a lot of nonsense. I find this a rather shame!
If this continues I will not post any news or explanations here anymore.
I really hope for a change in behaviour in the forum.
I think we can improve our way of discussing and I hope that doing sensible brainstorming will make again sense in the forum.
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 14:19
| deep sub micron wrote:
|
Gunnar, can you explain the difference between the HW decompression you propose and the most similar standard (de-)compression algorithm.
|
Sure. I think there are two options for acceleration. One option would be using a 68050 core dedicated to this job.
This core can of course support any algorithm.
But a full 68K core needs quite some FPGA space. The other option is using a dedicated unit.
There are many compression algorithms some are optimised for really simple decompression. If you use a dedicated unit you can pick one of those simple=cheap algorithm. Lets say we pick NOVA. Nova is real simple to decompress.
You could add the NOVA decompressor as option to the IDE DMA unit, this would only add minor extra cost to the IDE channel. NOVA compresses about as good as Powerpacker. To give an example:
Lets say your FPGA has 55000 LE.
Lets say your IDE DMA channels costs 250 LE.
Lets say for 200 LE more you get a channel which can optionally decompress on the flight. This is a reasonable option. And the costs are neglect able.
| |
Deep Sub Micron
Germany
| | (MX-Board Owner)
Posts 567
05 Feb 2011 14:27
| Gunnar von Boehn wrote:
|
I think we can improve our way of discussing and I hope that doing sensible brainstorming will make again sense in the forum.
|
Brainstorming is usually done in two phases:
- collecting ideas - even silly onces - do not comment on ideas
- filtering good stuff outAct as role model !
| |
Thierry Atheist
Canada
| |
Posts 1828
05 Feb 2011 15:04
| PowerPacker on the highest setting was real good at compressing text files.... But it wasn't fast. Of course, I only ever used it at 7.12 MHz. If you copy a file from DF0: to DF1: or say to RAM:, does it decompress, then compress again??? The MOVE command could also experience that. RENAME also is able to move files to another (sub)directory.
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 15:41
| Thierry Atheist wrote:
|
PowerPacker on the highest setting was real good at compressing text files.... But it wasn't fast. Of course, I only ever used it at 7.12 MHz.
If you copy a file from DF0: to DF1: or say to RAM:, does it decompress, then compress again??? The MOVE command could also experience that. RENAME also is able to move files to another (sub)directory.
|
We are talking here only about decompression acceleration not compression. Whether you actually compress anything or not is your decision.
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 15:51
| You have to see the acceleration as option.
Example:
Lets say we create a AROS or AMIGA OS X distribution.
Lets say the SYS partition has a size of 300 MB.
Now we ONCE compress this distribution.
= 150 MB saved, double loading speed of the system and the workbench.
2nd Example:
Our Shoot Em Up game currently has a size of a bit over 200MB.
I assume in the final version it will be around 500MB.
The 500 MB will probably compress down to 200 MB.
300 MB space saved and loading speed more than doubled.
3rd Example:
Lets say the game data for one game level is 300 MB.
Lets say your drive supports 20MB/sec.
This means a level will load in about 15 seconds.
With streaming compression this could go down to 7 seconds.
Nice speedup.
| |
Wojtek P
Poland
| |
Posts 1597
05 Feb 2011 17:18
| Gunnar von Boehn wrote:
|
Team Chaos Leader wrote:
|
It would be really really helpful if the hardware compressor was compatible to an existing compressor such as either z.library, lha, or zip. Will you make it compatible to one of those?
|
A compatible one would best be done in SW.
For this using a maidcore would be useful.
But a non-compatible could be chosen to be a lot simpler to do in HW than zip and therefore be real cheap in a FPGA.
|
Software only unzip is very fast.
| |
Wojtek P
Poland
| |
Posts 1597
05 Feb 2011 17:19
| deep sub micron wrote:
|
Gunnar von Boehn wrote:
|
Wojtek P wrote:
|
And it will be a ballast for future versions that have to be kept as is.
|
Thanks for your opinions.
But none of you has a clue what he talks about.
|
Is it really necessary to write such sentences?
That way you will probably soon get similar responses.
Gunnar von Boehn wrote:
|
None of you knows how much space this would take in an FPGA.
So what value have your comment? None!
|
The same is true for your response. But there was a valid point.
It is known fact that if a feature is added to a system then some software will use it and rely on it. Once that happened all following systems need a way to provide the same feature.
I think at least for the beginning a decompression unit must be marked in the programmer reference as experimental and optional.
Problem solved. Next!
|
Cool feature that nobody will use.
Because it's USELESS.
| |
Wojtek P
Poland
| |
Posts 1597
05 Feb 2011 17:25
|
I think there are two options for acceleration.
One option would be using a 68050 core dedicated to this job.
This core can of course support any algorithm.
But a full 68K core needs quite some FPGA space.
|
extra CPU is good. you MAY use it for unpacking, or for whatever you need.
This extra CPU don't need to be cache coherent with first actually.
And it may be vastly simplified. No interrupts, no instructions that you keep just to be compatible (like BCD operations), no floating point etc.
[qoote]
The other option is using a dedicated unit.
There are many compression algorithms some are optimised for really simple decompression. If you use a dedicated unit you can pick one of those simple=cheap algorithm. Lets say we pick NOVA. Nova is real simple to decompress.
You could add the NOVA decompressor as option to the IDE DMA unit, this would only add minor extra cost to the IDE channel. NOVA compresses about as good as Powerpacker.
To give an example:
Lets say your FPGA has 55000 LE.
Lets say your IDE DMA channels costs 250 LE.
Lets say for 200 LE more you get a channel which can optionally decompress on the flight.
|
How much LEs do compressor require?Once again it doesn't make sense to have only unpacker. Amiga programs doesn't take gigabytes and terabytes. With unpacker&packer it may sense. but not fixed to disk channel!
Hardware packer/depacker is useful for example with network or even in memory-memory operations sometimes if fast enough.
|
|
Wojtek P
Poland
| |
Posts 1597
05 Feb 2011 17:29
| Gunnar von Boehn wrote:
| |
We are talking here only about decompression acceleration not compression.
|
And that's exactly the reason i call this idea useless.
unzipping can be done faster than 10MB/s on fast 486.
much faster on n68k.
Compression can not. What you really have to accelerate is compression.Or at least provide decompressor for STRONG compression like gzip or better - so all games, programs etc. can have data compressed by programmer and take less space with no overhead. It may speed up large games a bit and reduce usage. gzip is NOT complex to decompress.
| |
Wojtek P
Poland
| |
Posts 1597
05 Feb 2011 17:32
| deep sub micron wrote:
|
Gunnar, can you explain the difference between the HW decompression you propose and the most similar standard (de-)compression algorithm.
And don't forget to show which difference is necessary to simplify hardware implementation.
Remember ZIP is not a single algorithm it is a set of different |
ZIP itself is file format, deflate is the compression algorithm we have to care about, ignoring the rest.gzip is implementation of deflate compressor but NOT file format.
you pack one stream to another stream with gzip. You have to use other program to put many files in it. Deflate algorithm is slow to compress (but not that bad) and very fast to unpack.
n68k have 32KB cache, deflate use codes for repetitions up to 32kB back - so it fits well into n68k implementation.
| |
Wojtek P
Poland
| |
Posts 1597
05 Feb 2011 17:34
| Gunnar von Boehn wrote:
|
You have to see the acceleration as option.
Example:
Lets say we create a AROS or AMIGA OS X distribution.
Lets say the SYS partition has a size of 300 MB.
Now we ONCE compress this distribution.
= 150 MB saved, double loading speed of the system and the workbench.
|
Why you ignore seek time?
2nd Example:
Our Shoot Em Up game currently has a size of a bit over 200MB.
I assume in the final version it will be around 500MB.
The 500 MB will probably compress down to 200 MB.
300 MB space saved and loading speed more than doubled.
|
What incredibly good lossless compressor can compress graphics data so well??
| |
SID Hervé
France
| |
Posts 663
05 Feb 2011 19:01
| Hello Slightly off topic since the initial discussion focuses on hardware decompression: I wonder if the advent of the processor with multiple cores would not make uninteresting the problem of compression.
| |
Gunnar von Boehn
Germany
| | (Moderator)
Posts 5775
05 Feb 2011 19:53
| Wojtek P wrote:
|
deep sub micron wrote:
|
Gunnar, can you explain the difference between the HW decompression you propose and the most similar standard (de-)compression algorithm.
And don't forget to show which difference is necessary to simplify hardware implementation.
Remember ZIP is not a single algorithm it is a set of different |
ZIP itself is file format, deflate is the compression algorithm we have to care about, ignoring the rest.
|
Wojtek,
Let me tell you that we work on projects which have a deflate VHDL engine.
So we know exactly about what we speak here.As I have the impression that your posts were not posted with serious interest I think it makes not point to answer them.
| |
Team Chaos Leader
USA
| | (Moderator)
Posts 2094
05 Feb 2011 20:02
| I vote for having a second 68070 to do the compression/decompression. - It uses a ton of LE and SRAM + It is 100,000x more flexible.
+ Codecs can be changed at will
+ Multiple codecs can be used. Specialized codec for audio, a 2nd for text and exes, a 3rd for gfx, etc.
+ Makes excellent use of all Jens' 100s of hours of hard work debugging. You have a great codeslave. Why not use him? :D Most of the time this unit will be unused. While it is not being used for compression/decompression it can be used for any other thing imaginible. A hardcoded FPGA codec would be very kewl but it is just so unflexible. We get stuck with just 1 algorithm. When we discover a better algorithm in the future? Oh well, we are stuck with an old hardcoded algorithm. :( So I vote for using a 2nd 68070 to do the job. Like Wojtek said, we can leave out some lame instructions such as BCD, if that helps.
And I am sure that in the first Natami we would probably have to greatly reduce its cache size. But a few years down the road when better FPGA's are available then we can increase the cache fpr faster performance and 100% compatibility.
| |
|
