Then this is a major design mistake of the mentioned patch if it cannot be removed seamlessy. It is not an Os issue, but a major issue of the program.

There *are*, for example, patches that are well enough designed to be removed by debugging tools if needed.

Greetings,
Thomas

Falls man einen Patch nicht mehr entfernen kann, ist der Patch defekt, nicht das Betriebssystem.

Die Architektur der Speicherverwaltung von exec ist dokumentiert, und diese Dokumente sind auch zugänglich. Speicherdebugger wie MuGA nutzen diese *dokumentierten* Strukturen, um sich in das System zu setzen. Wenn ein Patch diese Interfaces nicht bedient und diese dokumentierten Strukturen nicht benutzt, ist das nicht der Fehler des Os oder des Debuggers, sondern des Patches.

Man kann nicht die exec-Speicherverwaltung umkrempeln und hoffen, dass alles so funktioniert wie bisher.

Falls es darum geht, die Speicherverwaltung von exec zu beschleunigen, so gibt es genug Programme, die dies machen, *ohne* die bereits bestehenden Strukturen zu beschädigen, und auch solche, die sich entfernen lassen. Man muss sie nur probieren wollen. "PoolMem" fällt mir da sofort ein.

Grüße,
Thomas

*** actually, I thought of writing a patch to exec to throw out the old routines completely and have it in from the beginning, but the benefit is small compared to a slightly later activation directly after the exec RomTag.

TLSFMem does what it does. It is purposely not meant to be compatible with tools that assume a certain layout of exec's memory management, even if it is said to be "documented". TLSFMem is compatible on the "application layer", conforms to what is officially documented in the autodocs, and may fail with low-level debugging tools. That's written in its docs and any discussion about it is moot. The only one who assumes that changing the memory management that radically without impact is Bernd, and this is his own personal problem. And nobody is impelled to use TLSFMem if the features do not fit his preferences.

I'm offering you to send you the source of TLSFMem to do whatever you like -- except giving it to Bernd, because I do not believe in his abilities.

And Bernd, please, stop whining. I (and a lot of other people) can't hear the same old song anymore.

Anyway, this is pretty much off-topic. Sorry.

Everyone and everything needs TLSFmem.

Example: YAM

YAM requires TLSFmem.

> But I use YAM without TLSFmem.

Yes but u have very few messages.

I have 10,000 messages in my YAM and YAM stopped working due to ridiculous memory fragementation.

But using the new TLSFmem allocator it runs "ok".

All MUI software, all linux software and all software in general that suffers from "malloc()/free() psychosis" requires TLSFmem to run reliably and quickly.

I assume that NetSurf web browser must do 100x more malloc()/free() than YAM (Yet Another Mailer).

I'm not kidding. That kinna stuff is documented.

And if you cannot get rid of it so the Os cannot continue going in a regular manner and allow tools to work with its structures in a regular manner, then that's broken to begin with.

Again, and even if I sound like a broken record, you can do that *without* breaking compatibility, and *without* having to replace exec structures.

So long,
Thomas

I have a dream, of one day using a new MuGuardianAngel without having to constantly reset my computer to eliminate the massive memory fragmentation that is wrought by the old MuGuardianAngel.

I have a dream, of being able to recruit new hardcore betatesters who are allowed to use NewMuGuardianAngel with modern TLSFmem allocator still active.

I have a dream, of not fragging out my playtesters ram every time they play my game under MuGuardianAngel.

I have a dream, of not fragging out NetSurf betatester's ram every time they test NetSurf under MuGuardianAngel.

So I decided to leave the pointer to the linked free list to NULL. Other than that, the data in memory header is fully backward compatible with the old exec.

In AmigaOS4, I think, the private exec structures are even more private now and especially the memory allocator is less backwards compatible than what TLSFMem provides. MorphOS comes with a TLSF allocator by default since V2.0.

Natami seems to be a project that has the efficiency as the main goal. Why sticking to a linked list to be compatible with some Developer Debugging Tools that *no user* will ever run in productivity is so important to you is just beyond me.

(And yes, that broken record metaphor fits.)

TLSFMem will still be O(1) regardless of the slow bitfield instructions. AFAIR per allocation, about 2-3 bfffo (find first one) have to be executed, which is rather not worth mentioning.

on the other hand if programs debugged on tlsf with some future version of muforce could still fail on original mem allocator that would not be much of help. but i do not suppose so?

so generally i would support the wish to have tlsf compatible with the debuging tools, but if there is another better solution, go on with it.

Need debugging tools not anyway a MMU and dont work on 68050(Cyberguard, Enforcer)?

Team Chaos Leader wrote:

Everyone and everything needs TLSFmem.     Example: YAM

Its to possible to avoid alot of Memallocations and Deallocations
in Programms.
If you use TLSFMem with normal programms, they dont
benefit. With programms that do alot allocs and defragment
the Memory you recognize that.
I use it first time with cmp.020(compare Files in 2 Dirs bit by bit), it was faster with TLSFmem.
But the results of cmp was wrong, it shows defective Files.
I burn the DVD again and it shows again defective Files, that
isnt possible. I try it without TSLFmem, results are right no errors. That means cmp or the CacheCDFS Filesystem wont work
with TSLF. I stop using it, my data is to importend.

Its better to leave the user the choice.

hope you understand what i mean
   
    should speedup allocmem too or only allocpooled
   
    is it better to use also on poolmem for every pool create the cache or used a shared cache.
   
    use for every pool a cache have more overhead

Besides, the point is also that I cannot remove the patch and bring the system back into its original state.

Technical:

a) Do we agree that the speed while *debugging* programs is irrelevant?

b) MuGA does not *patch* the exec memory handler, it *is* an exec memory handler - AllocMem,FreeMem,AllocAbs,... are all entirely replaced by MuGA.

c) Do we agree that, provided there are N patches out there, N different versions of memory debugging tools would be required to address all of its different, potentially incompatible structures?

d) Do we agree that there is one *documented* state of the memory system, namely the documented one?

e) Do we agree that we need only *one* version of each debugging tool or memory patch given all tools *start* in the Os stage, and all tools *end* in the Os state when terminated?

f) Do we agree that the author of a patch is most competent for writing a tool to remove the patch again?

g) Do we now *finally* come to a conclusion that it is up to you (and not me) to take that task?

To make this clear again, see point b). MuGA doesn't care about *how* you allocate memory. In fact, it will never use any of exec routines to allocate memory. It will provide its own. It *only* requires the system in a well defined state, and that is the *documented* state. Not any other.

All tools I write can be installed, and also cleanly uninstalled. MuGA can, PoolMem can. That makes them interoperable. It is not me where interoperability fails.

So long,
Thomas

Christian Kummerow wrote:

Its to possible to avoid alot of Memallocations and Deallocations   in Programms.

If programs are written in a proper way, they will allocate memory from their own memory pool. Actually, this is the default memory allocation for SAS/C "malloc()" anyhow, so I wonder why people don't make use of that in first place. Layers, being a major player in the fragmentation game, got smarter memory allocators in V45 as well, so that hole also got plugged. ViNCEd can deal with memory pretty smart, it will even move its internal buffers in memory to defragment itself.

There are many things that can be done by a competent author. The simplest is often the best: Use malloc(), not AllocMem(). That alone will already help a lot.

So long,
Thomas

B) But there is a difference between TSLF and MuGA.  To be better, TSLF has a different structure and cannot maintain the old one.  Interoperability with the old state is obviously lost here.  TSLF is clearly better in performance and efficiency.  So some, like myself, think it is worth the trade offs.

C) Yes

D) No.  Now there are two, the antique state that the OS was abandoned in, and a vastly better one, TSLF.  What ought to happen is to have one again, namely TSLF.

E) For each state, yes.

F) That would be nice.  :)

G) No.  What sounds better is for there to be a TSLF-MuGA.  IMO,  TSLF should not be compromised simply because it should be the new standard, even with the loss of interoperability and some compatibility.

H) I think we agree that interoperability does not fail with you.  So join the TSLF bandwagon and be interoperable with it!  :)

If there are some concerns over certain programs, maybe what can happen is this:  TSLF is the new standard.  But there is a sandbox under the new way in place that uses the old way in which the program in question can be run, debugged, or whatever.

To give you an idea what's all happening there if you allocate memory:

a) the memory request is checked for consistency,
b) The MMU is reprogrammed,
c) the memory is allocated,
d) it is checked whether the parts of the memory that could not be protected by the MMU still contain the valid "magic cookie".
e) a pre- and postguard is created, including the information the allocating program
f) the memory is filled by another magic cookie (or zero'd out if MEMF_CLEAR).

I hope it is enough to ensure you that the time spend in c) is actually pretty irrelevant compared to all other steps.

One Thousand wrote:

B) But there is a difference between TSLF and MuGA.  To be better, TSLF has a different structure and cannot maintain the old one.  Interoperability with the old state is obviously lost here.  TSLF is clearly better in performance and efficiency.  So some, like myself, think it is worth the trade offs.

And why does that stop Chris writing code to remove the patch again? Obviously, the patch knows which memory is free and which isn't. Thus, free memory can obviously be represented again in a freelist.

And, how come that *I can be better than exec* as well without loosing the compatibility?
 

One Thousand wrote:

D) No.  Now there are two, the antique state that the OS was abandoned in, and a vastly better one, TSLF.  What ought to happen is to have one again, namely TSLF.

And then three, four, and, hey, here's another cool one... No thanks. I don't work like that because it is a waste of time. The problem can be addressed by making the patch smart enough to remove itself, or making a different patch that respects the rules the grandfathers have set.

One Thousand wrote:

G) No.  What sounds better is for there to be a TSLF-MuGA.  IMO,  TSLF should not be compromised simply because it should be the new standard, even with the loss of interoperability and some compatibility.

And here I simply don't agree. You're wasting time in irrelevant micro-optimizations that should better be spend by a good heuristic, and smarter programs using the heap *properly*. Again, I still stand here claiming that you can reach almost similar performance without having to mess with exec.

It is not a secret that on more reasonable operating systems memory allocation is a two-step process. Step one takes memory from a program private heap, using something optimized, but program or library specific. Step two, if that heap runs dry, allocates the memory from the system.

But nevertheless, as programs can (and did) depend on what's documented, it is neither much of a choice you have. And it still doesn't explain why the patch isn't removable.

One Thousand wrote:

If there are some concerns over certain programs, maybe what can happen is this:  TSLF is the new standard.  But there is a sandbox under the new way in place that uses the old way in which the program in question can be run, debugged, or whatever.

Not without a MMU. Execbase contains a list of valid memory, and that list is organized in MemHeaders, which are again organized in chunks. These memheaders are shared for all tasks, as everything is shared under AOS.

Greetings,
Thomas

B) But your plan for a manager is NOT better than TSLF.  You are using the old structure which is slower.  That is the point.  TSLF should be the standard.

D) That is fine it you do not work like that.  Feel free to do your own thing.  :)  Certainly when it comes down to preferences, everyone gets to choose for themselves.

G) Then we disagree on many things here.  When the OS can do what it ought to well, then that means using your time better.  You stand there claiming "almost performance", but you are nowhere near it and have no proof and have to take extra time to work to make it.  O(1) is very hard to beat and is already available, and it would be in the OS itself where you can easily use it without wasting your own time.  You claim this micro-optimization is worthless vs a higher level algorithm, but you are doing the same basic thing in wanting keep the old, slow exec structure.  And btw, I also think one should make smart programs.  Use the best of all worlds.  :)

The explanation of why the patch is not removable?  I do not know.  But I do not care anyway.  I think it should be permanent.  :)

However, I think Chris may have thought you meant TSLF ought to upkeep the old structure while running, not that you meant returning the computer to its old exec state when removed.  You only meant the latter, right?

So you need an MMU for the sandbox. Doesn't MuGA need one too?  What is your point?

One Thousand wrote:

B) But your plan for a manager is NOT better than TSLF.  You are using the old structure which is slower.  That is the point.  TSLF should be the standard.

The main point you seem to forget is that there *is* already a standard, though. And "better" requires a metric (as in "better means what"). I regard a program that doesn't satisfy defined interfaces not as "better" because certain programs simply won't run. That is not better. That is worse.

A second thing you seem for forget is that I don't *plan* a manager. I *do have* a manager. You can download it. Install it. *And* remove it. And it is compatible. So where is your point?

One Thousand wrote:

You stand there claiming "almost performance", but you are nowhere near it and have no proof and have to take extra time to work to make it.

*Sigh* Now again for the fourth time: You *can* download PoolMem. It exists. It works.

One Thousand wrote:

However, I think Chris may have thought you meant TSLF ought to upkeep the old structure while running, not that you meant returning the computer to its old exec state when removed.  You only meant the latter, right?

I only mean the latter. If you want to install a program that depends on the old structures, you need to be able to remove the patch. This is pretty much it. PoolMem comes with its own internal structures, but keeps the "big chunk" memory in the traditional freelist for optimal compatibility. Small memory fragments are maintained in a scratch list pool and are not part of the exec freelist. Thus, compatibility is gained - programs using direct access to the freemem pool continue to work, and programs doing a lot of memory allocations on small chunks will work fast (and again with O(1) due to the scratch list).

Thus: Compatibility and speed. And it can be removed.

One Thousand wrote:

So you need an MMU for the sandbox. Doesn't MuGA need one too?  What is your point?

That this doesn't work. You somehow seem to believe (as many others) that "one process means one MMU configuration". But it doesn't work like this. Memory (under AmigaOs) does not "belong" to a process. It can be passed around, ownership can change without interfering the operating system. It is the total lack of any resource management which is the (main) constructional problem of AmigaOs. It is a blatant design error that can no longer be undone.

So long,
Thomas
 

Would it be possible for anyone to write a new PoolMem, lets call it PoolMemTLSF, that would use the darque majix of PoolMem coding technique along with the brilliant memory allocations of TLSFmem?

Or is there something about the TLSF algorithm that just makes it impossible to ever satisfy the stringent quality-control requirements of Thomas Richter Gmbh? :)

P.S. Natami comes with a real 060 with a real MMU.