Subject: Re: Strange behaviour with large lists

From: Jan Wielemaker <[email-masked]>
Date: Tue, 17 Oct 2000 09:24:11 +0200

> 	From: Luis Iraola <[email-masked]>
> 
> 	Version 3.4.0 on Windows NT and 98 behaves funny when managing large lists
> 	(10^5 elements) and the same goal is repeated several times.
> 	
> Version 3.3.8 on UNIX appears to have the same bug.
> The first time the query works, and the second time 
> 	ERROR: Out of global stack
> 	ERROR: Out of global stack
> 	% Execution Aborted
> appears.  I did not try any more repetitions.
> Garbage collector stress test?

No, a stress test on recovery from stack overflows.  In the current
implementation GC is only invoked at the call-port and this works
poorly with the example given.  Findall use $record_bag to store
the solutions on-by-one and then $collect_bag/1 to all alternatives
into a list.  The latter is a single call and so no GC is performed
during its execution.  If the list however is 100,000 long it will
push a considerable amount of data on the stacks.

Now what happens?  Each time the stack needs to be expanded, the system
considers to signal a request for GC at the next opportunity.  This
won't help as there is no next opportunity.  Then, if it hits the
stack-limit, it will signal an exception and expand the stack by
one more page to make the exception effective.  The tight foreign
loop pushing data onto the stack however doesn't allow for exception
handling and therefore the stack keeps growing.  The next time it
runs out is won't use the elegant exception route, but the C-longjmp()
route.  Here was a little bug making the cleanup code push more
data onto the stack.  The last resort for stack overflows is an abort.

The real solution is to allow for GC during the execution of foreign
code.  Actually this should be quite feasible.  The only place where
asynchronous GC is really not allowed is at certain points in the
virtual machine as I currently have no means to find out which variables
in the current frame are in use and which contain uninitialised data.
While executing foreign code this problem does not exist.

	Regards --- Jan