Subject: Mysterious memory allocation

From: Pascal Vaillant <[email-masked]>
Date: Sat, 19 Feb 2000 00:08:43 +0000 (GMT)

Dear all,

I have a problem of memory allocation when using SWI Prolog:
When my program performs some operations (involving file
opening and closing to find terms, and assertions to store
the terms found in the dynamic database), the portion of
RAM used by Prolog grows greedily (After a couple of times,
malloc fails). However, the heap stays approximately the same
size (well, grows no more than expected).

Example, with a complex operation involving many file lookups:

 Before:

------------------------------------------------------------------
?- statistics.
4.79 seconds cpu time for 3,321,435 inferences
6,168 atoms, 1,169 functors, 1,569 predicates, 22 modules, 118,022 VM-codes

                       Limit    Allocated       In use
Heap         :                               1,209,884 Bytes
Local  stack :     2,048,000       12,288          708 Bytes
Global stack :     4,096,000       20,480          880 Bytes
Trail  stack :     4,096,000       12,288          296 Bytes

Yes
------------------------------------------------------------------

 After:

------------------------------------------------------------------
?- statistics.
8.59 seconds cpu time for 3,845,807 inferences
6,180 atoms, 1,169 functors, 1,569 predicates, 22 modules, 118,875 VM-codes

                       Limit    Allocated       In use
Heap         :                               1,277,220 Bytes
Local  stack :     2,048,000       12,288          708 Bytes
Global stack :     4,096,000       20,480          880 Bytes
Trail  stack :     4,096,000       12,288          296 Bytes

Yes
------------------------------------------------------------------

However, at the same time (according to gmemusage), the RAM used
by pl went from 13044 bytes to 36796 bytes. 

Hence my question: does anybody know where Prolog "hides" these
23 Mb that won't appear in the heap and won't be released by
the standard trim_stacks occurring in the toplevel loop after
every operation?

I suspect the choicepoints created while looking into the files
are somehow kept in memory after leaving the file-searching-pre-
dicate. Here is how I look for something in files (the purpose
to keep information in files is not to store too much unnecessary
information in RAM, but obviously in these conditions it is not
worth it...):

------------------------------------------------------------------

find_term_in_file(Searched_Term,File) :-
  open(File,read,Stream),
  find_term_in_stream(Searched_Term,Stream).

find_term_in_stream(Searched_Term,Stream) :-
  find_term(Stream,Searched_Term).

find_term_in_stream(_Searched_Term,Stream) :-
  close(Stream),
  fail.

find_term(Stream,Searched_Term) :-
  find_term_or_end_of_file(Stream,Searched_Term,End_Of_File),
  var(End_Of_File).

find_term_or_end_of_file(Stream,Searched_Term,End_Of_File) :-
  repeat,
  read_term(Stream,Found_Term,[]),
  (Found_Term = Searched_Term;
   Found_Term = end_of_file ->
   !,
   (End_Of_File = yes)).

------------------------------------------------------------------

Perhaps I made some mistake there? (The method above works but it
"burns" a quantity of RAM in average equivalent to 2.5 times the
size of the file being searched). Don't the choicepoints created
by 'repeat' disappear after the exit of 'find_term_or_end_of_file'?

Thanks for any suggestion,

Pascal

--
Pascal Vaillant                                Jägerstraße 10/11
Humboldt Universität zu Berlin                 10117 Berlin (Allemagne)
Lehstuhl Computerlinguistik                    Tel: (+49/30) 20 19 25 54