malloc() fails: 32MB heap limit and how to work around?

[Thomas Hoffmann]

I ran into problems with an application that malloc()s memory from
within a DLL.
Now I vaguely remember to have read about problems with using more than
32MB heap
(a Readme for gcc 3.x said:

I can't allocate more than 32 Mb of memory. What to do?

That's an EMX feature. Usually EMX applications are limited to
using a 32MB heap because the fork() logic expects the heap to be
allocated in one
big segment. If you don't use fork(), you can use the _uflags() function
to tell the EMX runtime that you don't want your heap to be contiguous.
)

Then I looked into the EMX docs, they say (for _uflags()):

_UF_SBRK_CONTIGUOUS sbrk() always allocates contiguous memory; the size
of the heap is
limited to the initial heap size. This is the initial setting; however,
the malloc() implementation of emx
selects _UF_SBRK_ARBITRARY

Now I am somewhat puzzled: What ARE the restrictions for heap usage and
HOW can I work around
them?

Thomas.

[Ilya Zakharevich]

Post by Thomas Hoffmann
That's an EMX feature. Usually EMX applications are limited to
using a 32MB heap

Bullocks. EMX programs can use as much memory as is available via
low-memory user memory space.

Post by Thomas Hoffmann
_UF_SBRK_CONTIGUOUS sbrk() always allocates contiguous memory; the size
of the heap is
limited to the initial heap size. This is the initial setting; however,
the malloc() implementation of emx
selects _UF_SBRK_ARBITRARY

Actually, this means that the default for programs which *do not* use
EMX's malloc() (and do not call CRT library entry points which use
EMX's malloc()) is *not* _UF_SBRK_ARBITRARY.

_UF_SBRK_ARBITRARY is set during the first invocation of malloc().
E.g., all Perl (which uses its own malloc()) needs to do is

_uflags (_UF_SBRK_MODEL, _UF_SBRK_ARBITRARY);

Hope this helps,
Ilya

[Thomas Hoffmann]

Okay, so let me rephrase my problem: As soon as I use the -Zcrtdll
option, the program
cannot malloc() more than 32MB memory.

I used the following test program:

#include <stdio.h>
#include <malloc.h>
#include <umalloc.h>

#include <unistd.h>

void us(void) {
_HEAPSTATS hs;

if (0==_ustats(_udefault(NULL),&hs))
printf ("%lu %lu\n",
(unsigned long)(hs._provided), (unsigned long)(hs._used));
else
printf("heapstatus error\n");
}

int main (void)
{
void *p;
int i;
/* Avoid deadlock due to buffer allocation in printf()! */
setvbuf (stdout, NULL, _IOLBF, BUFSIZ);

for (i=0; i<50; i++) {
p=malloc(10000000);
if (NULL==p) printf("%i: malloc failed\n",i);
us();
}
return 0;
}

When compiling with "gcc proggi.c" it exhausts the user memory after 29
mallocs()
...
298516032 290005120
29: malloc failed
301989440 290005120
30: malloc failed
...

When compiling with "gcc proggi.c -Zcrtdll" it cannot malloc more than 32MB:

10092160 10005144
20119168 20005144
30146176 30005144
3: malloc failed
33554048 30005144
4: malloc failed
....

Any clues/ workarounds?

Thomas.

Post by Ilya Zakharevich

Post by Thomas Hoffmann
That's an EMX feature. Usually EMX applications are limited to
using a 32MB heap

Bullocks. EMX programs can use as much memory as is available via
low-memory user memory space.

Post by Thomas Hoffmann
_UF_SBRK_CONTIGUOUS sbrk() always allocates contiguous memory; the size
of the heap is
limited to the initial heap size. This is the initial setting; however,
the malloc() implementation of emx
selects _UF_SBRK_ARBITRARY

Actually, this means that the default for programs which *do not* use
EMX's malloc() (and do not call CRT library entry points which use
EMX's malloc()) is *not* _UF_SBRK_ARBITRARY.
_UF_SBRK_ARBITRARY is set during the first invocation of malloc().
E.g., all Perl (which uses its own malloc()) needs to do is
_uflags (_UF_SBRK_MODEL, _UF_SBRK_ARBITRARY);
Hope this helps,
Ilya
_______________________________________________
Emx mailing list
http://xfreeos2.dyndns.org/mailman/listinfo/emx

[Ilya Zakharevich]

Post by Thomas Hoffmann
Okay, so let me rephrase my problem: As soon as I use the -Zcrtdll
option, the program
cannot malloc() more than 32MB memory.

Then you are not using EMX (read again what I wrote - carefully).
Probably some Innotech stuff? Ask them. [If I understood correct,
for some unfathomable reason Innotech started with -Zsys stuff - which
is not EMX. Not even speaking about their horrible choice of DLL
names for the "system calls" and CRTL. All that "gcc295a73.dll"
nonsense all over again.]

Hope this helps,
Ilya

P.S. BTW, what is the latest version of gcc ports which may compile
to EMX targets? Meansing working with EMX startup code, and EMX dlls?

Probably the answer is different for AOUT and for OMF?

[Thomas Hoffmann]

Post by Ilya Zakharevich

Post by Thomas Hoffmann
Okay, so let me rephrase my problem: As soon as I use the -Zcrtdll
option, the program
cannot malloc() more than 32MB memory.

Then you are not using EMX (read again what I wrote - carefully).
Probably some Innotech stuff? Ask them. [If I understood correct,
for some unfathomable reason Innotech started with -Zsys stuff - which
is not EMX. Not even speaking about their horrible choice of DLL
names for the "system calls" and CRTL. All that "gcc295a73.dll"
nonsense all over again.]

Not exactly: I use gcc 3.2.1. When compiling w/o -Zcrtdll my test
program uses (chk4dlls output):

D:\thoffman\work\R\heap>chk4dlls heapchk2.exe
Loading DLL 'emx' --> D:\USR\LIB\EMX.DLL.
Loading DLL 'doscalls' --> loaded.
All DLL's used by 'heapchk2.exe' could be loaded.

When compiling w/ -Zcrtdll it uses (w/ -Zmt, otherwise ...s.dll instead
of ...m.dll):

D:\thoffman\work\R\heap>chk4dlls heapchk2.exe
Loading DLL 'emx' --> D:\USR\LIB\EMX.DLL.
Loading DLL 'EMXLIBCM' --> D:\USR\LIB\EMXLIBCM.DLL.
Loading DLL 'gcc321m' --> D:\USR\LIB\GCC321M.DLL.
All DLL's used by 'heapchk2.exe' could be loaded.

BTW: Using your proposed _uflags() call at the begin of the program
fixes the 32MB limit for -Zcrtdll, too.

And so obviously the cited statement from the gcc 3.x readme should be
read "... this port of gcc has a 32MB heap limit...".

Thank you for your explanations,
Thomas.

[Frank Gießler]

Post by Thomas Hoffmann
BTW: Using your proposed _uflags() call at the begin of the program
fixes the 32MB limit for -Zcrtdll, too.

Out of curiosity, is the limit also fixed when you link with '-lgcc' instead of calling
_uflags()?

Frank.

[Thomas Hoffmann]

Sorry for the long delay: indeed, using -lgcc removes the 32MB barrier, too.

Thomas.

Post by Frank Gießler

Post by Thomas Hoffmann
BTW: Using your proposed _uflags() call at the begin of the program
fixes the 32MB limit for -Zcrtdll, too.

Out of curiosity, is the limit also fixed when you link with '-lgcc'
instead of calling _uflags()?
Frank.

[Andreas Buening]

Post by Thomas Hoffmann
Sorry for the long delay: indeed, using -lgcc removes the 32MB barrier, too.

Padden? What's the sense of this behaviour? Why isn't this the
default?


Bye,
Andreas

[Dave and Natalie]

Post by Thomas Hoffmann
When compiling with "gcc proggi.c" it exhausts the user memory after 29
mallocs()
...
298516032 290005120
29: malloc failed
301989440 290005120
30: malloc failed
...
10092160 10005144
20119168 20005144
30146176 30005144
3: malloc failed
33554048 30005144
4: malloc failed
....
Any clues/ workarounds?

Hmm, here using gcc 2.81 I run out of memory at
264961600 260005120
26: malloc failed

with and without -Zcrtdll
Dave

[Dave and Natalie]

Post by Thomas Hoffmann
Not exactly: I use gcc 3.2.1. When compiling w/o -Zcrtdll my test

For your info, your test program can only allocate 32 MB with pgcc 2.95.3 as well as 3.2.1. GCC 2.81 and
innoteks GCC 3.22 can allocate as much memory with -Zcrtdll as without
Dave