The initial implementation of EDMM support pre-accepted every mmapped
region of enclave pages. However in some cases, applications (e.g.,
Java runtime) may rely on lazy allocation of pages (default on Linux),
where the VMAs are reserved but not actually committed to physical
memory. In particular, `mmap(..., MAP_NORESERVE)` requests are often
used in such cases -- to mmap a huge chunk of memory (possibly never
used in the future) at once and then commit pages on demand on page
fault events. Due to the lack of lazy allocation concept in Gramine,
this incurred performance and physical memory overhead on allocating and
accepting the whole range of enclave pages.

The commit adds the lazy allocation support by deferring page accepts to
page-fault events when Gramine notices `MAP_NORESERVE` flag in the mmap
request. It also introduces a two-level bitvector into PAL indicating
the commit status of each enclave page and an upcall from the SGX EDMM
backend into the LibOS to ask for additional info on a particular page.
Further, since the checkpointing decisions are purely done by LibOS, for
correct and fast handover of `MAP_NORESERVE` enclave pages, we now
introduce a new PAL API to let LibOS learn the enclave page tracker
state in PAL. Another new PAL API is also added for
`madvise(MADV_DONTNEED)` where we need to free the pages then move them
to the "lazy-alloc-on-access" status.

Signed-off-by: Kailun Qin <kailun.qin@intel.com>

!TODO: use below commit msg:

[LibOS,PAL/Linux-SGX] Add EDMM lazy allocation support

The initial implementation of EDMM support pre-accepted every mmapped
region of enclave pages. However in some cases, applications (e.g.,
Java runtime) may rely on lazy allocation of pages (default on Linux),
where the VMAs are reserved but not actually committed to physical
memory. In particular, `mmap(..., MAP_NORESERVE)` requests are often
used in such cases -- to mmap a huge chunk of memory (possibly never
used in the future) at once and then commit pages on demand on page
fault events. Due to the lack of lazy allocation concept in Gramine,
this incurred performance and physical memory overhead on allocating and
accepting the whole range of enclave pages.

The commit adds the lazy allocation support by deferring page accepts to
page-fault events when Gramine notices `MAP_NORESERVE` flag in the mmap
request. It also introduces a two-level bitvector into PAL indicating
the commit status of each enclave page and an upcall from the SGX EDMM
backend into the LibOS to ask for additional info on a particular page.
Further, since the checkpointing decisions are purely done by LibOS, for
correct and fast handover of `MAP_NORESERVE` enclave pages, we now
introduce a new PAL API to let LibOS learn the enclave page tracker
state in PAL. Another new PAL API is also added for
`madvise(MADV_DONTNEED)` where we need to free the pages then move them
to the "lazy-alloc-on-access" status. However it is currently not used
due to data races in the SGX driver.

Signed-off-by: Kailun Qin <kailun.qin@intel.com>

Signed-off-by: Kailun Qin <kailun.qin@intel.com>

Signed-off-by: Kailun Qin <kailun.qin@intel.com>

!TODO: use below commit msg:

[LibOS,PAL/Linux-SGX] Add EDMM lazy allocation support

The initial implementation of EDMM support pre-accepted every mmapped
region of enclave pages. However in some cases, applications (e.g.,
Java runtime) may rely on lazy allocation of pages (default on Linux),
where the VMAs are reserved but not actually committed to physical
memory. In particular, `mmap(..., MAP_NORESERVE)` requests are often
used in such cases -- to mmap a huge chunk of memory (possibly never
used in the future) at once and then commit pages on demand on page
fault events. Due to the lack of lazy allocation concept in Gramine,
this incurred performance and physical memory overhead on allocating and
accepting the whole range of enclave pages.

The commit adds the lazy allocation support by deferring page accepts to
page-fault events when Gramine notices `MAP_NORESERVE` flag in the mmap
request. It also introduces a two-level bitvector into PAL indicating
the commit status of each enclave page and an upcall from the SGX EDMM
backend into the LibOS to ask for additional info on a particular page.
Further, since the checkpointing decisions are purely done by LibOS, for
correct and fast handover of `MAP_NORESERVE` enclave pages, we now
introduce a new PAL API (`PalGetLazyCommitPages()`) to let LibOS learn
the enclave page tracker state in PAL. Another new PAL API
(`PalFreeThenLazyReallocCommittedPages()`) is added for
`madvise(MADV_DONTNEED)` where we need to free the pages then move them
to the "lazy-alloc-on-access" status.

Due to data races in the SGX driver, the modification to
`madvise(MADV_DONTNEED)` is currently inactive (i.e., it is still
emulated by zero-filling the committed pages), and a temporary
workaround is implemented in the lazy allocation flow to handle the
corner case of a benign spurious #PF on a non-present page during
EACCEPT.

Signed-off-by: Kailun Qin <kailun.qin@intel.com>

Signed-off-by: Kailun Qin <kailun.qin@intel.com>