工作應用/Java

Java JNI GC Thread Error (EINVAL)

contents

  1. 1. Description
  2. 2. Options
  3. 3. Conclusion

Keywords: JNI, Java, GC Thread, add_workers, static library, Thread Error

Solution: -XX:+AdjustStackSizeForTLS from JDK 14

Description

We cannot launch GC thread of JVM from JNI. And, it shows

1
[os,thread] Failed to start thread - pthread_create failed (EINVAL) for attributes: stacksize: 1024k, guardsize: 0k, detached.

The main reason is the glibc bugs for native C code issues. For example, we allocate thread-local variables by static library linking.

1
2
3
__thread int buf[1048576];
thread_local int buf[1048576];
#pragma omp threadprive(buf);

And then, glibc 2.12 will move them on stack, and require minimum stack size at least 1024k bytes. But, the default configuration JVM of JDK11 will minimum stacksize to be 1024k bytes as default in Linux OS. Therefore, the pthread_create throws the error EINVAL for specific stacksize setting.

Options

https://www.oracle.com/java/technologies/javase/14all-relnotes.html

The glibc library allocates some thread-local storage (TLS) in the stack of a newly created thread, leaving less stack than requested for the thread to do its work. This is particularly a problem for threads with small stack sizes. It is an inherited issue from a well-known glibc problem, ‘Program with large TLS segments fail’ [0] and has been observed in Java applications. In one of the reported JVM failure instances, the issue manifests as a StackOverflowError on the process reaper thread, which has a small stack size. The java.lang.Thread constructor enables users to specify the stack size for a new thread. The created thread may encounter the TLS problem when the specified size is too small to accommodate the on-stack TLS blocks.

In JDK 8, a system property, jdk.lang.processReaperUseDefaultStackSize, was introduced to address the TLS issue only for reaper threads. Setting the property gives a bigger stack size to the reaper threads.

To address the issue for all threads, a general purpose workaround was implemented in Java which adjusts thread stack size for TLS. It can be enabled by using the AdjustStackSizeForTLS command-line option:

When creating a new thread, if AdjustStackSizeForTLS is true, the static TLS area size is added to the user requested stack size. AdjustStackSizeForTLS is disabled by default.

Conclusion

整合進大型程式時，要多觀察靜態編譯的部分，如果有一些 TLS 的宣告，則會增加其他預設執行緒的最小使用堆疊大小。而 JVM 又傾向使用最少資源去完成，則很容易在計算最小資源時發生問題。

如果去查閱 JDK15 的 代碼 os_linux.cpp 時，有一段到動態鏈結函數庫 glibc 的詢問最低需求來解決此問題。這一段牽涉到 hack 技巧，偷偷利用 symbol 爬出私有函數，不算正常操作行為。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
// On Linux, glibc places static TLS blocks (for __thread variables) on
// the thread stack. This decreases the stack size actually available
// to threads.
//
// For large static TLS sizes, this may cause threads to malfunction due
// to insufficient stack space. This is a well-known issue in glibc:
// http://sourceware.org/bugzilla/show_bug.cgi?id=11787.
//
// As a workaround, we call a private but assumed-stable glibc function,
// __pthread_get_minstack() to obtain the minstack size and derive the
// static TLS size from it. We then increase the user requested stack
// size by this TLS size.
//
// Due to compatibility concerns, this size adjustment is opt-in and
// controlled via AdjustStackSizeForTLS.
typedef size_t (*GetMinStack)(const pthread_attr_t *attr);
GetMinStack _get_minstack_func = NULL;
static void get_minstack_init() {
  _get_minstack_func =
        (GetMinStack)dlsym(RTLD_DEFAULT, "__pthread_get_minstack");
  log_info(os, thread)("Lookup of __pthread_get_minstack %s",
                       _get_minstack_func == NULL ? "failed" : "succeeded");
}
// Returns the size of the static TLS area glibc puts on thread stacks.
// The value is cached on first use, which occurs when the first thread
// is created during VM initialization.
static size_t get_static_tls_area_size(const pthread_attr_t *attr) {
  size_t tls_size = 0;
  if (_get_minstack_func != NULL) {
    // Obtain the pthread minstack size by calling __pthread_get_minstack.
    size_t minstack_size = _get_minstack_func(attr);
    // Remove non-TLS area size included in minstack size returned
    // by __pthread_get_minstack() to get the static TLS size.
    // In glibc before 2.27, minstack size includes guard_size.
    // In glibc 2.27 and later, guard_size is automatically added
    // to the stack size by pthread_create and is no longer included
    // in minstack size. In both cases, the guard_size is taken into
    // account, so there is no need to adjust the result for that.
    //
    // Although __pthread_get_minstack() is a private glibc function,
    // it is expected to have a stable behavior across future glibc
    // versions while glibc still allocates the static TLS blocks off
    // the stack. Following is glibc 2.28 __pthread_get_minstack():
    //
    // size_t
    // __pthread_get_minstack (const pthread_attr_t *attr)
    // {
    //   return GLRO(dl_pagesize) + __static_tls_size + PTHREAD_STACK_MIN;
    // }
    //
    //
    // The following 'minstack_size > os::vm_page_size() + PTHREAD_STACK_MIN'
    // if check is done for precaution.
    if (minstack_size > (size_t)os::vm_page_size() + PTHREAD_STACK_MIN) {
      tls_size = minstack_size - os::vm_page_size() - PTHREAD_STACK_MIN;
    }
  }
  log_info(os, thread)("Stack size adjustment for TLS is " SIZE_FORMAT,
                       tls_size);
  return tls_size;
}