Ref: https://github.com/fxn/zeitwerk/pull/308
```ruby
require 'benchmark'
$LOAD_PATH << 'relative-path'
autoload :FOO, '/tmp/foo.rb'
puts Benchmark.realtime {
500_000.times do
Object.autoload?(:FOO)
end
}
```
The above script takes 2.5 seconds on `master`, and only
50ms on this branch.
When we're looking for a feature with an absolute path, we don't
need to call the expensive `get_expanded_load_path`.
When searching for native extensions, if the name does not end in ".so"
then we create a new string and append ".so" so it. If the native extension
is in static_ext_inits, then we could trigger a GC in the rb_filesystem_str_new_cstr.
This could cuase the GC to free lookup_name since we don't use the local
variable anymore.
This bug was caught in this ASAN build:
http://ci.rvm.jp/results/trunk_asan@ruby-sp1/5479182
==435614==ERROR: AddressSanitizer: use-after-poison on address 0x715a63022da0 at pc 0x5e7463873e4e bp 0x7fff383c8b00 sp 0x7fff383c82c0
READ of size 14 at 0x715a63022da0 thread T0
#0 0x5e7463873e4d in __asan_memcpy (/tmp/ruby/build/trunk_asan/ruby+0x214e4d) (BuildId: 607411c0626a2f66b4c20c02179b346aace20898)
#1 0x5e7463b50a82 in memcpy /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10
#2 0x5e7463b50a82 in ruby_nonempty_memcpy /tmp/ruby/src/trunk_asan/include/ruby/internal/memory.h:671:16
#3 0x5e7463b50a82 in str_enc_new /tmp/ruby/src/trunk_asan/string.c:1035:9
#4 0x5e74639b97dd in search_required /tmp/ruby/src/trunk_asan/load.c:1126:21
#5 0x5e74639b97dd in require_internal /tmp/ruby/src/trunk_asan/load.c:1274:17
#6 0x5e74639b83c1 in rb_require_string_internal /tmp/ruby/src/trunk_asan/load.c:1401:22
#7 0x5e74639b83c1 in rb_require_string /tmp/ruby/src/trunk_asan/load.c:1387:12
If there's a syntax error during iseq compilation then prism would leak
memory because it would not free the pm_parse_result_t.
This commit changes pm_iseq_new_with_opt to have a rb_protect to catch
when an error is raised, and return NULL and set error_state to a value
that can be raised by calling rb_jump_tag after memory has been freed.
For example:
10.times do
10_000.times do
eval("/[/=~s")
rescue SyntaxError
end
puts `ps -o rss= -p #{$$}`
end
Before:
39280
68736
99232
128864
158896
188208
217344
246304
275376
304592
After:
12192
13200
14256
14848
16000
16000
16000
16064
17232
17952
Many libraries should be loaded on the main ractor because of
setting constants with unshareable objects and so on.
This patch allows to call `requore` on non-main Ractors by
asking the main ractor to call `require` on it. The calling ractor
waits for the result of `require` from the main ractor.
If the `require` call failed with some reasons, an exception
objects will be deliverred from the main ractor to the calling ractor
if it is copy-able.
Same on `require_relative` and `require` by `autoload`.
Now `Ractor.new{pp obj}` works well (the first call of `pp` requires
`pp` library implicitly).
[Feature #20627]
While working on a separate issue we found that in some cases
`ary_heap_realloc` was being called on frozen arrays. To fix this, this
change does the following:
1) Updates `rb_ary_freeze` to assert the type is an array, return if
already frozen, and shrink the capacity if it is not embedded, shared
or a shared root.
2) Replaces `rb_obj_freeze` with `rb_ary_freeze` when the object is
always an array.
3) In `ary_heap_realloc`, ensure the new capa is set with
`ARY_SET_CAPA`. Previously the change in capa was not set.
4) Adds an assertion to `ary_heap_realloc` that the array is not frozen.
Some of this work was originally done in
https://github.com/ruby/ruby/pull/2640, referencing this issue
https://bugs.ruby-lang.org/issues/16291. There didn't appear to be any
objections to this PR, it appears to have simply lost traction.
The original PR made changes to arrays and strings at the same time,
this PR only does arrays. Also it was old enough that rather than revive
that branch I've made a new one. I added Lourens as co-author in addtion
to Aaron who helped me with this patch.
The original PR made this change for performance reasons, and while
that's still true for this PR, the goal of this PR is to avoid
calling `ary_heap_realloc` on frozen arrays. The capacity should be
shrunk _before_ the array is frozen, not after.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: methodmissing <lourens@methodmissing.com>
This patch removes the `VALUE flags` member from the `rb_ast_t` structure making `rb_ast_t` no longer an IMEMO object.
## Background
We are trying to make the Ruby parser generated from parse.y a universal parser that can be used by other implementations such as mruby.
To achieve this, it is necessary to exclude VALUE and IMEMO from parse.y, AST, and NODE.
## Summary (file by file)
- `rubyparser.h`
- Remove the `VALUE flags` member from `rb_ast_t`
- `ruby_parser.c` and `internal/ruby_parser.h`
- Use TypedData_Make_Struct VALUE which wraps `rb_ast_t` `in ast_alloc()` so that GC can manage it
- You can retrieve `rb_ast_t` from the VALUE by `rb_ruby_ast_data_get()`
- Change the return type of `rb_parser_compile_XXXX()` functions from `rb_ast_t *` to `VALUE`
- rb_ruby_ast_new() which internally `calls ast_alloc()` is to create VALUE vast outside ruby_parser.c
- `iseq.c` and `vm_core.h`
- Amend the first parameter of `rb_iseq_new_XXXX()` functions from `rb_ast_body_t *` to `VALUE`
- This keeps the VALUE of AST on the machine stack to prevent being removed by GC
- `ast.c`
- Almost all change is replacement `rb_ast_t *ast` with `VALUE vast` (sorry for the big diff)
- Fix `node_memsize()`
- Now it includes `rb_ast_local_table_link`, `tokens` and script_lines
- `compile.c`, `load.c`, `node.c`, `parse.y`, `proc.c`, `ruby.c`, `template/prelude.c.tmpl`, `vm.c` and `vm_eval.c`
- Follow-up due to the above changes
- `imemo.{c|h}`
- If an object with `imemo_ast` appears, considers it a bug
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Because Kernel#autoload? uses the current namespace, it can lead to
potentially confusing results. We should make it clearer that modules
count as separate namespaces to lookup in.
Co-authored-by: Jeremy Evans <code@jeremyevans.net>
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
This `st_table` is used to both mark and pin classes
defined from the C API. But `vm->mark_object_ary` already
does both much more efficiently.
Currently a Ruby process starts with 252 rooted classes,
which uses `7224B` in an `st_table` or `2016B` in an `RArray`.
So a baseline of 5kB saved, but since `mark_object_ary` is
preallocated with `1024` slots but only use `405` of them,
it's a net `7kB` save.
`vm->mark_object_ary` is also being refactored.
Prior to this changes, `mark_object_ary` was a regular `RArray`, but
since this allows for references to be moved, it was marked a second
time from `rb_vm_mark()` to pin these objects.
This has the detrimental effect of marking these references on every
minors even though it's a mostly append only list.
But using a custom TypedData we can save from having to mark
all the references on minor GC runs.
Addtionally, immediate values are now ignored and not appended
to `vm->mark_object_ary` as it's just wasted space.
assert does not print the bug report, only the file and line number of
the assertion that failed. RUBY_ASSERT prints the full bug report, which
makes it much easier to debug.
Before this commit, we were mixing a lot of concerns with the prism
compile between RubyVM::InstructionSequence and the general entry
points to the prism parser/compiler.
This commit makes all of the various prism-related APIs mirror
their corresponding APIs in the existing parser/compiler. This means
we now have the correct frame naming, and it's much easier to follow
where the logic actually flows. Furthermore this consolidates a lot
of the prism initialization, making it easier to see where we could
potentially be raising errors.
This is a C API for extensions to resolve and get function symbols of other extensions.
Extensions can check the expected symbol is correctly loaded and accessible, and
use it if it is available.
Otherwise, extensions can raise their own error to guide users to setup their
environments correctly and what's missing.
Loading an extension library with ".dll" suffix on Windows was very
old behavior, and the suffix must be ".so" since 2004. See commits
removing DLEXT2 181a3a2af5df88d145b73a060d51fe437c8c4ad4 and
b76ad15ed0da636161de0243c547ee1e6fc95681.
Instead, use macOS as an example, which uses ".bundle".
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
`Kernel#require` converts feature name objects that have the `to_path`
method such as `Pathname`, but had used the original object on error
and had resulted in an unexpected `TypeError`.
Number of lstat and stat syscalls for each 'require'd file is doubled,
because rb_realpath_internal is called from two places with the same
arguments in require_internal; once for checking the realpaths cache,
and once in load_iseq_eval when iseq is not found.
Introduce rb_realpath_internal_cached function to reuse the realpath_map
cache which memoizes rb_realpath_internal function, leading to less
syscalls and increased startup performance depending on the cost of the syscalls
in a particular environment.
Introduce Universal Parser mode for the parser.
This commit includes these changes:
* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
After [1], using ext/Setup to link some, but not all extensions failed
during linking. I did not know about this option, and had assumed that
only `--with-static-linked-ext` builds can include statically linked
extensions.
Include the support code for statically linked extensions in all
configurations like before [1]. Initialize the table lazily to minimize
footprint on builds that have no statically linked extensions.
[1]: 790cf4b6d0475614afb127b416e87cfa39044d67 "Fix autoload status of
statically linked extensions"
Rebuilding the loaded feature index slowed down with the bug fix
for #17885 in 79a4484a072e9769b603e7b4fbdb15b1d7eccb15. The
slowdown was extreme if realpath emulation was used, but even when
not emulated, it could be about 10x slower.
This adds loaded_features_realpath_map to rb_vm_struct. This is a
hidden hash mapping loaded feature paths to realpaths. When
rebuilding the loaded feature index, look at this hash to get
cached realpath values, and skip calling rb_check_realpath if a
cached value is found.
Fixes [Bug #19246]
This reverts commit 35136e1e9c232ad7a03407b992b2e86b6df43f63.
test-spec has been failing since this revision.
.github/workflows/compilers.yml:82
https://github.com/ruby/ruby/actions/runs/4276884159/jobs/7445299562
```
env:
# Minimal flags to pass the check.
default_cc: 'gcc-11 -fcf-protection -Wa,--generate-missing-build-notes=yes'
optflags: '-O2'
LDFLAGS: '-Wl,-z,now'
# FIXME: Drop skipping options
# https://bugs.ruby-lang.org/issues/18061
# https://sourceware.org/annobin/annobin.html/Test-pie.html
TEST_ANNOCHECK_OPTS: "--skip-pie --skip-gaps"
```
Failure:
```
1)
An exception occurred during: Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
Kernel#require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
2)
An exception occurred during: Kernel#require ($LOADED_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
Kernel#require ($LOADED_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
3)
An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
Kernel#require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
4)
An exception occurred during: Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
Kernel#require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
LeakError: Leaked file descriptor: 9 : #<File:../code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
5)
An exception occurred during: Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
Kernel#require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
LeakError: Closed file descriptor: 8
Closed file descriptor: 9
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:5:in `<top (required)>'
6)
An exception occurred during: Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:317
Kernel.require (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
7)
An exception occurred during: Kernel.require ($LOADED_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:330
Kernel.require ($LOADED_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
8)
An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:535
Kernel.require ($LOADED_FEATURES) does not load a non-canonical path for a file already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
9)
An exception occurred during: Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:551
Kernel.require ($LOADED_FEATURES) does not load a ../ relative path for a file already loaded ERROR
LeakError: Leaked file descriptor: 9 : #<File:../code/load_fixture.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
10)
An exception occurred during: Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required
/__w/ruby/ruby/src/spec/ruby/core/kernel/shared/require.rb:563
Kernel.require ($LOADED_FEATURES) complex, enumerator, rational, thread, ruby2_keywords are already required ERROR
LeakError: Closed file descriptor: 8
Closed file descriptor: 9
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_spec.rb:23:in `<top (required)>'
11)
An exception occurred during: Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:197
Kernel#require_relative with a relative path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `<top (required)>'
12)
An exception occurred during: Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:205
Kernel#require_relative with a relative path ($LOADED_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:4:in `<top (required)>'
13)
An exception occurred during: Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:399
Kernel#require_relative with an absolute path (file extensions) does not load a C-extension file if a complex-extensioned .rb file is already loaded ERROR
LeakError: Leaked file descriptor: 8 : #<File:/__w/ruby/ruby/src/spec/ruby/fixtures/code/load_fixture.ext.rb>
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `<top (required)>'
14)
An exception occurred during: Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:407
Kernel#require_relative with an absolute path ($LOAD_FEATURES) stores an absolute path ERROR
LeakError: Closed file descriptor: 8
/__w/ruby/ruby/src/spec/ruby/core/kernel/require_relative_spec.rb:277:in `<top (required)>'
```
When loading Ruby source files, we can save the result of
successful opens as open(2)/openat(2) are a fairly expensive
syscalls. This also avoids a time-of-check-to-time-of-use
(TOCTTOU) problem.
This reduces open(2) syscalls during `require'; but should be
most apparent when users have a small $LOAD_PATH. Users with
large $LOAD_PATH will benefit less since there'll be more
open(2) failures due to ENOENT.
With `strace -c -e openat ruby -e exit' under Linux, this
results in a ~14% reduction of openat(2) syscalls
(glibc uses openat(2) to implement open(2)).
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
0.00 0.000000 0 296 110 openat
0.00 0.000000 0 254 110 openat
Additionally, the introduction of `struct ruby_file_load_state'
may make future optimizations more apparent.
This change cannot benefit binary (.so) loading since the
dlopen(3) API requires a filename and I'm not aware of an
alternative that takes a pre-existing FD. In typical
situations, Ruby source files outnumber the mount of .so
files.
[Bug #19415]
If multiple threads attemps to load the same file concurrently
it's not a circular dependency issue.
So we check that the existing ThreadShield is owner by the current
fiber before warning about circular dependencies.
[Bug #19415]
If multiple threads attemps to load the same file concurrently
it's not a circular dependency issue.
So we check that the existing ThreadShield is owner by the current
fiber before warning about circular dependencies.
Previously, for statically-linked extensions, we used
`vm->loading_table` to delay calling the init function until the
extensions are required. This caused the extensions to look like they
are in the middle of being loaded even before they're required.
(`rb_feature_p()` returned true with a loading path output.) Combined
with autoload, queries like `defined?(CONST)` and `Module#autoload?`
were confused by this and returned nil incorrectly. RubyGems uses
`defined?` to detect if OpenSSL is available and failed when OpenSSL was
available in builds using `--with-static-linked-ext`.
Use a dedicated table for the init functions instead of adding them to
the loading table. This lets us remove some logic from non-EXTSTATIC
builds.
[Bug #19115]
We use this code path when we require the same extension twice, so it's
not necessarily about the feature being statically linked. Removing this
code when there is no statically linked extensions leads to breakage.
