After reading [this blog
post](https://blog.rubygems.org/2011/08/31/shaving-the-yaml-yak.html),
published almost 10 years ago already, my understanding is that this
problem could come up in two ways:
* Rubygems.org serving corrupted gemspecs". As far as I understand this
was fixed in rubygems.org a lot time ago, since
https://github.com/rubygems/rubygems.org/pull/331.
* Clients having a ten years old gemspec cache with some of these bad
gemspecs. In this case, there's no easy solution but I think ten years
is enough and rebuilding the cache should do the trick.
So, I think it's time we remove this.
https://github.com/rubygems/rubygems/commit/afcb15d556
The `openssl` require when openssl not present was having the
side-effect the our custom require fallbacks would end up loading `Gem::Specification.stubs`.
Co-authored-by: Alyssa Ross <hi@alyssa.is>
https://github.com/rubygems/rubygems/commit/22c4ded4ad
It was added 10 years ago on a "146 additions, 170 deletions" commit
named "Deprecation removals and minor cleanup." that didn't explain much
other than that.
This TODO doesn't necessarily apply nowadays. I don't see how anyways.
TODO notes, if useful at all, should include a clear explanation of what
should be done either via the note itself or the commit message. This
note doens't meet any of these.
So I want to remove it because it distracts me every time I go through
it.
https://github.com/rubygems/rubygems/commit/58d81e8a32
the Gem module's auto-loads will handle loading these as needed,
this started as a redundancy found in *rubygems.rb* which had:
`autoload :Specification, 'rubygems/specification'` as well as
`require 'rubygems/specification'`
https://github.com/rubygems/rubygems/commit/43ceae7ac0
- In one of the cases, filenames were checked for ending with "gz" -
this is changed to check for ending with ".gz"
- The change was made to make it even easier to read the code, and to
match only from the start of the input (as opposed to start of the
line)
https://github.com/rubygems/rubygems/commit/aac4290271
The rationale is that:
* The change has caused realworld issues. See for example
https://github.com/ruby/did_you_mean/issues/117 and specifically [this
comment](https://github.com/ruby/did_you_mean/issues/117#issuecomment-482733159)
for a great explanation of the issue it caused for `did_you_mean`.
* The change also causes problems for our development workflows. For
example, because of it, our `bundler` specs cannot currently be run with
`bin/rake` and we have to use `bin/rspec` or `bin/parallel_spec`
directly. The explanation for this is:
- Our specs install test dependencies to `tmp` before running specs.
- `rake` is one of these test dependencies.
- Before installing each test dependency, we check whether it has
matching installed specs: 2bbcdcde08/bundler/spec/support/rubygems_ext.rb (L109-L114).
- Normally, if `rake` has not yet been installed to `tmp`, this check
fails and `rake` is installed, but since the loaded specs are now
added to `Gem::Specification.stubs` and `rake`'s specification _is_
loaded because we're running through `bin/rake`, the check incorrectly
assumes that `rake` is already installed to `tmp` and skips
installation.
- At a later point the specs check whether `rake` is actually
installed and fail if it's not: 2bbcdcde08/bundler/spec/support/builders.rb (L372-L383)
Essentially, both of the issues are the same. If at runtime we change
the location of gems, we'll _want_ to not consider loaded specifications
when dealing with the new gem location, because the loaded
specifications have not been loaded from there. Loaded specifications is
something different from installed stub specifications and those should
not be mixed.
The PR still seemed to have fixed an issue, so I did my archaeology job
and investigated the original issue to double check if reverting is ok.
The logs for the original error can be found here:
https://ci.appveyor.com/project/rubygems/rubygems/build/1172/job/ogubyucpljcv22ux.
So I installed ruby 2.4.4, checked out the commit reference before the
offending PR, and the exact error reproduced. 🎉
```
$ rake test
/home/deivid/Code/rubygems/lib/rubygems/resolver.rb:231:in `search_for': Unable to resolve dependency: user requested 'bundler (= 1.16.2)' (Gem::UnsatisfiableDependencyError)
from /home/deivid/Code/rubygems/lib/rubygems/resolver.rb:283:in `block in sort_dependencies'
from /home/deivid/Code/rubygems/lib/rubygems/resolver.rb:277:in `each'
from /home/deivid/Code/rubygems/lib/rubygems/resolver.rb:277:in `sort_by'
from /home/deivid/Code/rubygems/lib/rubygems/resolver.rb:277:in `with_index'
from /home/deivid/Code/rubygems/lib/rubygems/resolver.rb:277:in `sort_dependencies'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/delegates/specification_provider.rb:52:in `block in sort_dependencies'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/delegates/specification_provider.rb:69:in `with_no_such_dependency_error_handling'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/delegates/specification_provider.rb:51:in `sort_dependencies'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/resolution.rb:165:in `initial_state'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/resolution.rb:106:in `start_resolution'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/resolution.rb:64:in `resolve'
from /home/deivid/Code/rubygems/lib/rubygems/resolver/molinillo/lib/molinillo/resolver.rb:42:in `resolve'
from /home/deivid/Code/rubygems/lib/rubygems/resolver.rb:188:in `resolve'
from /home/deivid/Code/rubygems/lib/rubygems/request_set.rb:396:in `resolve'
from /home/deivid/Code/rubygems/lib/rubygems/request_set.rb:408:in `resolve_current'
from /home/deivid/Code/rubygems/lib/rubygems.rb:243:in `finish_resolve'
from /home/deivid/Code/rubygems/lib/rubygems/rdoc.rb:13:in `<top (required)>'
from /home/deivid/Code/rubygems/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
from /home/deivid/Code/rubygems/lib/rubygems/core_ext/kernel_require.rb:54:in `require'
from /home/deivid/Code/rubygems/lib/rubygems/test_case.rb:1563:in `<top (required)>'
from /home/deivid/Code/rubygems/test/rubygems/test_bundled_ca.rb:2:in `require'
from /home/deivid/Code/rubygems/test/rubygems/test_bundled_ca.rb:2:in `<top (required)>'
from /home/deivid/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:15:in `require'
from /home/deivid/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:15:in `block in <main>'
from /home/deivid/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:4:in `select'
from /home/deivid/.rbenv/versions/2.4.4/lib/ruby/gems/2.4.0/gems/rake-12.0.0/lib/rake/rake_test_loader.rb:4:in `<main>'
rake aborted!
Command failed with status (1)
Tasks: TOP => test
```
Now the explanation of the error:
* Rubygems base `TestCase` class requires `bundler` because some tests
use `bundler`:
2bbcdcde08/lib/rubygems/test_case.rb (L26)
* That `require` (our custom rubygems require) would activate the
default bundler spec (1.16.1 for ruby 2.4.4) but then overwrite it with
a 1.16.2 version (the locally provided bundler those days) due to [this
old
hack](9f7bf0ac3a/lib/bundler/version.rb (L7-L23)).
* Rubygems base `TestCase` class requires `rubygems/rdoc`:
2bbcdcde08/lib/rubygems/test_case.rb (L1536)
* And that file ends up calling `Gem.finish_resolve`:
2bbcdcde08/lib/rubygems/rdoc.rb (L13)
* `Gem.finish_resolve` adds the currently loaded specs to the
resolution:
2bbcdcde08/lib/rubygems.rb (L235)
* That means it would try to resolve bundler 1.16.2, but no
specification for that version was installed since the default was
1.16.1. That explains why upgrading to rubygems 2.7.7 fixed the issue,
since it provided bundler 1.16.2 by default so there was not bundler
version discrepancy.
After understanding the error, I conclude that:
* Only this part of the original patch was actually needed to resolve
the error, not any of the changes in `Gem::Specification.stubs` and
`Gem::Specification.stubs_for`:
```diff
diff --git a/lib/rubygems/test_case.rb b/lib/rubygems/test_case.rb
index f1cd3d274c..92c848e870 100644
--- a/lib/rubygems/test_case.rb
+++ b/lib/rubygems/test_case.rb
@@ -13,6 +13,15 @@ else
require 'rubygems'
end
+# If bundler gemspec exists, add to stubs
+bundler_gemspec = File.expand_path("../../../bundler/bundler.gemspec", __FILE__)
+if File.exist?(bundler_gemspec)
+ Gem::Specification.dirs.unshift File.dirname(bundler_gemspec)
+ Gem::Specification.class_variable_set :@@stubs, nil
+ Gem::Specification.stubs
+ Gem::Specification.dirs.shift
+end
+
begin
gem 'minitest'
rescue Gem::LoadError
```
So, I propose to revert adding loaded specification to
`Gem::Specification.stubs` and `Gem::Specification.stubs_for` because I
think it's safe, it fixes the issues caused by their addition, and it
simplifies `Gem::Specification` code, which is already complicated
enough.
https://github.com/rubygems/rubygems/commit/5269cd617c
Originally, the call to `.stubs_for` allowed to incrementally populate
the `@@stubs_by_name` (especially see the `"#{name}-*.gemspec"` pattern
in 4fa03bb7aac9f25f44394e818433fdda9962ae8d). Now it looks like it
expects that all stubs are loaded, but the `.stubs_for` still matches
the .gemspec files by the `name` pattern:
6d45e0f7ac/lib/rubygems/specification.rb (L845)
I think this was done by mistake incrementally by PR #1239 and
4cee8ca9199ac7b3ab8647e0b78615f55d3eb02b. I think the best option is to
get back to the original implementation, to let RubyGems incrementally
populate the array. Other option would be to replace the logic in
`.stub_for` by call to `.stubs`, but the means the performance
improvement from the original commit was lost.
https://github.com/rubygems/rubygems/commit/4d0e18185a
This requirement was introduced 14 years ago in
7ce7039b390440754954df5efea619e9f57ef823, and I don't think it's
necessary anymore. I made several tests introducing UTF-8 characters in
gemspec files and generating indexes out of them, and couldn't find any
issues. Gemspecs are read with UTF-8 encoding these days.
