worker: use copy of process.env

Instead of sharing the OS-backed store for all `process.env` instances,
create a copy of `process.env` for every worker that is created.

The copies do not interact. Native-addons do not see modifications to
`process.env` from Worker threads, but child processes started from
Workers do default to the Worker’s copy of `process.env`.

This makes Workers behave like child processes as far as `process.env`
is concerned, and an option corresponding to the `child_process`
module’s `env` option is added to the constructor.

Fixes: https://github.com/nodejs/node/issues/24947

PR-URL: https://github.com/nodejs/node/pull/26544
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Vse Mozhet Byt <vsemozhetbyt@gmail.com>
Reviewed-By: Yongsheng Zhang <zyszys98@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
This commit is contained in:
Anna Henningsen 2019-02-22 20:11:19 +01:00
parent 1ee37aac09
commit 9fbf0c60b5
No known key found for this signature in database
GPG Key ID: 9C63F3A6CD2AD8F9
13 changed files with 229 additions and 51 deletions

View File

@ -954,6 +954,11 @@ emitMyWarning();
<!-- YAML
added: v0.1.27
changes:
- version: REPLACEME
pr-url: https://github.com/nodejs/node/pull/26544
description: Worker threads will now use a copy of the parent threads
`process.env` by default, configurable through the `env`
option of the `Worker` constructor.
- version: v10.0.0
pr-url: https://github.com/nodejs/node/pull/18990
description: Implicit conversion of variable value to string is deprecated.
@ -983,8 +988,9 @@ An example of this object looks like:
```
It is possible to modify this object, but such modifications will not be
reflected outside the Node.js process. In other words, the following example
would not work:
reflected outside the Node.js process, or (unless explicitly requested)
to other [`Worker`][] threads.
In other words, the following example would not work:
```console
$ node -e 'process.env.foo = "bar"' && echo $foo
@ -1027,7 +1033,12 @@ console.log(process.env.test);
// => 1
```
`process.env` is read-only in [`Worker`][] threads.
Unless explicitly specified when creating a [`Worker`][] instance,
each [`Worker`][] thread has its own copy of `process.env`, based on its
parent threads `process.env`, or whatever was specified as the `env` option
to the [`Worker`][] constructor. Changes to `process.env` will not be visible
across [`Worker`][] threads, and only the main thread can make changes that
are visible to the operating system or to native add-ons.
## process.execArgv
<!-- YAML

View File

@ -125,6 +125,25 @@ if (isMainThread) {
}
```
## worker.SHARE_ENV
<!-- YAML
added: REPLACEME
-->
* {symbol}
A special value that can be passed as the `env` option of the [`Worker`][]
constructor, to indicate that the current thread and the Worker thread should
share read and write access to the same set of environment variables.
```js
const { Worker, SHARE_ENV } = require('worker_threads');
new Worker('process.env.SET_IN_WORKER = "foo"', { eval: true, env: SHARE_ENV })
.on('exit', () => {
console.log(process.env.SET_IN_WORKER); // Prints 'foo'.
});
```
## worker.threadId
<!-- YAML
added: v10.5.0
@ -380,7 +399,11 @@ Notable differences inside a Worker environment are:
and [`process.abort()`][] is not available.
- [`process.chdir()`][] and `process` methods that set group or user ids
are not available.
- [`process.env`][] is a read-only reference to the environment variables.
- [`process.env`][] is a copy of the parent thread's environment variables,
unless otherwise specified. Changes to one copy will not be visible in other
threads, and will not be visible to native add-ons (unless
[`worker.SHARE_ENV`][] has been passed as the `env` option to the
[`Worker`][] constructor).
- [`process.title`][] cannot be modified.
- Signals will not be delivered through [`process.on('...')`][Signals events].
- Execution may stop at any point as a result of [`worker.terminate()`][]
@ -439,13 +462,18 @@ if (isMainThread) {
If `options.eval` is `true`, this is a string containing JavaScript code
rather than a path.
* `options` {Object}
* `env` {Object} If set, specifies the initial value of `process.env` inside
the Worker thread. As a special value, [`worker.SHARE_ENV`][] may be used
to specify that the parent thread and the child thread should share their
environment variables; in that case, changes to one threads `process.env`
object will affect the other thread as well. **Default:** `process.env`.
* `eval` {boolean} If `true`, interpret the first argument to the constructor
as a script that is executed once the worker is online.
* `workerData` {any} Any JavaScript value that will be cloned and made
available as [`require('worker_threads').workerData`][]. The cloning will
occur as described in the [HTML structured clone algorithm][], and an error
will be thrown if the object cannot be cloned (e.g. because it contains
`function`s).
* `execArgv` {string[]} List of node CLI options passed to the worker.
V8 options (such as `--max-old-space-size`) and options that affect the
process (such as `--title`) are not supported. If set, this will be provided
as [`process.execArgv`][] inside the worker. By default, options will be
inherited from the parent thread.
* `stdin` {boolean} If this is set to `true`, then `worker.stdin` will
provide a writable stream whose contents will appear as `process.stdin`
inside the Worker. By default, no data is provided.
@ -453,11 +481,11 @@ if (isMainThread) {
not automatically be piped through to `process.stdout` in the parent.
* `stderr` {boolean} If this is set to `true`, then `worker.stderr` will
not automatically be piped through to `process.stderr` in the parent.
* `execArgv` {string[]} List of node CLI options passed to the worker.
V8 options (such as `--max-old-space-size`) and options that affect the
process (such as `--title`) are not supported. If set, this will be provided
as [`process.execArgv`][] inside the worker. By default, options will be
inherited from the parent thread.
* `workerData` {any} Any JavaScript value that will be cloned and made
available as [`require('worker_threads').workerData`][]. The cloning will
occur as described in the [HTML structured clone algorithm][], and an error
will be thrown if the object cannot be cloned (e.g. because it contains
`function`s).
### Event: 'error'
<!-- YAML
@ -628,6 +656,7 @@ active handle in the event system. If the worker is already `unref()`ed calling
[`vm`]: vm.html
[`worker.on('message')`]: #worker_threads_event_message_1
[`worker.postMessage()`]: #worker_threads_worker_postmessage_value_transferlist
[`worker.SHARE_ENV`]: #worker_threads_worker_share_env
[`worker.terminate()`]: #worker_threads_worker_terminate_callback
[`worker.threadId`]: #worker_threads_worker_threadid_1
[Addons worker support]: addons.html#addons_worker_support

View File

@ -45,6 +45,8 @@ const kOnCouldNotSerializeErr = Symbol('kOnCouldNotSerializeErr');
const kOnErrorMessage = Symbol('kOnErrorMessage');
const kParentSideStdio = Symbol('kParentSideStdio');
const SHARE_ENV = Symbol.for('nodejs.worker_threads.SHARE_ENV');
let debuglog;
function debug(...args) {
if (!debuglog) {
@ -79,12 +81,33 @@ class Worker extends EventEmitter {
}
}
let env;
if (typeof options.env === 'object' && options.env !== null) {
env = Object.create(null);
for (const [ key, value ] of Object.entries(options.env))
env[key] = `${value}`;
} else if (options.env == null) {
env = process.env;
} else if (options.env !== SHARE_ENV) {
throw new ERR_INVALID_ARG_TYPE(
'options.env',
['object', 'undefined', 'null', 'worker_threads.SHARE_ENV'],
options.env);
}
const url = options.eval ? null : pathToFileURL(filename);
// Set up the C++ handle for the worker, as well as some internal wiring.
this[kHandle] = new WorkerImpl(url, options.execArgv);
if (this[kHandle].invalidExecArgv) {
throw new ERR_WORKER_INVALID_EXEC_ARGV(this[kHandle].invalidExecArgv);
}
if (env === process.env) {
// This may be faster than manually cloning the object in C++, especially
// when recursively spawning Workers.
this[kHandle].cloneParentEnvVars();
} else if (env !== undefined) {
this[kHandle].setEnvVars(env);
}
this[kHandle].onexit = (code) => this[kOnExit](code);
this[kPort] = this[kHandle].messagePort;
this[kPort].on('message', (data) => this[kOnMessage](data));
@ -253,6 +276,7 @@ function pipeWithoutWarning(source, dest) {
module.exports = {
ownsProcessState,
isMainThread,
SHARE_ENV,
threadId,
Worker,
};

View File

@ -2,6 +2,7 @@
const {
isMainThread,
SHARE_ENV,
threadId,
Worker
} = require('internal/worker');
@ -18,6 +19,7 @@ module.exports = {
MessageChannel,
moveMessagePortToContext,
threadId,
SHARE_ENV,
Worker,
parentPort: null,
workerData: null,

View File

@ -447,12 +447,12 @@ inline uint64_t Environment::timer_base() const {
return timer_base_;
}
inline std::shared_ptr<KVStore> Environment::envvars() {
return envvars_;
inline std::shared_ptr<KVStore> Environment::env_vars() {
return env_vars_;
}
inline void Environment::set_envvars(std::shared_ptr<KVStore> envvars) {
envvars_ = envvars;
inline void Environment::set_env_vars(std::shared_ptr<KVStore> env_vars) {
env_vars_ = env_vars;
}
inline bool Environment::printed_error() const {

View File

@ -178,7 +178,7 @@ Environment::Environment(IsolateData* isolate_data,
set_as_callback_data_template(templ);
}
set_envvars(per_process::real_environment);
set_env_vars(per_process::system_environment);
// We create new copies of the per-Environment option sets, so that it is
// easier to modify them after Environment creation. The defaults are

View File

@ -556,11 +556,11 @@ class KVStore {
virtual v8::Maybe<bool> AssignFromObject(v8::Local<v8::Context> context,
v8::Local<v8::Object> entries);
static std::shared_ptr<KVStore> CreateGenericKVStore();
static std::shared_ptr<KVStore> CreateMapKVStore();
};
namespace per_process {
extern std::shared_ptr<KVStore> real_environment;
extern std::shared_ptr<KVStore> system_environment;
}
class AsyncHooks {
@ -812,8 +812,8 @@ class Environment {
inline ImmediateInfo* immediate_info();
inline TickInfo* tick_info();
inline uint64_t timer_base() const;
inline std::shared_ptr<KVStore> envvars();
inline void set_envvars(std::shared_ptr<KVStore> envvars);
inline std::shared_ptr<KVStore> env_vars();
inline void set_env_vars(std::shared_ptr<KVStore> env_vars);
inline IsolateData* isolate_data() const;
@ -1100,7 +1100,7 @@ class Environment {
ImmediateInfo immediate_info_;
TickInfo tick_info_;
const uint64_t timer_base_;
std::shared_ptr<KVStore> envvars_;
std::shared_ptr<KVStore> env_vars_;
bool printed_error_ = false;
bool emit_env_nonstring_warning_ = true;
bool emit_err_name_warning_ = true;

View File

@ -43,7 +43,7 @@ bool SafeGetenv(const char* key, std::string* text, Environment* env) {
if (env != nullptr) {
HandleScope handle_scope(env->isolate());
TryCatch ignore_errors(env->isolate());
MaybeLocal<String> value = env->envvars()->Get(
MaybeLocal<String> value = env->env_vars()->Get(
env->isolate(),
String::NewFromUtf8(env->isolate(), key, NewStringType::kNormal)
.ToLocalChecked());

View File

@ -40,7 +40,7 @@ class RealEnvStore final : public KVStore {
Local<Array> Enumerate(Isolate* isolate) const override;
};
class GenericKVStore final : public KVStore {
class MapKVStore final : public KVStore {
public:
Local<String> Get(Isolate* isolate, Local<String> key) const override;
void Set(Isolate* isolate, Local<String> key, Local<String> value) override;
@ -50,8 +50,8 @@ class GenericKVStore final : public KVStore {
std::shared_ptr<KVStore> Clone(Isolate* isolate) const override;
GenericKVStore() {}
GenericKVStore(const GenericKVStore& other) : map_(other.map_) {}
MapKVStore() {}
MapKVStore(const MapKVStore& other) : map_(other.map_) {}
private:
mutable Mutex mutex_;
@ -60,7 +60,7 @@ class GenericKVStore final : public KVStore {
namespace per_process {
Mutex env_var_mutex;
std::shared_ptr<KVStore> real_environment = std::make_shared<RealEnvStore>();
std::shared_ptr<KVStore> system_environment = std::make_shared<RealEnvStore>();
} // namespace per_process
Local<String> RealEnvStore::Get(Isolate* isolate,
@ -207,7 +207,7 @@ std::shared_ptr<KVStore> KVStore::Clone(v8::Isolate* isolate) const {
HandleScope handle_scope(isolate);
Local<Context> context = isolate->GetCurrentContext();
std::shared_ptr<KVStore> copy = KVStore::CreateGenericKVStore();
std::shared_ptr<KVStore> copy = KVStore::CreateMapKVStore();
Local<Array> keys = Enumerate(isolate);
uint32_t keys_length = keys->Length();
for (uint32_t i = 0; i < keys_length; i++) {
@ -218,9 +218,9 @@ std::shared_ptr<KVStore> KVStore::Clone(v8::Isolate* isolate) const {
return copy;
}
Local<String> GenericKVStore::Get(Isolate* isolate, Local<String> key) const {
Local<String> MapKVStore::Get(Isolate* isolate, Local<String> key) const {
Mutex::ScopedLock lock(mutex_);
String::Utf8Value str(isolate, key);
Utf8Value str(isolate, key);
auto it = map_.find(std::string(*str, str.length()));
if (it == map_.end()) return Local<String>();
return String::NewFromUtf8(isolate, it->second.data(),
@ -228,31 +228,30 @@ Local<String> GenericKVStore::Get(Isolate* isolate, Local<String> key) const {
.ToLocalChecked();
}
void GenericKVStore::Set(Isolate* isolate, Local<String> key,
Local<String> value) {
void MapKVStore::Set(Isolate* isolate, Local<String> key, Local<String> value) {
Mutex::ScopedLock lock(mutex_);
String::Utf8Value key_str(isolate, key);
String::Utf8Value value_str(isolate, value);
Utf8Value key_str(isolate, key);
Utf8Value value_str(isolate, value);
if (*key_str != nullptr && *value_str != nullptr) {
map_[std::string(*key_str, key_str.length())] =
std::string(*value_str, value_str.length());
}
}
int32_t GenericKVStore::Query(Isolate* isolate, Local<String> key) const {
int32_t MapKVStore::Query(Isolate* isolate, Local<String> key) const {
Mutex::ScopedLock lock(mutex_);
String::Utf8Value str(isolate, key);
Utf8Value str(isolate, key);
auto it = map_.find(std::string(*str, str.length()));
return it == map_.end() ? -1 : 0;
}
void GenericKVStore::Delete(Isolate* isolate, Local<String> key) {
void MapKVStore::Delete(Isolate* isolate, Local<String> key) {
Mutex::ScopedLock lock(mutex_);
String::Utf8Value str(isolate, key);
Utf8Value str(isolate, key);
map_.erase(std::string(*str, str.length()));
}
Local<Array> GenericKVStore::Enumerate(Isolate* isolate) const {
Local<Array> MapKVStore::Enumerate(Isolate* isolate) const {
Mutex::ScopedLock lock(mutex_);
std::vector<Local<Value>> values;
values.reserve(map_.size());
@ -265,12 +264,12 @@ Local<Array> GenericKVStore::Enumerate(Isolate* isolate) const {
return Array::New(isolate, values.data(), values.size());
}
std::shared_ptr<KVStore> GenericKVStore::Clone(Isolate* isolate) const {
return std::make_shared<GenericKVStore>(*this);
std::shared_ptr<KVStore> MapKVStore::Clone(Isolate* isolate) const {
return std::make_shared<MapKVStore>(*this);
}
std::shared_ptr<KVStore> KVStore::CreateGenericKVStore() {
return std::make_shared<GenericKVStore>();
std::shared_ptr<KVStore> KVStore::CreateMapKVStore() {
return std::make_shared<MapKVStore>();
}
Maybe<bool> KVStore::AssignFromObject(Local<Context> context,
@ -307,7 +306,7 @@ static void EnvGetter(Local<Name> property,
}
CHECK(property->IsString());
info.GetReturnValue().Set(
env->envvars()->Get(env->isolate(), property.As<String>()));
env->env_vars()->Get(env->isolate(), property.As<String>()));
}
static void EnvSetter(Local<Name> property,
@ -338,7 +337,7 @@ static void EnvSetter(Local<Name> property,
return;
}
env->envvars()->Set(env->isolate(), key, value_string);
env->env_vars()->Set(env->isolate(), key, value_string);
// Whether it worked or not, always return value.
info.GetReturnValue().Set(value);
@ -348,7 +347,7 @@ static void EnvQuery(Local<Name> property,
const PropertyCallbackInfo<Integer>& info) {
Environment* env = Environment::GetCurrent(info);
if (property->IsString()) {
int32_t rc = env->envvars()->Query(env->isolate(), property.As<String>());
int32_t rc = env->env_vars()->Query(env->isolate(), property.As<String>());
if (rc != -1) info.GetReturnValue().Set(rc);
}
}
@ -357,7 +356,7 @@ static void EnvDeleter(Local<Name> property,
const PropertyCallbackInfo<Boolean>& info) {
Environment* env = Environment::GetCurrent(info);
if (property->IsString()) {
env->envvars()->Delete(env->isolate(), property.As<String>());
env->env_vars()->Delete(env->isolate(), property.As<String>());
}
// process.env never has non-configurable properties, so always
@ -369,7 +368,7 @@ static void EnvEnumerator(const PropertyCallbackInfo<Array>& info) {
Environment* env = Environment::GetCurrent(info);
info.GetReturnValue().Set(
env->envvars()->Enumerate(env->isolate()));
env->env_vars()->Enumerate(env->isolate()));
}
MaybeLocal<Object> CreateEnvVarProxy(Local<Context> context,

View File

@ -68,7 +68,8 @@ Worker::Worker(Environment* env,
exec_argv_(exec_argv),
platform_(env->isolate_data()->platform()),
profiler_idle_notifier_started_(env->profiler_idle_notifier_started()),
thread_id_(Environment::AllocateThreadId()) {
thread_id_(Environment::AllocateThreadId()),
env_vars_(env->env_vars()) {
Debug(this, "Creating new worker instance with thread id %llu", thread_id_);
// Set up everything that needs to be set up in the parent environment.
@ -254,6 +255,7 @@ void Worker::Run() {
Environment::kNoFlags,
thread_id_));
CHECK_NOT_NULL(env_);
env_->set_env_vars(std::move(env_vars_));
env_->set_abort_on_uncaught_exception(false);
env_->set_worker_context(this);
@ -469,6 +471,25 @@ void Worker::New(const FunctionCallbackInfo<Value>& args) {
new Worker(env, args.This(), url, per_isolate_opts, std::move(exec_argv_out));
}
void Worker::CloneParentEnvVars(const FunctionCallbackInfo<Value>& args) {
Worker* w;
ASSIGN_OR_RETURN_UNWRAP(&w, args.This());
CHECK(w->thread_joined_); // The Worker has not started yet.
w->env_vars_ = w->env()->env_vars()->Clone(args.GetIsolate());
}
void Worker::SetEnvVars(const FunctionCallbackInfo<Value>& args) {
Worker* w;
ASSIGN_OR_RETURN_UNWRAP(&w, args.This());
CHECK(w->thread_joined_); // The Worker has not started yet.
CHECK(args[0]->IsObject());
w->env_vars_ = KVStore::CreateMapKVStore();
w->env_vars_->AssignFromObject(args.GetIsolate()->GetCurrentContext(),
args[0].As<Object>());
}
void Worker::StartThread(const FunctionCallbackInfo<Value>& args) {
Worker* w;
ASSIGN_OR_RETURN_UNWRAP(&w, args.This());
@ -566,6 +587,8 @@ void InitWorker(Local<Object> target,
w->InstanceTemplate()->SetInternalFieldCount(1);
w->Inherit(AsyncWrap::GetConstructorTemplate(env));
env->SetProtoMethod(w, "setEnvVars", Worker::SetEnvVars);
env->SetProtoMethod(w, "cloneParentEnvVars", Worker::CloneParentEnvVars);
env->SetProtoMethod(w, "startThread", Worker::StartThread);
env->SetProtoMethod(w, "stopThread", Worker::StopThread);
env->SetProtoMethod(w, "ref", Worker::Ref);

View File

@ -43,6 +43,9 @@ class Worker : public AsyncWrap {
bool is_stopped() const;
static void New(const v8::FunctionCallbackInfo<v8::Value>& args);
static void CloneParentEnvVars(
const v8::FunctionCallbackInfo<v8::Value>& args);
static void SetEnvVars(const v8::FunctionCallbackInfo<v8::Value>& args);
static void StartThread(const v8::FunctionCallbackInfo<v8::Value>& args);
static void StopThread(const v8::FunctionCallbackInfo<v8::Value>& args);
static void Ref(const v8::FunctionCallbackInfo<v8::Value>& args);
@ -78,6 +81,7 @@ class Worker : public AsyncWrap {
static constexpr size_t kStackBufferSize = 192 * 1024;
std::unique_ptr<MessagePortData> child_port_data_;
std::shared_ptr<KVStore> env_vars_;
// The child port is kept alive by the child Environment's persistent
// handle to it, as long as that child Environment exists.

View File

@ -0,0 +1,32 @@
'use strict';
const common = require('../common');
const assert = require('assert');
const { Worker, parentPort, SHARE_ENV, workerData } = require('worker_threads');
if (!workerData) {
process.env.SET_IN_PARENT = 'set';
assert.strictEqual(process.env.SET_IN_PARENT, 'set');
const w = new Worker(__filename, {
workerData: 'runInWorker',
env: SHARE_ENV
}).on('exit', common.mustCall(() => {
// Env vars from the child thread are not set globally.
assert.strictEqual(process.env.SET_IN_WORKER, 'set');
}));
process.env.SET_IN_PARENT_AFTER_CREATION = 'set';
w.postMessage({});
} else {
assert.strictEqual(workerData, 'runInWorker');
// Env vars from the parent thread are inherited.
assert.strictEqual(process.env.SET_IN_PARENT, 'set');
process.env.SET_IN_WORKER = 'set';
assert.strictEqual(process.env.SET_IN_WORKER, 'set');
parentPort.once('message', common.mustCall(() => {
assert.strictEqual(process.env.SET_IN_PARENT_AFTER_CREATION, 'set');
}));
}

View File

@ -0,0 +1,54 @@
'use strict';
const common = require('../common');
const child_process = require('child_process');
const assert = require('assert');
const { Worker, workerData } = require('worker_threads');
// Test for https://github.com/nodejs/node/issues/24947.
if (!workerData && process.argv[2] !== 'child') {
process.env.SET_IN_PARENT = 'set';
assert.strictEqual(process.env.SET_IN_PARENT, 'set');
new Worker(__filename, { workerData: 'runInWorker' })
.on('exit', common.mustCall(() => {
// Env vars from the child thread are not set globally.
assert.strictEqual(process.env.SET_IN_WORKER, undefined);
}));
process.env.SET_IN_PARENT_AFTER_CREATION = 'set';
new Worker(__filename, {
workerData: 'resetEnv',
env: { 'MANUALLY_SET': true }
});
common.expectsError(() => {
new Worker(__filename, { env: 42 });
}, {
type: TypeError,
code: 'ERR_INVALID_ARG_TYPE',
message: 'The "options.env" property must be one of type object, ' +
'undefined, null, or worker_threads.SHARE_ENV. Received type number'
});
} else if (workerData === 'runInWorker') {
// Env vars from the parent thread are inherited.
assert.strictEqual(process.env.SET_IN_PARENT, 'set');
assert.strictEqual(process.env.SET_IN_PARENT_AFTER_CREATION, undefined);
process.env.SET_IN_WORKER = 'set';
assert.strictEqual(process.env.SET_IN_WORKER, 'set');
Object.defineProperty(process.env, 'DEFINED_IN_WORKER', { value: 42 });
assert.strictEqual(process.env.DEFINED_IN_WORKER, '42');
const { stderr } =
child_process.spawnSync(process.execPath, [__filename, 'child']);
assert.strictEqual(stderr.toString(), '', stderr.toString());
} else if (workerData === 'resetEnv') {
assert.deepStrictEqual(Object.keys(process.env), ['MANUALLY_SET']);
assert.strictEqual(process.env.MANUALLY_SET, 'true');
} else {
// Child processes inherit the parent's env, even from Workers.
assert.strictEqual(process.env.SET_IN_PARENT, 'set');
assert.strictEqual(process.env.SET_IN_WORKER, 'set');
}