Issue
I am working on an open source Ruby on Rails project with a full suite of rspec tests. The tests work normally on the CI/CD pipeline server, but locally on my MacBook running Monterey they fail intermittently. Sometimes zero tests will fail, sometimes a handful, sometimes hundreds — all with no code changes.
The error messages
The intermittently failing tests produce a variety of errors:
Errno::EMFILE:
Failed to open TCP connection to 127.0.0.1:9520 (Too many open files - socket(2) for "127.0.0.1" port 9520)
The above error appears with various port numbers. I have spotted 9515, 9517, 9518, 9520, 9522, 9529, 9533 and 9536.
I'm also seeing a lot of other errors that complain of too many open files:
Failure/Error: Capybara::Webmock.start Errno::EMFILE: Too many open files
Errno::EMFILE: Too many open files - rackup
Errno::EMFILE: Too many open files - /Users/username/.webdrivers/chromedriver
I think it likely that a single issue underlies all these intermittent errors. Search results for a lot of these errors lead to Chromedriver, Selenium, Capybara, and WebMock.
What I have tried
- I have increased the number of available file descriptors with
ulimit -Sn 61440
- Increased the files available to
launchctl limit maxfiles
- I have checked that my
chromedriver -v
corresponds to the Chrome browser version installed on my MacBook - I have cleared caches by running
bundle exec rails tmp:clear
andRails.cache.clear
, and by setting theCache-Control
header topublic, no-cache, must-revalidate
and other settings - I have added
net_http_connect_on_start: true
to my WebMock configuration, per their documentation - I have tried
killall chromedriver
to ensure there isn't a zombie chromedriver process
My code
My WebMock configuration should allow local connections as it includes the following:
require 'webmock/rspec'
WebMock.disable_net_connect!(
allow: [
/localhost/,
/127\.0\.0\.1/,
/codeclimate.com/, # For uploading coverage reports
/chromedriver\.storage\.googleapis\.com/, # For fetching a chromedriver binary
],
)
My Capybara configuration is here.
Solution
To answer my own question, I increased the available macOS file descriptors to solve this problem. MacOS apparently does not take you seriously when you increase the ulimit
from the command line; it demands a configuration file and a restart.
I'm not convinced I found the underlying issue — I suspect there may be some problem in my configuration causing file descriptors to be exhausted. Perhaps there are some connections that aren't being property closed. Yet, increasing the limit solves the immediate problem.
I created a property list file owned by root:
sudo nano /Library/LaunchDaemons/limit.maxfiles.plist
I paste this text into the file:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>launchctl</string>
<string>limit</string>
<string>maxfiles</string>
<string>524288</string>
<string>524288</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>ServiceIPC</key>
<false/>
</dict>
</plist>
I saved the file. Just to be sure, I printed the file to the terminal with cat
to confirm it exists. And I checked ls
to ensure that root wheel
owned the file.
cat /Library/LaunchDaemons/limit.maxfiles.plist
ls -la /Library/LaunchDaemons/limit.maxfiles.plist
I restarted my MacBook to cause the .plist to take effect. Then I checked the file descriptor limits again with:
ulimit -n
launchctl limit maxfiles
Both now return a limit of 524288. The "too many open files" messages are gone.
Further reading:
- El Capitan ulimit shenanigans by Dejan Kitic
- "Too many open files" limit/ulimit on Mac OS X by Thomas Jung
- Ruby 2.7.1 - Errno::EMFILE: Failed to open TCP connection to 127.0.0.1:9*** (Too many open files - socket(2) for "127.0.0.1" port 9***) by ndbroadbent
Answered By - John Skiles Skinner
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.