Skip to content

S3 container sometimes doesn't resolve from PHP container #761

Description

@mikelittle

Sometimes, whether from a brand-new stack or an existing stack, uploading images results in an error that boils down to the PHP container being unable to resolve the MinIO s3 emulation container.

Here's a brain dump of notes from a brief investigation into the problem. Note, sometimes the S3 container does resolve, but the Tachyon one doesn't. This has only occurred on one machine. A previous macBook does not exhibit the problem (but then my current one didn't used to either). It doesn't occur (for me) on Linux (Ubuntu 24.04).

  • Found fatal error uploading an image as author
qatest-php  | [07-Aug-2024 09:35:43 UTC] PHP Warning:  Uncaught exception 'Aws\S3\Exception\S3Exception' with message 'Error executing "ListObjectsV2" on "https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777"; AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777'
qatest-php  |
qatest-php  | GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777 in /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:275
qatest-php  | Stack trace:
qatest-php  | #0 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(205): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
qatest-php  | #1 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(157): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
qatest-php  | #2 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(267): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
qatest-php  | #3 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(180): GuzzleHttp\Handler\CurlMultiHandler->processMessages()
qatest-php  | #4 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(206): GuzzleHttp\Handler\CurlMultiHandler->tick()
qatest-php  | #5 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
qatest-php  | #6 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
qatest-php  | #7 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #8 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #9 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #10 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #11 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #12 /usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
qatest-php  | #13 /usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute(Object(Aws\Command))
qatest-php  | #14 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
qatest-php  | #15 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #16 [internal function]: Aws\flatmap(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #17 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
qatest-php  | #18 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
qatest-php  | #19 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php(676): scandir('s3://s3-qatest/...')
qatest-php  | #20 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): S3_Uploads\Plugin->get_files_for_unique_filename_file_list(NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #21 /usr/src/app/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters(NULL, Array)
qatest-php  | #22 /usr/src/app/wordpress/wp-includes/functions.php(2667): apply_filters('pre_wp_unique_f...', NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #23 /usr/src/app/wordpress/wp-admin/includes/file.php(984): wp_unique_filename('s3://s3-qatest/...', 'zed1-img_1777.j...', NULL)
qatest-php  | #24 /usr/src/app/wordpress/wp-admin/includes/file.php(1105): _wp_handle_upload(Array, Array, '2024-08-07 09:3...', 'wp_handle_uploa...')
qatest-php  | #25 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(1322): wp_handle_upload(Array, Array, '2024-08-07 09:3...')
qatest-php  | #26 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(269): WP_REST_Attachments_Controller->upload_from_file(Array, Array, '2024-08-07 09:3...')
qatest-php  | #27 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(158): WP_REST_Attachments_Controller->insert_attachment(Object(WP_REST_Request))
qatest-php  | #28 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1230): WP_REST_Attachments_Controller->create_item(Object(WP_REST_Request))
qatest-php  | #29 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1063): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/wp/v2/media', Array, NULL)
qatest-php  | #30 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(439): WP_REST_Server->dispatch(Object(WP_REST_Request))
qatest-php  | #31 /usr/src/app/wordpress/wp-includes/rest-api.php(420): WP_REST_Server->serve_request('/wp/v2/media')
qatest-php  | #32 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): rest_api_loaded(Object(WP))
qatest-php  | #33 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters(false, Array)
qatest-php  | #34 /usr/src/app/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
qatest-php  | #35 /usr/src/app/wordpress/wp-includes/class-wp.php(418): do_action_ref_array('parse_request', Array)
qatest-php  | #36 /usr/src/app/wordpress/wp-includes/class-wp.php(813): WP->parse_request('')
qatest-php  | #37 /usr/src/app/wordpress/wp-includes/functions.php(1336): WP->main('')
qatest-php  | #38 /usr/src/app/wordpress/wp-blog-header.php(16): wp()
qatest-php  | #39 /usr/src/app/index.php(21): require('/usr/src/app/wo...')
qatest-php  | #40 {main}
qatest-php  |
qatest-php  | Next Aws\S3\Exception\S3Exception: Error executing "ListObjectsV2" on "https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777"; AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/?list-type=2&delimiter=%2F&prefix=uploads%2F2024%2F08%2Fzed1-img_1777 in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:196
qatest-php  | Stack trace:
qatest-php  | #0 /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(97): Aws\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\Psr7\Request), Object(Aws\Command), Array)
qatest-php  | #1 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(209): Aws\WrappedHttpHandler->Aws\{closure}(Array)
qatest-php  | #2 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\Promise\Promise::callHandler(2, Array, NULL)
qatest-php  | #3 /usr/src/app/vendor/guzzlehttp/promises/src/RejectedPromise.php(49): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}(Array)
qatest-php  | #4 /usr/src/app/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}()
qatest-php  | #5 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(167): GuzzleHttp\Promise\TaskQueue->run()
qatest-php  | #6 /usr/src/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(206): GuzzleHttp\Handler\CurlMultiHandler->tick()
qatest-php  | #7 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
qatest-php  | #8 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
qatest-php  | #9 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #10 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #11 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #12 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
qatest-php  | #13 /usr/src/app/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
qatest-php  | #14 /usr/src/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
qatest-php  | #15 /usr/src/app/vendor/aws/aws-sdk-php/src/ResultPaginator.php(138): Aws\AwsClient->execute(Object(Aws\Command))
qatest-php  | #16 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(52): Aws\ResultPaginator->valid()
qatest-php  | #17 /usr/src/app/vendor/aws/aws-sdk-php/src/functions.php(69): Aws\map(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #18 [internal function]: Aws\flatmap(Object(Aws\ResultPaginator), Object(Closure))
qatest-php  | #19 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-stream-wrapper.php(695): Generator->valid()
qatest-php  | #20 [internal function]: S3_Uploads\Stream_Wrapper->dir_readdir()
qatest-php  | #21 /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php(676): scandir('s3://s3-qatest/...')
qatest-php  | #22 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): S3_Uploads\Plugin->get_files_for_unique_filename_file_list(NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #23 /usr/src/app/wordpress/wp-includes/plugin.php(205): WP_Hook->apply_filters(NULL, Array)
qatest-php  | #24 /usr/src/app/wordpress/wp-includes/functions.php(2667): apply_filters('pre_wp_unique_f...', NULL, 's3://s3-qatest/...', 'zed1-img_1777.j...')
qatest-php  | #25 /usr/src/app/wordpress/wp-admin/includes/file.php(984): wp_unique_filename('s3://s3-qatest/...', 'zed1-img_1777.j...', NULL)
qatest-php  | #26 /usr/src/app/wordpress/wp-admin/includes/file.php(1105): _wp_handle_upload(Array, Array, '2024-08-07 09:3...', 'wp_handle_uploa...')
qatest-php  | #27 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(1322): wp_handle_upload(Array, Array, '2024-08-07 09:3...')
qatest-php  | #28 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(269): WP_REST_Attachments_Controller->upload_from_file(Array, Array, '2024-08-07 09:3...')
qatest-php  | #29 /usr/src/app/wordpress/wp-includes/rest-api/endpoints/class-wp-rest-attachments-controller.php(158): WP_REST_Attachments_Controller->insert_attachment(Object(WP_REST_Request))
qatest-php  | #30 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1230): WP_REST_Attachments_Controller->create_item(Object(WP_REST_Request))
qatest-php  | #31 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1063): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/wp/v2/media', Array, NULL)
qatest-php  | #32 /usr/src/app/wordpress/wp-includes/rest-api/class-wp-rest-server.php(439): WP_REST_Server->dispatch(Object(WP_REST_Request))
qatest-php  | #33 /usr/src/app/wordpress/wp-includes/rest-api.php(420): WP_REST_Server->serve_request('/wp/v2/media')
qatest-php  | #34 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(324): rest_api_loaded(Object(WP))
qatest-php  | #35 /usr/src/app/wordpress/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters(false, Array)
qatest-php  | #36 /usr/src/app/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
qatest-php  | #37 /usr/src/app/wordpress/wp-includes/class-wp.php(418): do_action_ref_array('parse_request', Array)
qatest-php  | #38 /usr/src/app/wordpress/wp-includes/class-wp.php(813): WP->parse_request('')
qatest-php  | #39 /usr/src/app/wordpress/wp-includes/functions.php(1336): WP->main('')
qatest-php  | #40 /usr/src/app/wordpress/wp-blog-header.php(16): wp()
qatest-php  | #41 /usr/src/app/index.php(21): require('/usr/src/app/wo...')
qatest-php  | #42 {main}
qatest-php  |   thrown in /usr/src/app/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php on line 196
qatest-php  | [07-Aug-2024 09:35:43 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 41943040 bytes) in /usr/src/app/vendor/humanmade/s3-uploads/inc/class-plugin.php on line 676

  • Further testing.
  • Tried connecting to S3 from the docker shell - didn't work. Same from v19
$ composer server ssh
www-data@a7c1f17b35b0:/usr/src/app$ curl "https://altis19.altis.dev/s3-altis19/"
curl: (7) Failed to connect to altis19.altis.dev port 443 after 42 ms: Couldn't connect to server
www-data@a7c1f17b35b0:/usr/src/app$
  • testing inside qa-test v20 gave this
www-data@08674edaddfe:/usr/src/app$ curl -if 'http://s3:9000/minio/health/live'
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
X-Amz-Bucket-Region: us-east-1
X-Amz-Request-Id: 17E97BFE1103A8B1
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Date: Wed, 07 Aug 2024 15:31:03 GMT

www-data@08674edaddfe:/usr/src/app$
  • testing qatest v20 again. FQDN for S3 does not work
www-data@08674edaddfe:/usr/src/app$ curl "http://s3-qatest.altis.dev:9000/s3-qatest/"
curl: (7) Failed to connect to s3-qatest.altis.dev port 9000 after 4 ms: Couldn't connect to server
  • but short name does work
www-data@08674edaddfe:/usr/src/app$ curl "http://s3:9000/s3-qatest/"
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>s3-qatest</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter></Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>www-data@08674edaddfe:/usr/src/app$
  • same problem with qatest. Also, s3-uploads command broken too
$ composer server cli -- s3-uploads upload-directory tmp uploads
Error: Error executing "PutObject" on "https://s3-qatest.altis.dev/s3-qatest/uploads/zed1-img_1922.jpg"; 
  AWS HTTP error: cURL error 7: Failed to connect to s3-qatest.altis.dev port 443 after 0 ms: 
  Couldn\'t connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://s3-qatest.altis.dev/s3-qatest/uploads/zed1-img_1922.jpg

Acceptance criteria:

  • ...

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugExisting functionality isn't behaving as expecteddeveloper advocacyDeveloper Advocacy related tasksmust haveMust be done, high priority

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions