Trying to connect to Min.io on-prem S3 bucket. Confirmed API is working:
However, I don’t know how to point to https://nas2.piorl.us:9000/bucket1/ in CDataSync Cross-Platform Edition. I’ve tried defining the full path in the Bucket field as well as “Path” in advanced options, but neither want to work. There is a reference to a “Custom URL” field in the advanced connection properties, but I can’t find the variable anywhere:
Thanks.
Page 1 / 1
Hi @joshphillipssr
While we have had to test scenarios with Mini.io instances before, we have not tested this specifically in Sync. However we would be happy to look into this.
One thing you could try is navigating down to the Other property in the Advanced settings and setting the following:
StorageBaseURL=https://nas2.piorl.us:9000
It is important to specify the URL in the defined format http://<HOSTNAME>:9000 and set the name of the bucket you want to connect to in the Bucket property. Also, ensure that you are selecting the correct AWS Region when testing the connection.
Let me know if it works for you.
Since the AWS Region is list only, I’m assuming I’ll have to set an “AWS” region in Min.io so they will match?
Almost got it. Had to set Virtual Hosting to False (though I could have created a DNS entry instead I guess). Now I’m getting a “authorization header is malformed; the region is wrong”
Here’s the full log where is seems region is matching up:
2023-12-19T17:01:02.422-0500 3 -HTTP] Request Sent 2023-12-19T17:01:02.422-0500 3 *HTTP] Response Received 2023-12-19T17:01:02.438-0500 3 hHTTP] Status Line: HTTP/1.1 400 Bad Request 2023-12-19T17:01:02.438-0500 3 2HTTP] Accept-Ranges: bytes 2023-12-19T17:01:02.438-0500 3 tHTTP] Content-Length: 426 2023-12-19T17:01:02.439-0500 3 tHTTP] Content-Type: application/xml 2023-12-19T17:01:02.439-0500 3 HTTP] Server: MinIO 2023-12-19T17:01:02.439-0500 3 bHTTP] Strict-Transport-Security: max-age=31536000; includeSubDomains 2023-12-19T17:01:02.440-0500 3 RHTTP] Vary: Origin 2023-12-19T17:01:02.440-0500 3 �HTTP] Vary: Accept-Encoding 2023-12-19T17:01:02.440-0500 3 �HTTP] X-Amz-Bucket-Region: NORTHERNVIRGINIA 2023-12-19T17:01:02.441-0500 3 /HTTP] X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 2023-12-19T17:01:02.441-0500 3 PHTTP] X-Amz-Request-Id: 17A25AA25936CDF3 2023-12-19T17:01:02.442-0500 3 HTTP] X-Content-Type-Options: nosniff 2023-12-19T17:01:02.442-0500 3 4HTTP] X-Xss-Protection: 1; mode=block 2023-12-19T17:01:02.443-0500 3 0HTTP] Date: Tue, 19 Dec 2023 22:01:02 GMT 2023-12-19T17:01:02.443-0500 3 2HTTP] <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region is wrong; expecting 'NORTHERNVIRGINIA'.</Message><BucketName>bucket1</BucketName><Resource>/bucket1/</Resource><Region>NORTHERNVIRGINIA</Region><RequestId>17A25AA25936CDF3</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error> 2023-12-19T17:01:02.443-0500 3 -HTTP] Transfer Complete 2023-12-19T17:01:02.444-0500 2 3HTTP] Response: HTTP/1.1 400 Bad Request, 426 Bytes Transferred 2023-12-19T17:01:02.446-0500 2 AuthorizationHeaderMalformed: The authorization header is malformed; the region is wrong; expecting 'NORTHERNVIRGINIA'. XcoreXamazons3syncX230X8669.kde at XcoreXamazons3syncX230X8669.at.a(Unknown Source) at XcoreXamazons3syncX230X8669.ct.a(Unknown Source) at XcoreXamazons3syncX230X8669.at.a(Unknown Source) at XcoreXamazons3syncX230X8669.ct.a(Unknown Source) at XcoreXamazons3syncX230X8669.ct.c(Unknown Source) at XcoreXamazons3syncX230X8669.at.q(Unknown Source) at XcoreXamazons3syncX230X8669.at.ag(Unknown Source) at XcoreXamazons3syncX230X8669.uwd.i(Unknown Source) at XcoreXamazons3syncX230X8669.tee.a(Unknown Source) at XcoreXamazons3syncX230X8669.tee.a(Unknown Source) at XcoreXamazons3syncX230X8669.yvc.a(Unknown Source) at XcoreXamazons3syncX230X8669.yvc.a(Unknown Source) at XcoreXamazons3syncX230X8669.yvc.a(Unknown Source) at XcoreXamazons3syncX230X8669.itc.a(Unknown Source) at XcoreXamazons3syncX230X8669.itc.a(Unknown Source) at XcoreXamazons3syncX230X8669.hxc.b(Unknown Source) at XcoreXamazons3syncX230X8669.hxc.a(Unknown Source) at XcoreXamazons3syncX230X8669.hxc.a(Unknown Source) at XcoreXamazons3syncX230X8669.hxc.a(Unknown Source) at XcoreXamazons3syncX230X8669.muc.b(Unknown Source) at XcoreXamazons3syncX230X8669.jub.a(Unknown Source) at XcoreXamazons3syncX230X8669.jub.a(Unknown Source) at XcoreXamazons3syncX230X8669.lxc.b(Unknown Source) at XcoreXamazons3syncX230X8669.jzc.b(Unknown Source) at XcoreXamazons3syncX230X8669.stc.a(Unknown Source) at XcoreXamazons3syncX230X8669.rxc.g(Unknown Source) at XcoreXamazons3syncX230X8669.zuc.g(Unknown Source) at XcoreXamazons3syncX230X8669.vwb.g(Unknown Source) at XcoreXamazons3syncX230X8669.dpd.r(Unknown Source) at XcoreXamazons3syncX230X8669.dpd.g(Unknown Source) at XcoreXamazons3syncX230X8669.fac.g(Unknown Source) at XcoreXamazons3syncX230X8669.xub.g(Unknown Source) at XcoreXamazons3syncX230X8669.kqb.n(Unknown Source) at cdata.jdbc.amazons3sync.AmazonS3SyncConnection.testConnection(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at XcoreXcdata.realdb.zrd.testConnection(Unknown Source) at XcoreXcdata.syncX233X8700.fha.testConnection(Unknown Source) at XcoreXcdata.syncX233X8700.gha.c(Unknown Source) at sync.core.QueryExec$4.a(Unknown Source) at sync.core.QueryExec.a(Unknown Source) at sync.core.QueryExec.g(Unknown Source) at rssbus.syncops.CallStackKeeper.TestConnection(Unknown Source) at rssbus.syncops.TestConnection.exec(Unknown Source) at XcoreXcdata.syncX233X8700.wra.a(Unknown Source) at XcoreXcdata.syncX233X8700.wra.a(Unknown Source) at XcoreXcdata.syncX233X8700.wra.a(Unknown Source) at XcoreXcdata.syncX233X8700.ipa.a(Unknown Source) at rssbus.src.testConnection_rsb.a(Unknown Source) at XcoreXcdata.syncX233X8700.eta.Exec(Unknown Source) at rssbus.src.testConnection_rsb.execRSB(Unknown Source) at XcoreXcdata.syncX233X8700.ppa.execRSB(Unknown Source) at XcoreXcdata.syncX233X8700.zra.a(Unknown Source) at XcoreXcdata.syncX233X8700.cra.a(Unknown Source) at XcoreXcdata.syncX233X8700.cra.a(Unknown Source) at XcoreXcdata.syncX233X8700.spa.call(Unknown Source) at XcoreXcdata.syncX233X8700.mma.call(Unknown Source) at XcoreXcdata.syncX233X8700.mma.call(Unknown Source) at rssbus.RSBScript.a(Unknown Source) at rssbus.RSBScript.a(Unknown Source) at rssbus.RSBScript.service(Unknown Source) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1410) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at b.c.doFilter(Unknown Source) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:538) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558) at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379) at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) at java.base/java.lang.Thread.run(Unknown Source) 2023-12-19T17:01:02.462-0500 5 j |Q-Id] META|Schema] Engine Invalid object name 'sys_disconnect' 2023-12-19T17:01:02.462-0500 4 36|Q-Id] �INFO|Connec] Executed sys_disconnect: Success: (0 ms) 2023-12-19T17:01:02.462-0500 1 p6|Q-Id] .INFO|Connec] Closed AmazonS3Sync connection
Hi @joshphillipssr
In the screenshot you have provided it looks like you are setting the AWS Region to “us-orl-nas2” when testing the API in Postman. If this is the correct Region, can you try adding the same in the Other property:
That did the trick. I had tried renaming the Minio Region to NORTHERNVIRGINIA so they would match up, then tried changing it in both Minio and CDataSync to “UAE” just in case it was an issue with the length, same error. Then renamed back to us-orl-nas2 in Minio and added the region variable above, and viola.
Glad to hear that worked for you. Generally speaking the driver should autodetect the AWS Region in most cases however since this is a Mini.io instance I believe the driver has failed to autodetect the region for some reason.
In case you were wondering, Region is a hidden property that can be used in scenarios like this one where the driver is unable to detect the AWS Region automatically from the URL provided which means we can get around this by explicitly setting the Region property in Other.