标签：opt dremio fs 21.1 demo dist COPY true
主要演示下，具体参数以前我有说明过
环境准备
- dockerfile
FROM dremio/dremio-oss:21.1
USER root
COPY dremio-env-profile /opt/dremio/conf/dremio-env
COPY --from=hengyunabc/arthas:latest /opt/arthas /opt/arthas
COPY core-site.xml /opt/dremio/conf/core-site.xml
COPY dremio.conf /opt/dremio/conf/dremio.conf
COPY jprofiler_linux_12_0_4.deb /opt/jprofiler_linux_12_0_4.deb
RUN dpkg -i /opt/jprofiler_linux_12_0_4.deb
COPY my-mysql-19.0.0-202110270341120595-3e3e05e7.jar /opt/dremio/jars/my-mysql-19.0.0-202110270341120595-3e3e05e7.jar
COPY mysql-connector-java-5.1.47.jar /opt/dremio/jars/3rdparty/mysql-connector-java-5.1.47.jar
COPY mysql-connector-java-8.0.20.jar /opt/dremio/jars/3rdparty/mysql-connector-java-8.0.20.jar
USER dremio
- core-site.xml
<?xml version="1.0"?>
<configuration>
<property>
<name>fs.dremioS3.impl</name>
<description>The FileSystem implementation. Must be set to com.dremio.plugins.s3.store.S3FileSystem</description>
<value>com.dremio.plugins.s3.store.S3FileSystem</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<description>key</description>
<value>minio</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>secret</description>
<value>minio123</value>
</property>
<property>
<name>fs.s3a.aws.credentials.provider</name>
<description>The credential provider type.</description>
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
</property>
<property>
<name>fs.s3a.endpoint</name>
<description>Endpoint can either be an IP or a hostname, where Minio server is running . However the endpoint value cannot contain the http(s) prefix. E.g. 175.1.2.3:9000 is a valid endpoint. </description>
<value>minio:9000</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>dremio.s3.compat</name>
<description>Value has to be set to true.</description>
<value>true</value>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<description>Value can either be true or false, set to true to use SSL with a secure Minio server.</description>
<value>false</value>
</property>
</configuration>
- dremio.conf
paths: {
# the local path for dremio to store data.
local: ${DREMIO_HOME}"/data"
# the distributed path Dremio data including job results, downloads, uploads, etc
#dist: "pdfs://"${paths.local}"/pdfs"
dist: "dremioS3:///myappdemo/app1/"
accelerator: ${paths.dist}/accelerator,
downloads: ${paths.dist}/downloads,
uploads: ${paths.dist}/uploads,
results: ${paths.dist}/results
scratch: ${paths.dist}/scratch
}
services: {
coordinator.enabled: true,
coordinator.master.enabled: true,
executor.enabled: true,
flight.use_session_service: true
}
- docker-compose 文件
version: '3'
services:
pg:
image: postgres:14.3-alpine3.15
environment:
- POSTGRES_PASSWORD=dalongdemo
ports:
- 5432:5432
minio:
image: minio/minio
ports:
- "9002:9000"
- "19001:19001"
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
command: server --console-address :19001 --quiet /data
dremio:
build: ./
image: dalongrong/dremio:21.1-jprofiler
ports:
- "9047:9047"
- "8849:8849"
- "31010:31010"
s3 效果
具体配置比较简单，我只截图说明下，因为21.1 之后反射必须使用分布式存储了
说明
以上只是演示下，因为21.1 必须使用分布式存储了，所以后续必须升级了，刚好进行下功能测试
参考资料
https://docs.dremio.com/software/advanced-administration/dremio-conf/
https://docs.dremio.com/software/deployment/dist-store-config/
https://github.com/rongfengliang/dremio-s3-dist-store
标签：opt,dremio,fs,21.1,demo,dist,COPY,true 来源： https://www.cnblogs.com/rongfengliang/p/16303869.html
本站声明： 1. iCode9 技术分享网（下文简称本站）提供的所有内容，仅供技术学习、探讨和分享； 2. 关于本站的所有留言、评论、转载及引用，纯属内容发起人的个人观点，与本站观点和立场无关； 3. 关于本站的所有言论和文字，纯属内容发起人的个人观点，与本站观点和立场无关； 4. 本站文章均是网友提供，不完全保证技术分享内容的完整性、准确性、时效性、风险性和版权归属；如您发现该文章侵犯了您的权益，可联系我们第一时间进行删除； 5. 本站为非盈利性的个人网站，所有内容不会用来进行牟利，也不会利用任何形式的广告来间接获益，纯粹是为了广大技术爱好者提供技术内容和技术思想的分享性交流网站。