Manipulating Data from Oracle Object Storage to ADW with Oracle Data Integrator (ODI)
0. Introduction and Prerequisites
This article presents an overview on how to use Oracle Data Integrator in order to manipulate data from Oracle Cloud Infrastructure Object Storage. The scenarios here present loading the data from an object storage in Oracle Cloud Infrastructure and then move the data to Oracle Autonomous Data Warehouse (ADW[SR1] ).
This document could be a reference for customer have data storage in different regions and want to do the data integration& feed into Data warehouse.
Main steps are listed here:
1. Install ODI 12.2.1.3.0.
2. Patch p26669648_122130_Generic to upgrade ODI to version 12.2.1.3.1.
3. Set up Source Data Server/Physical Schema/Model in Object Storage.
4. Set up Target Data Server/Physical Schema/Model in ADW.
5. Creating a Mapping and test it.
You should have Object storage and ADW instance provisioned.
1. Install ODI 12.2.1.3.0
You can refer below link for reference.
https://docs.oracle.com/en/middleware/lifecycle/12.2.1.3/oding/installing-and-configuring-oracle-data-integrator.pdf
2. Patch p26669648_122130_Generic to upgrade to 12.2.1.3.1
You need to patch ODI to version 12.2.1.3.1 firstly.
3. Now you get ODI 12.2.1.3.1
4. New a Data Server
Let’s setup the topology. Right click Oracle Object Storage
An overview:
Let me explain the items above.
a. Region:
Oracle Object Storage region. A region is a localized geographic area, and an availability domain is one or more data centers located within a region. A region is composed of several availability domains. Most Oracle Cloud Infrastructure resources are either region-specific, such as a virtual cloud network, or availability domain-specific, such as a compute instance.
b. Tenant OCID:
Tenant’s Oracle Cloud ID. Every Oracle Cloud Infrastructure resource has an Oracle-assigned unique ID called an Oracle Cloud Identifier (OCID). It's included as part of the resource's information in both the Console and API. To find your tenancy's OCID. Go to Administration-> Tenancy Details.
c. User OCID:
Oracle Cloud ID of the user logging into Oracle Object Storage.
In the Console on the page showing the user's details. To get to that page:
- If you're signed in as the user, click the user icon present in the top-right corner of the Console, and then click User Settings.
- If you're an administrator doing this for another user, instead click Identity, click Users, and then select the user from the list.
User OCID: api.user
Security
d. Private Key File – Click the browse button to choose the location of the private key file (in PEM format)
Follow the steps to generate the private key and fingerprint
https://docs.cloud.oracle.com/iaas/Content/API/Concepts/apisigningkey.htm#How
- Passphrase – Passphrase is the password used while generating the private key
e. fingerprint
f. username:
Specify the user api.user, need to be same with item c. User OCID
Caution: Upload the public key to Object Storage.
You can upload the PEM public key in the Console, located at https://console.us-ashburn-1.oraclecloud.com. If you don't have a login and password for the Console, contact an administrator.
- Open the Console, and sign in.
- View the details for the user who will be calling the API with the key pair:
- If you're signed in as this user, click your username in the top-right corner of the Console, and then click User Settings.
- If you're an administrator doing this for another user, instead click Identity, click Users, and then select the user from the list.
- Click Add Public Key.
- Paste the contents of the PEM public key in the dialog box and click Add.
Test Connection
5. Creating an Oracle Object Storage Physical Schema
Create an Oracle Object Storage physical schema using the standard procedure, in Administering Oracle Data Integrator.
Oracle Object Storage specific parameters are:
- Name: Name of the physical schema created
- Bucket (Schema): It specifies the Oracle Object Storage Bucket name from which upload, download or the delete operation will happen. Select the required bucket from the Bucket Name drop-down list.
- Directory (Work Schema): This is the temporary folder on the local system used for getting files from Oracle Object Storage bucket during reverse engineering. If the directory does not exist it will be created. Specify the required location in the local system.
And the logical schema:
6. Creating and Reverse-Engineering an Oracle Object Storage Model
Creating an Oracle Object Storage Model
An Oracle Object Storage model is a set of data stores, corresponding to files stored in an Oracle Object Storage bucket. In a given context, the logical schema corresponds to one physical schema. You can create a model from the logical schema for the Oracle Object Storage technology. The bucket schema of this physical schema is the Oracle Object Storage bucket containing all the files. You can create new ODI Data store that will represent a file in Oracle Object Storage so that it can be used in mappings.
Input the information required and Save.
Reverse-Engineering Delimited Files from Oracle Object Storage
To perform a delimited file reverse engineering:
- In the Models accordion, right click your Object Storage Model and select New Data store. The Data Store Editor opens.
- In the Definition tab, enter the following fields:
- Name: Name of this data store
- Resource Name: Click the Search icon, to select the required file from the list of files present in Oracle Object Storage for the configured bucket.
- Go to the Storage tab, to describe the type of file. Set the fields as follows:
- File Format: Delimited
- Heading (Number of Lines): Enter the number of lines of the header. Note that if there is a header, Oracle Data Integrator uses the first line of the header to name the columns in the file.
- Select a Record Separator.
- Select or enter the character used as a Field Separator.
- Enter a Text Delimiter if your file uses one.
- Enter a Decimal Separator, if your file contains decimals.
- From the File main menu, select Save.
- In the Data Store Editor, go to the Attributes tab.
- In the editor toolbar, click Reverse Engineer.
Click Reverse Engineer, ODI will generate the Metadata based on the header of the file.
- Verify the data type and length for the reverse engineered attributes. Oracle Data Integrator infers the field data types and lengths from the file content, but may set default values (for example 50 for the strings field length) or incorrect data types in this process.
- From the File main menu, select Save.
7. Create a Connection with ADW
Create a Data Server for ADW. Specify the Credential file and choose the connection details from dropdown list.
JDBC information will be there, no need to update.
And Test the connection
And then new a Physical Schema.
New the Model and Reverse Engineer.
8. New a Project, Mapping and Test
Set the AP(Access Point) as below:
Caution: You need to run the store procedure to create credential on ADW before running the Mapping.
set define off
begin
DBMS_CLOUD.create_credential(
credential_name => 'ODI',
username => 'api.user',
password => '.};rKwO6t8***'
);
end;
/
set define on
Mapping run finished.
And Review the data loaded in ADW.
Comparing with the source csv file in Oracle Object Storage:
Looks good.
Conclusion:
With the Oracle Data Integrator 12c releases Oracle introduced several new enhancements, more source and target are supported (Oracle Object Storage, Oracle Autonomous Data Warehouse Cloud (ADW), Oracle Autonomous Transaction Processing (ATP), Oracle Enterprise Resource Planning (ERP) Cloud etc.). This document could help customer to achieve their data integration over different regions or oversea.
The ODI 12c releases continue to improve Oracle’s strategic Data Integration platform while preserving the key product differentiators: Declarative Design, Knowledge Modules, Hot-Plug-ability, and E-LT architecture.
Manipulating Data from Oracle Object Storage to ADW with Oracle Data Integrator (ODI)的更多相关文章
- Oracle Schema Objects(Schema Object Storage And Type)
One characteristic of an RDBMS is the independence of physical data storage from logical data struct ...
- [译]OpenStack Object Storage Monitoring
注:翻译的不完整,主要是有些地方翻译后反而妨碍理解,有些不知道怎么翻,anyway,需要时拿来用用也是可行的,顺便共享啦.欢迎提意见. 一个OpenStack Object Storage(OSOS) ...
- centos6.4 ceph安装部署之ceph object storage
preface: ceph-deploy does not provide a rapid installation for Ceph Object Storage install Configura ...
- [转]Build An Image Manager With NativeScript, Node.js, And The Minio Object Storage Cloud
本文转自:https://www.thepolyglotdeveloper.com/2017/04/build-image-manager-nativescript-node-js-minio-obj ...
- golang 操作ceph object storage
ceph的object storage 提供了和amazon s3兼容的接口以供客户访问. 在ceph的官网上,可以看到它提供了多种语言的访问范本,例如python的(http://docs.ceph ...
- js & h5 & app & object storage
js & h5 & app & object storage API https://developer.mozilla.org/en-US/docs/Web/API Stor ...
- Oracle客户端工具出现“Cannot access NLS data files or invalid environment specified”错误的解决办法
Oracle客户端工具出现"Cannot access NLS data files or invalid environment specified"错误的解决办法 方法一:参考 ...
- 问题-Error creating object. Please verify that the Microsoft Data Access Components 2.1(or later) have been properly installed.
问题现象:软件在启动时报如下错误信息:Exception Exception in module zhujiangguanjia.exe at 001da37f. Error creating obj ...
- openStack 对象存储object storage swift
随机推荐
- H3C 域名解析显示及维护
- 51nod 挑剔的美食家
挑剔的美食家 基准时间限制:1 秒 空间限制:131072 KB 分值: 5 与很多奶牛一样,Farmer John那群养尊处优的奶牛们对食物越来越挑剔,随便拿堆草就能打发她们午饭的日子自然是一 ...
- WindowsDOS命令添加/创建/修改/删除服务
添加服务 sc <server> create [service name] [binPath= ] <option1> <option2>... 在注册表和服务数 ...
- vue-learning:12 - 2 - 区分:outerHTML - innerTHML - outerText - innerText - textContent
区分:outerHTML - innerTHML - outerText - innerText - textContent 获取值 <div id="outer"> ...
- 一款类似loadRunner的优秀国产压力测试工具——kylinTOP测试与监控平台
市面上流行的压力/负载/性能测试工具多是来自国外,近年来国内的性能测试工具也如雨后春笋般崛起,但大部分产品是基于Jmeter开源内核包装起来的性能测试工具,其中也不乏佼佼者,如:kylinTOP测试与 ...
- python tkinter动态追加按钮等控件可能遇到的问题
小爬最近给同事制作一个小爬虫:具体要求: 1.每天自动定时触发: 2.模拟用户自动登陆: 3.自动爬取对应API接口数据: 4.对爬取结果进行逻辑判断,对符合条件的数据进行规则化列示: 5.列示的行项 ...
- Spring Boot 单元测试示例
Spring 框架提供了一个专门的测试模块(spring-test),用于应用程序的单元测试. 在 Spring Boot 中,你可以通过spring-boot-starter-test启动器快速开启 ...
- java做插入时ID为自增获取到ID
<selectKey keyProperty="id" resultType="int" order="AFTER"> sele ...
- (openssh、telnet、vsftpd、nfs、rsync、inotify、samba)
(openssh.telnet.vsftpd.nfs.rsync.inotify.samba) 一:OpenSSH服务与Telnet服务(必须掌握) 前言:OpenSSH是加密传输,Telnet是明文 ...
- .Net Core Web Api实践(二).net core+Redis+IIS+nginx实现Session共享
前言:虽说公司app后端使用的是.net core+Redis+docker+k8s部署的,但是微信公众号后端使用的是IIS部署的,虽说公众号并发量不大,但领导还是使用了负载均衡,所以在介绍docke ...