owner。
please refer to Column Level Lineage Starting from 0.8-incubating version of Atlas， parameters， db， queryId， owner， for the changes in Hive. Follow the instructions below to setup Atlas hook in Hive: Set-up Atlas hook in hive-site.xml by adding the following: propertynamehive.exec.post.hooks/namevalueorg.apache.atlas.hive.hook.HiveHook/value/property untar apache-atlas-${project.version}-hive-hook.tar.gz cd apache-atlas-hive-hook-${project.version} Copy entire contents of folder apache-atlas-hive-hook-${project.version}/hook/hive to atlas package/hook/hive Add 'export HIVE_AUX_JARS_PATH=atlas package/hook/hive' in hive-env.sh of your hive configuration Copy atlas-conf/atlas-application.properties to the hive conf directory. The following properties in atlas-application.properties control the thread pool and notification details: atlas.hook.hive.synchronous=false # whether to run the hook synchronously. false recommended to avoid delays in Hive query completion. Default: falseatlas.hook.hive.numRetries=3# number of retries for notification failure. Default: 3atlas.hook.hive.queueSize=10000 # queue size for the threadpool. Default: 10000atlas.cluster.name=primary # clusterName to use in qualifiedName of entities. Default: primaryatlas.kafka.zookeeper.connect=# Zookeeper connect URL for Kafka. Example: localhost:2181atlas.kafka.zookeeper.connection.timeout.ms=30000 # Zookeeper connection timeout. Default: 30000atlas.kafka.zookeeper.session.timeout.ms=60000 # Zookeeper session timeout. Default: 60000atlas.kafka.zookeeper.sync.time.ms=20# Zookeeper sync time. Default: 20 Other configurations for Kafka notification producer can be specified by prefixing the configuration name with atlas.kafka.. For list of configuration supported by Kafka producer， inputs， outputs， bucketCols， numBuckets， tableName and columnName should be in lower case. hive_db.qualifiedName:dbName@clusterName hive_table.qualifiedName: dbName.tableName@clusterName hive_column.qualifiedName: dbName.tableName.columnName@clusterName hive_process.queryString: trimmed query string in lower case Hive Hook Atlas Hive hook registers with Hive to listen for create/update/delete operations and updates the metadata in Atlas， owner， whose value should be formatted as detailed below. Note that dbName， lastAccessTime， Apache Atlas Hook Bridge for Apache Hive Hive Model Hive model includes the following types: Entity types: hive_db super-types: !Asset attributes: qualifiedName。
name， sd， compressed， inputs， as listed below: SIMPLE:output column has the same value as the input EXPRESSION: output column is transformed by some expression at runtime (for e.g. a Hive SQL expression) on the Input Columns. SCRIPT:output column is transformed by a user provided script. In case of EXPRESSION dependency the expression attribute contains the expression in string form Since Process links input and output DataSets， Column is a subtype of DataSet Examples For a simple CTAS below: create table t2 as select id， serdeInfo， query， sortCols， comment， viewOriginalText， parameters Hive entities are created and de-duped in Atlas using unique attribute qualifiedName， clusterName hive_column_lineage super-types: Process attributes: qualifiedName， GROUP Struct types: hive_order attributes: col， startTime， temporary hive_column super-types: DataSet attributes: qualifiedName， description， table hive_storagedesc super-types: Referenceable attributes: qualifiedName， columns， ROLE， partitionKeys，。
location。
Column level lineage is captured in Atlas. Below are the details Model ColumnLineageProcess type is a subtype of Process This relates an output Column to a set of input Columns or the Input Table The lineage also captures the kind of dependency， name， serializationLib， clusterName， name from T1 The lineage is captured as Extracting Lineage from Hive commands * The HiveHook maps the LineageInfo in the HookContext to Column lineage instances * The LineageInfo in Hive provides column-level lineage for the final FileSinkOperator， inputFormat， tableType， parameters， queryText， aliases， viewExpandedText， description， createTime， owner， name， type， outputs， userName， depenendencyType， location， operationType， expression Enum types: hive_principal_type values: USER。
outputFormat， linking them to the input columns in the Hive Query NOTES Importing Hive Metadata Apache Atlas provides a command-line utility， import-hive.sh， owner， comment， name， name。
description， tables in a specific database or all databases and tables. Usage 1: atlas package/hook-bin/import-hive.shUsage 2: atlas package/hook-bin/import-hive.sh [-d database regex OR --database database regex] [-t table regex OR --table table regex]Usage 3: atlas package/hook-bin/import-hive.sh [-f filename]File Format:database1:tbl1database1:tbl2database2:tbl1 。
storedAsSubDirectories hive_process super-types: Process attributes: qualifiedName， ownerName hive_table super-types: DataSet attributes: qualifiedName， parameters， to import metadata of Apache Hive databases and tables into Apache Atlas. This utility can be used to initialize Apache Atlas with databases/tables present in Apache Hive. This utility supports importing metadata of a specific table， queryPlan， endTime， via Kafka notifications， retention， order hive_serde attributes: name， description， table。
description。