본문 바로가기
Study/Bigdata

HDP3 클러스터에 HDF(nifi)설치

redyuika 2018. 8. 22. 20:58


HDP3 되면서, 많은것이 바뀌었습니다


일단 눈에 띄는 변화는 Hadoop3 이 들어갔다는것과 제눈에 볼때 딱 달라진건 Falcon 이 없어진것


그리고 Flume 이 없어졌다는


https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_release-notes/content/deprecated_items.html

Deprecated Components and Product Capabilities

The following components are marked deprecated from HDP and will be removed in a future HDP release:

Component or CapabilityStatusMarked Deprecated as ofTarget Release for RemovalComments
Apache FalconDeprecatedHDP 2.6.0HDP 3.0.0Contact your Hortonworks account team for the replacement options.
Apache FlumeDeprecatedHDP 2.6.0HDP 3.0.0Consider Hortonworks DataFlow as an alternative for Flume use cases.
Apache MahoutDeprecatedHDP 2.6.0HDP 3.0.0Consider Apache Spark as an alternative depending on the workload.
Apache SliderDeprecatedHDP 2.6.0HDP 3.0.0Apache Slider functionality will be absorbed by Apache YARN.
CascadingDeprecatedHDP 2.6.0HDP 3.0.0 
HueDeprecatedHDP 2.6.0HDP 3.0.0Consider Ambari Views as the alternative.


저기서 Flume 이 없어졌는데 HDP3 되면서 싹 없애버렸네요.

사실 Nifi 가 나오면서 그 자리를 Nifi가 대체해버린다고 했는데, 정작 Nifi 는 보이질 않습니다 (HDP 내에서)


엄밀히 말하면, nifi는 HDF(Hortonworks Data Flow)라는 새로운 플랫폼으로 제공이 되고 있습니다.(물론 호튼웍스에서 제공하는 RPM으로 깔아도 됩니다.)


하지만 NIFI를 쓰고 싶은데, 굳이 Ambari에서 관리하고 싶으니 .. 


HDF를 설치하면 똑같이 Ambari에 Storm, Nifi가 들어간 플랫폼이 설치가 됩니다. 하지만 원하는건 HDP에 그대로 설치하는것이니 


물론 Hortonworks 에서 관련옵션을 제공합니다 


https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/planning-your-deployment/content/deployment_scenarios.html

ScenarioInstallation ScenarioSteps

Installing an HDF Cluster

This scenario applies if you want to install the entire HDF platform, consisting of all flow management and stream processing components on a new cluster.

The stream processing components include the new Streaming Analytics Manager (SAM) modules that are in GA (General Availability). This includes the SAM Stream Builder and Stream Operations modules but does not include installing the technical preview version of SAM Stream Insight, which is powered by Druid and Superset.

This scenario requires that you install an HDF cluster.

  1. Install Ambari.
  2. Install databases.
  3. Install the HDF management pack.
  4. Install an HDF cluster using Ambari.

Installing HDF Services on a New HDP Cluster

This scenario applies to you if you are both an Hortonworks Data Platform (HDP) and HDF customer and you want to install a fresh cluster of HDP and add HDF services.

The stream processing components include the new (SAM) and all of its modules. This includes installing the technical preview version of the SAM Stream Insight module, which is powered by Druid and Apache Superset.

This scenario requires that you install both an HDF cluster and an HDP cluster.

  1. Install Ambari.
  2. Install databases.
  3. Install an HDP cluster using Ambari.
  4. Install the HDF management pack.
  5. Update the HDF base URL.
  6. Add HDF services to an HDP cluster

Installing HDF Services on an Existing HDP Cluster

You have an existing HDP cluster with Apache Storm and or Apache Kafka services and want to install Apache NiFi or NiFi Registry modules on that cluster.

This requires that you upgrade to the latest version of Apache Ambari and HDP, and then use Ambari to add HDF services to the upgraded HDP cluster.

  1. Upgrade Ambari
  2. Upgrade HDP
  3. Install Databases
  4. Install HDF Management Pack
  5. Update HDF Base URL
  6. Add HDF Services to HDP cluster

Installing HDF Services on an Existing HDP Cluster


아마, 많은 경우는 기존에 HDP에 HDF올리는 방법이 생각됩니다.

1. upgrade Ambari -- 최신버전 Ambari로 업그레이드 하라는 내용입니다

2. Upgrade HDP -- 이건 HDP 2.6 쓰면 3버전으로 올리나는 내용입니다 --> 이미 HDP3 이니 패스 

3. HDF에서 사용할 메타 DB에 내용을 추가하라는 내용입니다. 선택지는 oracle,postgres,mysql 이 있지만 일단 저는 postgres


https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/installing-hdf-and-hdp/content/installing_databases.html


해당 페이지에 있는 내용 그대로 postgres에 쿼리 몇줄만 쓰면됩니다. 거의다 create database, table 이런거라 기존 ambari가 사용하는 테이블은 안건드리니 하셔도 될것 같습니다 



  • Configure Postgres to Allow Remote Connections
    It is critical that you configure Postgres to allow remote connections before you deploy a cluster. If you do not perform these steps in advance of installing your cluster, the installation fails.
아마 이부분은 확인은 안해봤는데, ambari 깔릴때 사용하는 postgres를 하면 이미 되어있거나 그럴것 같네요
  1. Log in to Postgres:
    sudo su postgres
psql
  2. Create a database called registry with the password registry:
    create database registry;
CREATE USER registry WITH PASSWORD 'registry';
GRANT ALL PRIVILEGES ON DATABASE "registry" to registry;
  3. Create a database called streamline with the password streamline:
    create database streamline;
CREATE USER streamline WITH PASSWORD 'streamline';
GRANT ALL PRIVILEGES ON DATABASE "streamline" to streamline;
  • Configure Druid and Superset Metadata Stores in Postgres
    Druid and Superset require a relational data store to store metadata. To use Postgres for this, install Postgres and create a database for the Druid metastore. If you have already created a data store using MySQL, you do not need to configure additional metadata stores in Postgres.

  1. Log in to Postgres:
    sudo su postgres
psql
  2. Create a database, user, and password, each called druid, and assign database privileges to the user druid:
    create database druid;
CREATE USER druid WITH PASSWORD 'druid';
GRANT ALL PRIVILEGES ON DATABASE  "druid" to druid;
  3. Create a database, user, and password, each called superset, and assign database privileges to the user superset:
    create database superset;
CREATE USER superset WITH PASSWORD 'superset';
GRANT ALL PRIVILEGES ON DATABASE "superset" to superset;Install HDF Management Pack



4. Install HDF Management Pack

Ambari 에 HDF 스택을 추가하는것입니다 



  1. Download the Hortonworks HDF management pack. You can find the download location for your operating system in the HDF Release Notes.
  2. Copy the bundle to /tmp on the node where you installed Ambari.
  3. Install the management pack:
    ambari-server install-mpack \
--mpack=/tmp/hdf-ambari-mpack-<version>.tar.gz \
--verbose
  4. Restart the Ambari server:
    ambari-server restart


https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_release-notes/content/ch_hdf_relnotes.html#repo-location


릴리즈 노트를 보시면 해당파일을 구하실수 있습니다 


Table 1.5. RHEL/Oracle Linux/CentOS 6 HDF repository & additional download locations

OSFormatDownload location
RHEL/Oracle Linux/CentOS 6 (64-bit):HDF Build number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.1.2.0
HDF Repohttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.1.2.0/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.1.2.0/HDF-3.1.2.0-centos6-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.1.2.0/HDF-3.1.2.0-centos6-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
MiNiFi C++http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.1.2.0/tars/nifi-minifi-cpp/nifi-minifi-cpp-0.4.0-bin.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.2.0/ambari.repo
HDPhttp://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.5.0/hdp.repo
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos6
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


​

Table 1.6. RHEL/Oracle Linux/CentOS 7 HDF repository & additional download locations

OSFormatDownload location
RHEL/Oracle Linux/CentOS 7 (64-bit):HDF Build Number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.2.0
HDF Repohttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.2.0/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.2.0/HDF-3.1.2.0-centos7-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.2.0/HDF-3.1.2.0-centos7-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
MiNiFi C++ Agenthttp://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.2.0/tars/nifi-minifi-cpp/nifi-minifi-cpp-0.4.0-bin.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/ambari.repo
HDPhttp://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/hdp.repo
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


​

Table 1.7. SLES 11 SP3/SP4 HDF repository & additional download locations

OSFormatDownload location
SUSE Enterprise Linux 11 SP3, SP4HDF Build Number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.1.2.0
Repohttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.1.2.0/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.1.2.0/HDF-3.1.2.0-suse11sp3-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.1.2.0/HDF-3.1.2.0-suse11sp3-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/suse11sp3/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/suse11/2.x/updates/2.6.2.0/ambari.repo
HDPhttp://public-repo-1.hortonworks.com/HDP/suse11sp3/2.x/updates/2.6.5.0/hdp.repo
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/suse11sp3
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


​

Table 1.8. SLES 12 HDF repository & additional download locations

OSFormatDownload location
SUSE Linux Enterprise Server (SLES) v12 SP1HDF Build Number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.1.2.0
Repohttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.1.2.0/hdf.repo
RPM tarballhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.1.2.0/HDF-3.1.2.0-sles12-rpm.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.1.2.0/HDF-3.1.2.0-sles12-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/sles12/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/sles12/2.x/updates/2.6.2.0/ambari.repo
HDPhttp://public-repo-1.hortonworks.com/HDP/sles12/2.x/updates/2.6.5.0/hdp.repo
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/sles12
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


​

Table 1.9. Ubuntu 14 HDF repository & additional download locations

OSFormatDownload location
Ubuntu Trusty (14.04) (64-bit)HDF Build Number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.1.2.0
Repohttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.1.2.0/hdf.list
Deb tarballhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.1.2.0/HDF-3.1.2.0-ubuntu14-deb.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.1.2.0/HDF-3.1.2.0-ubuntu14-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/ubuntu14/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.6.2.0/ambari.list
HDPhttp://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.6.5.0/hdp.list
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/ubuntu14
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


​

Table 1.10. Ubuntu 16 HDF repository & additional download locations

OSFormatDownload location
Ubuntu Trusty 16 (64-bit)HDF Build Number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/ubuntu16/3.x/updates/3.1.2.0
Repohttp://public-repo-1.hortonworks.com/HDF/ubuntu16/3.x/updates/3.1.2.0/hdf.list
Deb tarballhttp://public-repo-1.hortonworks.com/HDF/ubuntu16/3.x/updates/3.1.2.0/HDF-3.1.2.0-ubuntu16-deb.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/ubuntu16/3.x/updates/3.1.2.0/HDF-3.1.2.0-ubuntu16-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/ubuntu16/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
MiNiFi C++ Agenthttp://public-repo-1.hortonworks.com/HDF/ubuntu16/3.x/updates/3.1.2.0/tars/nifi-minifi-cpp/nifi-minifi-cpp-0.4.0-bin.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.6.2.0/ambari.list
HDPhttp://public-repo-1.hortonworks.com/HDP/ubuntu16/2.x/updates/2.6.5.0/hdp.list
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/ubuntu16
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


​

Table 1.11. Debian 7 HDF repository & additional download locations

OSFormatDownload location
Debian 7HDF Build Number3.1.2.0-7
HDF Base URLhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.1.2.0
Repohttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.1.2.0/hdf.list
Deb tarballhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.1.2.0/HDF-3.1.2.0-debian7-deb.tar.gz
Tars tarballhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.1.2.0/HDF-3.1.2.0-debian7-tars-tarball.tar.gz
HDF Management Packhttp://public-repo-1.hortonworks.com/HDF/debian7/3.x/updates/3.1.2.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.1.2.0-7.tar.gz
HDP and Ambari Repositories
Ambarihttp://public-repo-1.hortonworks.com/ambari/debian7/2.x/updates/2.6.2.0/ambari.list
HDPhttp://public-repo-1.hortonworks.com/HDP/debian7/2.x/updates/2.6.5.0/hdp.list
HDP-UTILS Base URLhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/debian7
OS Agnostic Downloads
NiFi only
NiFi Toolkit
Docker Hub

https://hub.docker.com/r/hortonworks/nifi/

Tags are latest and 3.1.2.0

NiFi Registry
MiNiFi Java Agent
MiNiFi Toolkit
iOS/Android Libraries


이렇게 하시고 ambari를 재시작하시면 HDF가 추가되어 있습니다 



Falcon 도 HDP3 되면서 없어지면


https://hortonworks.com/products/data-services/


DataPlane Service로 플랫폼 DPS?가 된것 같더군요. 아마도 이것도 이런식으로 추가해서 사용하면 되지 않을까.. 버전이 바뀔때마다 휙휙 바뀌니 손에 익어서 HDP는 계속 사용하고 있지만..


저작자표시 비영리 변경금지

'Study > Bigdata' 카테고리의 다른 글

HDP3 에서 Spark 로 Hive Table 를 조회했는데 빈값이 나온경우  (0) 2018.10.03
HDP3 spark, pyspark, zepplin에서 database가 안보일때,  (2) 2018.09.19
HDP3 제플린(Zepplin) 스케쥴(Cron) 활성화  (0) 2018.09.04
Spark(Yarn) + Intellj 원격 디버깅 하기  (0) 2018.08.21
intellj, Spark Assembly  (0) 2018.08.17
Hive Metastore not working - Syntax error 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1  (0) 2018.08.02

'Study/Bigdata' Related Articles

티스토리툴바