How to use Amazon EMR to process data using the broad ecosystem of Hadoop tools like Hive and Hue and work with Amazon DynamoDB, Amazon Redshift, Amazon QuickSight, Amazon Athena and Amazon Kinesis.
Harness the power of your data with AWS Analytics https://aws.amazon.com/blogs/big-data/harness-the-power-of-your-data-with-aws-analytics/
Data Ingestion and Transfer
- Amazon Kinesis Agent for Data Ingestion https://github.com/awslabs/amazon-kinesis-agent
- Apache Flume https://flume.apache.org/ can be installed and run on Amazon EC2 instances.
- You can also use S3DistCp to copy data between Amazon S3 buckets or from HDFS to Amazon S3 across AWS accounts http://docs.aws.amazon.com/emr/latest/ReleaseGuide/UsingEMR_s3distcp.html
- Apache Sqoop https://cwiki.apache.org/confluence/display/SQOOP/Home supports the transfer of data between Hadoop and structured data stores such as Amazon RDS.
- AWS IoT can collect and handle large quantities of data coming from a variety of sources https://aws.amazon.com/iot/ and makes it easy to use AWS services like AWS Lambda, Amazon Kinesis, Amazon S3, Amazon Machine Learning, and Amazon DynamoDB.
- AWS DataSync https://aws.amazon.com/datasync/ is a data transfer service that makes it easy for you to automate moving data between on-premises storage and Amazon S3 or Amazon Elastic File System (Amazon EFS).
- Amazon FSx for Lustre https://aws.amazon.com/fsx/lustre/ provides a high-performance file system optimized for fast processing of workloads such as machine learning, high performance computing (HPC), video processing, financial modeling, and electronic design automation (EDA).
- AWS Glue DataBrew https://aws.amazon.com/glue/features/databrew/ visual data preparation tool to clean and normalize data to prepare it for analytics and machine learning
Big Data Streaming and Amazon Kinesis
- Amazon Kinesis Data Streams resources https://aws.amazon.com/kinesis/data-streams/resources/ Tools and Libraries.
- Overview of Amazon Kinesis Data Firehose https://aws.amazon.com/kinesis/data-firehose/
- AWS Kinesis Data Analytics - SQL Functions https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-functions.html
- Using the Schema Discovery Feature on Streaming Data https://docs.aws.amazon.com/kinesisanalytics/latest/dev/sch-dis.html
- Apache Spark Streaming enables high-throughput, fault-tolerant, and scalable processing of live data streams. It divides the incoming data streams into batches before sending them to the Spark engine for processing. http://spark.apache.org/streaming/
- Amazon Managed Streaming for Kafka (MSK) https://aws.amazon.com/msk/ is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data.
Data Lake and Lake House Architectures
- Build a Lake House Architecture on AWS https://aws.amazon.com/blogs/big-data/build-a-lake-house-architecture-on-aws/
- What is a data lake? https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/
- Building Data Lakes on AWS https://d1.awsstatic.com/whitepapers/Storage/data-lake-on-aws.pdf AWS white paper.
- AWS Lake Formation https://aws.amazon.com/lake-formation/ is a service that makes it easy to set up a secure data lake in days.
- S3 Object Lifecycle Management http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html
- How to set up cross-origin resource sharing (CORS) http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors.html
- Manage fine-grained access control using AWS Lake Formation https://aws.amazon.com/blogs/big-data/manage-fine-grained-access-control-using-aws-lake-formation/
- EMR File System (EMRFS) consistent view https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-consistent-view.html
- Quick Start Data Lake with SnapLogic https://aws.amazon.com/quickstart/architecture/data-lake-with-snaplogic/ builds a data lake environment on AWS in about 15 minutes by deploying SnapLogic components and AWS services such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift.
- AWS Lake Formation Workshop https://lakeformation.aworkshop.io/
Hadoop Frameworks and Deployment Options
- Run Amazon EMR on Amazon EKS https://aws.amazon.com/emr/features/eks/
- Amazon EMR Serverless https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html
- About Amazon EMR Releases https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html Each release comprises different big-data applications, components, and features that you select to have Amazon EMR install and configure when you create a cluster.
- Apache Hive https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive.html
- Differences and Considerations for Hive on Amazon EMR https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-differences.html
- Presto on Amazon EMR https://aws.amazon.com/emr/features/presto/
- Apache Pig https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-pig.html
- PIGgy Bank https://cwiki.apache.org/confluence/display/PIG/PiggyBank is a place for Pig users to share their functions.
- Apache Spark on Amazon EMR https://aws.amazon.com/emr/features/spark/
- Apache MXNet https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-mxnet.html
- How do I restart a service in Amazon EMR? https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/
- Amazon EMR now supports a public EMR artifact repository for Maven builds https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-artifact-repository.html
EMR User Interfaces
- Amazon EMR Studio | Managed IDE Environment https://aws.amazon.com/emr/features/studio/
- View Web Interfaces Hosted on Amazon EMR Clusters https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-web-interfaces.html
- View On-Cluster Application User Interfaces https://docs.aws.amazon.com/emr/latest/ManagementGuide/on-cluster-app-UI.html
- Launching the Hue Web Interface https://docs.aws.amazon.com/emr/latest/ReleaseGuide/accessing-hue.html
- Apache Zeppelin https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-zeppelin.html
Spark
- Spark Or Hadoop: Which Is The Best Big Data Framework? https://www.datasciencecentral.com/profiles/blogs/spark-or-hadoop-which-is-the-best-big-data-framework Blog post from Data Science Central.
- Apache Spark home page: https://spark.apache.org/
- Spark RDD Programming Guide http://spark.apache.org/docs/latest/rdd-programming-guide.html
- Spark Streaming Programming Guide https://spark.apache.org/docs/latest/streaming-programming-guide.html
- Use Apache Spark with Amazon Sagemaker https://docs.aws.amazon.com/sagemaker/latest/dg/apache-spark.html
Management and Monitoring
- Best Practices for Amazon EMR white paper: https://d0.awsstatic.com/whitepapers/aws-amazon-emr-best-practices.pdf
- Monitor Metrics with CloudWatch https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_ViewingMetrics.html
- Ganglia https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-ganglia.html Ganglia Monitoring System http://ganglia.info/
- Using Automatic Scaling in Amazon EMR https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-automatic-scaling.html
- Upload Data to an Amazon ES Domain for Indexing https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-upload-data.html
Visualisation
- The best stats you've ever seen https://www.youtube.com/watch?v=usdJgEwMinM a TED-Ed talk by Hans Rosling in July 2013.
- Amazon QuickSight Features https://aws.amazon.com/quicksight/features/
- Demo QuickSight Dashboards https://democentral.learnquicksight.online/#Dashboard-DashboardDemo-Exec-Business-Summary
- Amazon QuickSight Community hub https://community.amazonquicksight.com/ - the place to ask, answer, and learn with others in the QuickSight community
TECHNICAL BLOGS and PRESENTATIONS
- Pushing Physical Limits with AWS Snowball Edge https://www.youtube.com/watch?v=__ooXhq5gZ4&t=47s
- Harness the power of your data with AWS Analytics https://aws.amazon.com/blogs/big-data/harness-the-power-of-your-data-with-aws-analytics/
- AWS serverless data analytics pipeline reference architecture https://aws.amazon.com/blogs/big-data/aws-serverless-data-analytics-pipeline-reference-architecture
- SQL Based Data Processing in Amazon ECS https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/sql_based_data_processing_amazon_ecs.pdf Build a configuration-driven, codeless extract-transform-load (ETL) alternative using a containerized ETL framework (ARC) https://arc.tripl.ai/ that simplifies and accelerates data processing with Apache Spark.
- This blog post describes how to Implement continuous integration and delivery of serverless AWS Glue ETL applications using AWS Developer Tools https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/
- Using Step Functions to Orchestrate Amazon EMR Workloads https://aws.amazon.com/blogs/aws/new-using-step-functions-to-orchestrate-amazon-emr-workloads/
- Amazon EMR Studio introduction video https://youtu.be/Xv0yhKJQPOc
CUSTOMER REFERENCES
- AWS Case Study: Hearst Corporation https://aws.amazon.com/solutions/case-studies/hearst/ . See also AWS re:Invent 2015 | (BDT306) How Hearst Publishing Manages Clickstream Analytics with AWS https://www.youtube.com/watch?v=6cwbbqi36k8 . The Hearst presentation starts at 18:30.
- Netflix case study https://aws.amazon.com/solutions/case-studies/netflix/
- Snowplow Analytics https://aws.amazon.com/solutions/case-studies/snowplow/ enables its clients to collect granular customer-level and event-level data from multiple platforms (web and mobile).
- AdRoll is a global leader in retargeting, serving 50 billion personal ad impressions every day. https://aws.amazon.com/solutions/case-studies/adroll/ use S3, Kinesis Apache Storm and Amazon DynamoDB.
- Data Xu makes use of Kinesis for data processing, and Athena on Amazon S3 as a single source of truth https://aws.amazon.com/solutions/case-studies/dataxu/
- Channel 4 https://aws.amazon.com/solutions/case-studies/channel-4/ using Amazon EMR enables them to run analyses on 100% of their available data instead of sampling.
Webpage constructed by Dennis Adams
Sr AWS Technical Instructor
