[nom]
Modernization with AWS Glue
Leonardo Gomez He/Him
Senior Analytics Specialists
AWS
2022, Amazon Web Services, Inc. or its affiliates. · 2022, Amazon Web Services, Inc. or its affiliates.
Make better
Put data
decisions
to work
Respond
faster
2022, Amazon Web Services, Inc. or its affiliates.
Improve
efficiencies
Uncover
opportunities
Customers want more value from their data
Growing · From new
exponentially · sources
Increasingly · Used by · Analyzed by many
diverse · many people · applications
2022, Amazon Web Services, Inc. or its affiliates.
More demanding workloads
Batch · Real-time
Nightly or hourly [nom] · Continuous operation
Latency-agnostic, long-running jobs · Latency-sensitive micro-batch jobs
2022, Amazon Web Services, Inc. or its affiliates.
Traditional [nom] isn’t scalable
Non-scalable
infrastructure
Costly
Lock-in
Lacks data
integration tools
for all users
On-premises [nom] tools are complex to install, manage, [nom] scale
Advanced features - like a centralized data catalog or managing streaming data - need to be licensed separately. Connectors are highly priced [nom] lack usage to justify pricing
Jobs written in traditional [nom] tools are proprietary to a specific vendor [nom] cannot be ported
[nom] is no longer a data engineering function. Business Analysts, Data Analysts, [nom] Data
Scientists want self-service features. Traditional [nom] tools don’t cater to all users
2022, Amazon Web Services, Inc. or its affiliates.
Introducing AWS Glue
2022, Amazon Web Services, Inc. or its affiliates.
AWS Glue
DISCOVER, PREPARE, [nom] INTEGRATE
ALL YOUR DATA AT ANY SCALE
2022, Amazon Web Services, Inc. or its affiliates.
All-in-one data integration service
Cost effective, serverless, [nom]
scalable
Tailored tools to support all data users
Support all workloads in one place
Scalable Data
Integration Engine
Built-in data transforms
Execution engine
Monitor
AWS Glue: Key Capabilities
SERVERLESS DATA INTEGRATION SERVICE
Centralized [nom] Unified · Connect [nom] · User Productivity
Data Governance · Ingest Data · [nom] Data Ops
Glue data catalog · Glue connectors · Persona specific tools
Glue crawlers · Glue connector marketplace · Productivity tools
Lake Formation · Variety of interfaces · Data ops tools
2022, Amazon Web Services, Inc. or its affiliates.
How
customers
use
AWS Glue
Execute modern data strategies (e.g. data lakes, data fabric) for scalable data analysis
Migrate from expensive traditional [nom] solutions to gain flexibility [nom] reduce costs
Catalog data assets to make them available across a modern data architecture
Process petabytes of data in batch, streaming,
[nom] real-time using Python [nom] Apache Spark
Prepare data for analytics [nom] Machine Learning
2022, Amazon Web Services, Inc. or its affiliates.
Persona based experience
2022, Amazon Web Services, Inc. or its affiliates. · 10
[nom] developer
AWS Glue Studio
Rich visual interface
250+ built-in transformations
Profile data to understand data patterns [nom] anomalies
Work on large datasets at scale
Business Analyst
Data scientist · Data engineer
AWS Glue Interactive
AWS Glue DataBrew · sessions
Run [nom] jobs without writing code · Clean [nom] normalize data with a
rich visual interface
Monitor thousands of jobs
through a [information personnelle] pane of glass · Choose from 250+ built-in
transformations to automate tasks
Distributed processing
Profile data to understand data patterns [nom] anomalies
Advanced transforms
though code snippets
2022, Amazon Web Services, Inc. or its affiliates.
Demo
2022, Amazon Web Services, Inc. or its affiliates. · 12
Variety of workloads
2022, Amazon Web Services, Inc. or its affiliates. · 13
AWS Glue handles variety of workloads
Streams Processing · Batch Processing
Easily integrate with · Combine, enrich,
Amazon MSK, · [nom] transform data
Amazon Kinesis, [nom] · in using
Apache Kafka · Glue Studio
2020, Amazon Web Services, Inc. or its Affiliates.
Event Processing
Adapt to varied
[nom] changing
schemas
Flexible
2022, Amazon Web Services, Inc. or its affiliates. · 15
Predicting capacity is difficult
Fluctuation in data · Needs tuning
Time of day · Admin inputs
Seasonality · Monitoring
Data skew · Cost controls
2022, Amazon Web Services, Inc. or its affiliates.
Provisioning
Over-provisioning
Extra cost
Performance variation
Auto-scaling
Start · AWS Glue job
Cost
Wide transform
Uneven distribution of data partitions
t1 · t2 · t3 · t4 · t5 · t6 · t7 · t8 · t9
List operation · Without Autoscaling · With Autoscaling
Job execution timeline
2022, Amazon Web Services, Inc. or its affiliates.
End
Potential
savings
t10 · t11
Cost Options
2022, Amazon Web Services, Inc. or its affiliates. · 18
Glue Flex
New execution option for AWS Glue that allows customers to reduce the costs by up to 35%
Standard
execution-class
10x faster job start times
Predictable job latencies
Enables micro-batching
Latency-sensitive workloads
Flex
execution-class
Up to 35% cost savings
Cost effective for non-time sensitive workloads
2022, Amazon Web Services, Inc. or its affiliates.
Smaller instance types reduces your streaming cost by 4x
Low volume data · Medium to Large data stream
processing · processing
NEW
G 0.25X · G 1X
(2 vCPU, 4 GB) · (4 vCPU, 16 GB)
$0.11 per hour · $0.44 per hour
Change data capture · Web log processing for
streams to implement near · fraud detection
real time data warehouse
2022, Amazon Web Services, Inc. or its affiliates.
Extremely large scale data stream processing
G 2X
(8 vCPU, 32 GB)
$0.88 per hour
IoT stream processing to detect complex events
Open Source Based
2022, Amazon Web Services, Inc. or its affiliates. · 21
2022, Amazon Web Services, Inc. or its affiliates. · 22
Integrated
2022, Amazon Web Services, Inc. or its affiliates. · 23
Connectors with AWS Glue
Built-in Connectors
Out of box connectors to support high performance ingestion
Custom Connectors
Flexible method to build your own connectors
Marketplace Connectors
Subscription based low cost connectors
2020, Amazon Web Services, Inc. or its Affiliates. · 100 + connectors
Data privacy
2022, Amazon Web Services, Inc. or its affiliates. · 25
AWS Glue sensitive data detection [nom] remediation
THREE SIMPLE STEPS
1
Type of Scan
Full Scan
Sample Scan
2 · 3
Entities to detect · Remediation
Built-in Entities
Store results
[pièce d'identité]
Redact/mask
Custom Entities · results
2022, Amazon Web Services, Inc. or its affiliates.
How does PII Detection work
Full Scan · Sample Scan
Entities to detect
Remediation
2022, Amazon Web Services, Inc. or its affiliates.
Demo
2022, Amazon Web Services, Inc. or its affiliates. · 28
Operationalize
2022, Amazon Web Services, Inc. or its affiliates. · 29
Operationalize [nom] execute data pipelines at-scale
Glue Workflows
Orchestrate Glue jobs by schedule, on demand, or by event
Glue APIs · Glue Monitor
Programmatic control · Monitor
jobs easily
2022, Amazon Web Services, Inc. or its affiliates.
Orchestrate jobs easily with AWS Glue workflows
Orchestrate Glue jobs [nom] other AWS services
Schedule jobs or trigger based on events
Monitor execution of the workflows in one place
2022, Amazon Web Services, Inc. or its affiliates.
Monitoring dashboard to check job status
2022, Amazon Web Services, Inc. or its affiliates.
[nom] Modernization
2022, Amazon Web Services, Inc. or its affiliates. · 33
[nom] Modernization Program
Goals
- Modernize legacy [nom] workloads
- Reduce data integration costs by migrating to AWS Glue
- Accelerate migration
Benefits
- No-cost AWS partner delivered assessment [nom] Proof-of- Concept (POC) for up to 5 [nom] jobs using automated tools*
- No-cost [nom] workload analysis [nom] Total Cost of Ownership (TCO)
- High level architecture [nom] roadmap for migration
- Funding options to offset migration to AWS
- Support through the migration
- Automatic code conversion to Glue Studio for Informatica, Talend, SSIS 2022, Amazon Web Services, Inc. or its affiliates.
[nom] Modernization Approach
Pilot
Assess
Scope pilot pipelines for migration
Analyze existing [nom]
[nom] identify patterns
[nom] new capabilities · Build reusable
frameworks for
Train your core data · identified patterns
engineering team
Modernize
Create migration plan for end to end migration
Train data
engineers [nom]
business teams to enable self service
Convert pipelines to
Organize Proof of · AWS Glue for
concept using · limited scope
automation
Assess total cost of
ownership · Deploy to production
Migrate in phases
Create operational
procedures for support teams
2022, Amazon Web Services, Inc. or its affiliates.
35
[nom] Migration Partner Program – Consulting Partners
Workshop · Free* Initial
Immersion · Assessment
Discovery · Paid
TCO Report · Comprehensive
Business Case · Assessment
Discovery · Assessment
Some of the participating SI’s…
Migration Plan
Skills/Team · Tooling ISV’s
Operating Model · Migrate
Security · Operate
Compliance · Modernize
POC
Mobilize · Modernize
2022, Amazon Web Services, Inc. or its affiliates.
Thank you!
Leonardo Gomez
2022, Amazon Web Services, Inc. or its affiliates. · 2022, Amazon Web Services, Inc. or its affiliates.
