diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..61e4efa --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ + +# Created by https://www.gitignore.io/api/macos +# Edit at https://www.gitignore.io/?templates=macos + +### macOS ### +# General +.DS_Store +.AppleDouble +.LSOverride + +# Icon must end with two \r +Icon + +# Thumbnails +._* + +# Files that might appear in the root of a volume +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent + +# Directories potentially created on remote AFP share +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk + +# End of https://www.gitignore.io/api/macos diff --git a/Labs/Accelerate_Database_Development_and_Testing_with_Amazon_Aurora.pdf b/Labs/Accelerate_Database_Development_and_Testing_with_Amazon_Aurora.pdf new file mode 100644 index 0000000..eba8430 Binary files /dev/null and b/Labs/Accelerate_Database_Development_and_Testing_with_Amazon_Aurora.pdf differ diff --git a/Labs/Building_Your_First_Graph_Application_with_Amazon_Neptune.md b/Labs/Building_Your_First_Graph_Application_with_Amazon_Neptune.md index 536b6b1..835f38d 100644 --- a/Labs/Building_Your_First_Graph_Application_with_Amazon_Neptune.md +++ b/Labs/Building_Your_First_Graph_Application_with_Amazon_Neptune.md @@ -1,3 +1,5 @@ +# Build Your First Graph Application with Amazon Neptune + ## Overview The following lab will walk you through an introduction to using the Gremlin graph traversal language, part of the Apache Tinkerpop graph framework. Amazon Neptune supports the Tinkerpop framework and Gremlin traversal language to allow users to create property graphs and graph-based applications. In the lab, you will use a sample of the open IMDB.com movie database stored in a graph data model within Neptune to answer specific questions. This lab starts off with two Jupyter Notebooks to teach you the basics of Gremiln. It ends with a Jupyter Notebook containing a set of challenges where you have to write Gremlin traversals on your own. @@ -35,9 +37,9 @@ We will be using Amazon SageMaker and AWS CloudFormation to set up infrastructur * The last output listed is for a SageMaker Notebook Instance. This contains a set of Jupyter Notebooks that we will use for the remainder of this lab. Click on the link for the SageMakerNotebook output. This will open the Jupyter Notebook console. * Within the Jupyter Notebook console, navigate the directory tree to the Neptune/notebooks folder. You will see the four Jupyter Notebooks. Start with the 01-Setup notebook and follow the instructions carefully. -**Note**: Make sure to delete the cloud formation stack after the workshop to delete all associated resources. Go to AWS Console > Cloud Formation in the same region that you created the stack in and delete the stack. +**Note**: Make sure to delete the CloudFormation stack after the workshop to delete all associated resources. Go to AWS Console > CloudFormation in the same region that you created the stack in and delete the stack. ## Web Application (Optional) -We have put together following sample web application that demonstrates how to setup to Neptune, issue gremlin queries and build your application. Please refer to the README for further instructions to setup the application: [neptune-workshop-ui](https://github.com/triggan/neptune-workshop-ui) +We have put together the following sample web application that demonstrates how to build a frontend application that consumes data from Neptune. Please refer to the README for further instructions to setup the application: [neptune-workshop-ui](https://github.com/triggan/neptune-workshop-ui) diff --git a/Labs/EC2LinuxHandsOnLab.pdf b/Labs/EC2LinuxHandsOnLab.pdf new file mode 100644 index 0000000..be372b3 Binary files /dev/null and b/Labs/EC2LinuxHandsOnLab.pdf differ diff --git a/Labs/RDS_HOL.pdf b/Labs/RDS_HOL.pdf index 88d4593..7005e82 100644 Binary files a/Labs/RDS_HOL.pdf and b/Labs/RDS_HOL.pdf differ diff --git a/Labs/Reinvent2018-DAT308.zip b/Labs/Reinvent2018-DAT308.zip index 5674ac8..cb4ad60 100644 Binary files a/Labs/Reinvent2018-DAT308.zip and b/Labs/Reinvent2018-DAT308.zip differ diff --git a/Labs/ReplicateManagedDBServerlessTech.zip b/Labs/ReplicateManagedDBServerlessTech.zip new file mode 100644 index 0000000..92e67a1 Binary files /dev/null and b/Labs/ReplicateManagedDBServerlessTech.zip differ diff --git a/Labs/rds_hol_consolidated.docx b/Labs/rds_hol_consolidated.docx new file mode 100644 index 0000000..a5542c9 Binary files /dev/null and b/Labs/rds_hol_consolidated.docx differ diff --git a/Labs/rds_hol_consolidated.pdf b/Labs/rds_hol_consolidated.pdf new file mode 100644 index 0000000..dffe414 Binary files /dev/null and b/Labs/rds_hol_consolidated.pdf differ diff --git a/Labs/reinvent-2018-dms-workshop-template.txt b/Labs/reinvent-2018-dms-workshop-template.txt index 3698e4f..27c7702 100644 --- a/Labs/reinvent-2018-dms-workshop-template.txt +++ b/Labs/reinvent-2018-dms-workshop-template.txt @@ -168,9 +168,9 @@ "DMSInstanceVersion": { "Description": "DMS Engine Version to use", "Type": "String", - "Default": "2.4.3", + "Default": "2.4.4", "AllowedValues": [ - "2.4.3", + "2.4.5", "2.4.4" ], "ConstraintDescription": "must be a valid DMS Engine Version." diff --git a/Labs/running-SQL-Server-on-Amazon-RDS-and-Migrating-to-MySQL.md b/Labs/running-SQL-Server-on-Amazon-RDS-and-Migrating-to-MySQL.md index cffc0e0..4096fd2 100644 --- a/Labs/running-SQL-Server-on-Amazon-RDS-and-Migrating-to-MySQL.md +++ b/Labs/running-SQL-Server-on-Amazon-RDS-and-Migrating-to-MySQL.md @@ -18,49 +18,60 @@ Download or otherwise save the .pem file [Download/Copy the Cloud Formation Template](https://github.com/wrbaldwin/db-week/blob/master/Labs/reinvent-2018-dms-workshop-template.txt) ## Launching your assets -Open the Cloud Formation Service +In the AWS Console, Open the Cloud Formation Service Choose: Create Stack Under choose a template select "upload a template" click "choose file" choose the file you just created ---> Supply a stack name ---> Source Database -Under SourceDBInstanceType - pick one (it's helpful if not everyone selects the default) -Under SourceDBEngineType pick sqlserver-se +1. Supply a stack name like DMS-SQLServer-to-MYSQL +2. Under Source RDS Database *(note the template supports Oracle to Postgres default or SQL Server to MySQL be sure to pick the appropriate drop downs for the version of the lab you are running or you will have to start over!) ---> Target Database -Under AuroraInstanceType - pick one ( bigger = more $$ but faster. It's helpful if not everyone selects the default) -Under AuroraEngineType - pick aurora-mysql + Leave defaults for parameters except: + Under SourceDBInstanceType - pick one (it's helpful if not everyone selects the default) + + **Under SourceDBEngineType pick SQLSERVER-SE _(Pick the right one to match your lab. default is Oracle)_** + +3. Target Database ---> Schema Conversion Tool Host Configuration -Under SCTHostKeyName - pick a key pair you have access to. This is required to unlock the password of the ec2 instance launched. -Under SCTHostInstanceType - pick one (again it's helpful if not everyone accepts the default) + Leave defaults for parameters except: + Under AuroraInstanceType - pick one ( It's helpful if not everyone selects the default) + + **Under AuroraEngineType - pick "aurora-mysql" or just "Aurora" _(Pick the right one to match your lab. default is Auora-Postgres)_** ---> Other Parameters -LaptopIPAddress - you can leave this wide open or you can supply your address to limit access. +4. Schema Conversion Tool Host Configuration -Click "Next" + Under SCTHostKeyName - pick a key pair you have access to. This is required to unlock the password of the ec2 instance launched. + Under SCTHostInstanceType - pick one (again it's helpful if not everyone accepts the default) + +5. Other Parameters + LaptopIPAddress - you can leave this wide open or you can supply your address to limit access. You can use http://checkip.amazonaws.com/ to find your laptop ip address + +Click "Next" button + +You can add a tag to your resources or not :smile: ... -You can tag your resources or just... Click "Next" -Click "Create" +check the acknowledgement box and +**Click "Create"** +You have time to go grab a cup of coffee:coffee: or whatever as this will take approximately 15 minutes to finish --> Once your stack is created you can begin the migration lab NOTE: Important information about your assets will be listed in your Cloud Formation "Outputs" listing for your stack +We suggest you copy this to a text editor so you can easily go back and forth. ## Logging into your SCT windows server Log into the ec2 console and find your instance. The name should include your cloudwatch stack name. ----> Get the password -Select your instance -Click "Connect" (button on top) -Click "Get Password" + +Select your instance +Click "Connect" (button on top) +Click "Get Password" Supply the full contents of the .pem file you created earlier or the one you will use for this class. -Click "Decrypt Password" -Copy and save the password. -Close the dialog. +Click "Decrypt Password" +Copy and save the password. +Close the dialog. Launch Remote Desktop (or a similar tool to connect to your windows server) Find the "public DNS" for your server (log into this) @@ -72,10 +83,12 @@ Click "YES" to log in despite certificate errors. ## Installing Drivers Unfortunately for legal reasons we are not allowed to pre-install the drivers for you. -You will need to install bot SQL Server and MySQL drivers in order to connect to your source and target databases. +Only install the drivers needed for your lab though the document lists several more. + +You will need to install both SQL Server and MySQL drivers in order to connect to your source and target databases. Instructions are on the desktop in the file "JDBC Driver Install Instructions" -Follow the instructions and install the appropriate drivers. +Follow the instructions and install the appropriate drivers you need. ## Launch the SCT and connect to your source and target @@ -111,11 +124,13 @@ Click "Next" Click "Next" --> Choose your target database + Server Name: (from CF Outputs) -Server port: 3306 -User Name: auradmin -Password: auroradmin123 -MySQL driver path: C:\JDBC Drivers\MySQL\mysql-connector-java-8.0.13.jar + Server port: 3306 + User Name: auradmin + Password: auroradmin123 + MySQL driver path: C:\JDBC Drivers\MySQL\mysql-connector-java-8.0.13.jar + Click: "Test connection" (lower left corner of dialog) @@ -124,11 +139,13 @@ Click "Finish" ...At this point you can poke around a bit. You'll notice there are some errors mostly with code objects. This is intentional to show how the tool will highlight issues that need attention. + ## Generate your target MySQL database In the left hand panel right click on the dms_sample.dbo schema. From the menu select "Convert schema" In the right hand panel find and right click on the dms_sample_dbo schema. + From the menu select "Apply to database" You have now used the SCT to convert your SQL Server schema to an Aurora MySQL schema! Woohoo!!! @@ -136,10 +153,11 @@ You have now used the SCT to convert your SQL Server schema to an Aurora MySQL s ## Next - we'll use the Database Migration Service to migrate the data from your SQL Server instance to your Aurora MySQL instance Log into the AWS Console and go to the Database Migration Service ---> Create your source endpoint -Click "Endpoints" on the left hand side -Click "Create endpoint" (button up top-left) -Endpoint type: Select "source" +### --> Create your source endpoint +Click "Endpoints" on the left hand side +Click "Create endpoint" (button up top-left) + +Endpoint type: Select "source" check "select RDS DB Instance" choose your source SQL Server instance from the RDS Instance drop down Source engine: sqlserver @@ -149,17 +167,21 @@ SSL mode: none user name: dbmaster password: srcadmin123 (from CF Ourputs) database name: dms_sample + vpc: select the vpc that contains your CF stack name replication instance: choose the instance that contains your CF stack name click "Run Test" + If the test is successful click "Save", otherwise fix any issues and/or ask for help ---> Create your target endpoint +### --> Create your target endpoint Click "Endpoints" on the left hand side Click "Create endpoint" (button up top-left) + Endpoint type: Select "target" check "select RDS DB Instance" -choose your target Aurora MySQL instance from the RDS Instance drop down +choose your target Aurora MySQL instance from the RDS Instance drop down if your instance doesn't show +then add your instance details manually. Target engine: aurora Server name: < should be supplied - verify if you want> port: 3306 @@ -167,39 +189,49 @@ SSL mode: none User name: auradmin Password: auroradmin123 (from CF Ourputs) -!!! Open the advanced section and turn off foreign key checks for the Aurora target !!! -Extra connection attributes: initstmt=SET FOREIGN_KEY_CHECKS=0 + **Open the advanced section or Endpoint-specific settings by clicking on it and turn off foreign key checks + for the Aurora target by putting !!!** + **Extra connection attributes to add to input box: _initstmt=SET FOREIGN_KEY_CHECKS=0_** vpc: select the vpc that contains your CF stack name replication instance: choose the instance that contains your CF stack name click "Run Test" + If the test is successful click "Save", otherwise fix any issues and/or ask for help ---> Create your task and migrate your data! +### --> Create your task and migrate your data! Select "Tasks" on the left hand side of the console Click "Create task" (button up top-left) Task name: reinvent-2018-dms-workshop (or whatever you want) + Replication instance: select the on that includes your CF stack name + Source endpoint: select the one you just created + Target endpoint: select the one you just created + Migration type: select migrate existing data and replicate ongoing changes ...keep defaults for the rest and scroll down the Table mappings... -From the "Guided" tab add some selection rules: +#### From the "Guided" tab add some selection rules: Schema name is: dbo Table name is like: % Action: include Click "Add selection rule" -Click "Create task" (button bottom-right) +##Click **"Create task" (button bottom-right)** +Monitor your job by selecting table statistics and see the row counts populate as tables complete. This will take a few minutes to run -## Tear Down -When you are done, you'll want to tear down your environment. To do so: -STOP your task! (this is important to ensure the replication instance is destroyed and you aren't charged $$$$$$) +# Tear Down +When you are done, you'll want to tear down your environment. To do so: + +STOP your task! _(this is important to ensure the replication instance is destroyed and you aren't charged $$$$$$) Once it's stopped... -Go to your Cloud Formation console -Select your CF Stack + +Go to your Cloud Formation in AWS console +Select your CF Stack From the actions drop down select "delete stack" +the deletion will take a few minutes diff --git a/README.md b/README.md index 22917bb..32bdecd 100644 --- a/README.md +++ b/README.md @@ -3,23 +3,22 @@ Database Week at the AWS Loft is an opportunity to learn about Amazon’s broad and deep family of managed database services. These services provide easy, scalable, reliable, and cost-effective ways to manage your data in the cloud. ## Current Labs -[Getting Started with Linux on Amazon EC2](https://github.com/wrbaldwin/db-week/blob/master/Labs/getting-started-with-ec2-linux.pdf) - This lab will walk you through launching, configuring, and customizing an EC2 virtual machine to run a web server. It will walk you though successfully provisioning and starting an EC2 instance using the AWS Management Console. +[Getting Started with Linux on Amazon EC2](https://github.com/wrbaldwin/db-week/blob/master/Labs/EC2LinuxHandsOnLab.pdf) - This lab will walk you through launching, configuring, and customizing an EC2 virtual machine to run a web server. It will walk you though successfully provisioning and starting an EC2 instance using the AWS Management Console. [Running SQL Server on Amazon RDS and Migrating to MySQL](https://github.com/wrbaldwin/db-week/blob/master/Labs/running-SQL-Server-on-Amazon-RDS-and-Migrating-to-MySQL.md) - If you'd like to move your SQL Server databases to the cloud, this workshop is for you. We review the basics of Amazon Relational Database Service (Amazon RDS) and how SQL Server databases run in Amazon RDS. We then leverage the combination of AWS Schema Conversion Tool (AWS SCT) and AWS Database Migration Service (AWS DMS), and we show you how to migrate your databases to Amazon Aurora MySQL. We provide an AWS CloudFormation template to set up the entire environment for the lab. You need a laptop with a Firefox or Chrome browser and a working AWS account. -[Getting Started with Amazon RDS](https://github.com/wrbaldwin/db-week/blob/master/Labs/RDS_HOL.pdf) - Amazon RDS is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. +[Getting Started with Amazon RDS](https://github.com/wrbaldwin/db-week/blob/master/Labs/rds_hol_consolidated.pdf) - Amazon RDS is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. -[Replicate, Analyze, and Visualize Datasets Using AWS Database Migration Service and Serverless Big Data Technologies](https://github.com/wrbaldwin/db-week/blob/master/Labs/Reinvent2018-DAT308.zip) - In this workshop, you will use AWS serverless technologies to replicate, analyze, and visualize data from relational and semi-structured datasets to gain insight and drive business outcomes. You will leverage AWS Schema Conversion Tool (SCT) and AWS Database Migration Service (DMS) to convert MS SQL server database to Amazon Aurora Serverless MySQL, use AWS Glue to build data catalog & transform data, use Amazon Athena to run ad-hoc SQL queries and interactively analyze data stored on Amazon S3 data store, and use Amazon QuickSight to visualize data and drive additional insights. - -[Accelerating Application Development with Amazon Aurora](https://github.com/wrbaldwin/db-week/blob/master/Labs/accelerating-application-development-with-Amazon-Aurora.md) - Learn how to leverage the unique features of the Amazon Aurora platform to build faster, more scalable database applications optimized for the cloud. We discuss architectural best practices and features, such as Aurora Serverless, Read Replica Auto-Scaling, Cross-Region Replicas, Backtrack, Fast Database Cloning, and Performance Insights. They are designed to help increase agility so you can develop applications faster to reach the widest possible audience. Through a hands-on lab, we help you understand how to best take advantage of the capabilities of the Aurora platform to effectively accelerate application development. +[Accelerate Database Development and Testing with Amazon Aurora](https://github.com/wrbaldwin/db-week/blob/master/Labs/Accelerate_Database_Development_and_Testing_with_Amazon_Aurora.pdf) - Learn how to leverage the unique features of Amazon Aurora to build faster and more scalable database applications optimized for the cloud. In this lab, you will leverage AWS CloudFormation to provision an Aurora MySQL compatible database cluster, along with a Linux EC2 instance to be used as a workstation. You will then load data from S3 into Aurora MySQL database; run a read only workload and see Aurora Auto Scaling in action. You will practice Aurora database Cloning and Backtracking. You will use AWS Systems Manager command document to execute a load test and view the load using RDS Performance Insights. Through this hands-on lab, we help you understand how to best take advantage of the capabilities of the Amazon Aurora to effectively accelerate application development. [Search Dynamo DB Data with Amazon Elasticsearch Service](https://github.com/wrbaldwin/db-week/blob/master/Labs/Search_DDB_Data_with_AES.pdf) - In this lab, you’ll explore using Amazon Elasticsearch Service (Amazon ES) to augment the more fundamental query capabilities of Amazon Dynamo DB (Dynamo DB). You’ll deploy a CloudFormation template that generates a Dynamo DB table and loads movie data into that table. You’ll employ Dynamo Streams to replicate the initial data load to an Amazon Elasticsearch Service domain. You’ll enable Cognito authentication to provide simple access to Kibana so that you can use Kibana’s UI to send searches to Amazon ES. Finally, you’ll run a Lambda function to generate updates to the Dynamo DB table, see those updates replicated to Amazon ES, and build Kibana visualizations for the changes. - [Building Your First Graph Application with Amazon Neptune](https://github.com/wrbaldwin/db-week/blob/master/Labs/Building_Your_First_Graph_Application_with_Amazon_Neptune.md) - In this session, get hands on with Amazon Neptune and build a cloud-based graph application. Learn how to quickly load data and begin writing Gremlin traversals. ## Old Labs +[Replicate, Analyze, and Visualize Datasets Using AWS Database Migration Service and Serverless Big Data Technologies](https://github.com/wrbaldwin/db-week/blob/master/Labs/ReplicateManagedDBServerlessTech.zip) - In this workshop, you will use AWS serverless technologies to replicate, analyze, and visualize data from relational and semi-structured datasets to gain insight and drive business outcomes. You will leverage AWS Schema Conversion Tool (SCT) and AWS Database Migration Service (DMS) to convert MS SQL server database to Amazon Aurora Serverless MySQL, use AWS Glue to build data catalog & transform data, use Amazon Athena to run ad-hoc SQL queries and interactively analyze data stored on Amazon S3 data store, and use Amazon QuickSight to visualize data and drive additional insights. + [Managed Database Basics](https://github.com/wrbaldwin/db-week/blob/master/Labs/managed-database-basics.md) - In this lab you will learn, how to create an EC2 instance for use as a SQL client that connects to an RDS/MySQL instance. Next, you will create a database, load a public Landsat dataset into it, and query your database using SQL. Finally, you will learn how to use the RDS console including stopping, starting, monitoring, snapshots and other RDS features. [Aurora MySQL](https://github.com/wrbaldwin/db-week/blob/master/Labs/Aurora%20MySQL%20Hands%20On%20Lab%20Manual%202.1.pdf) - In this lab you will create Aurora MySQL DB Cluster, modify security group to allow access to the Aurora MySQL DB instance from your computer, load data from S3 into Aurora MySQL database, create read replica instance & access table, create a database copy using Aurora “Clone” feature, and perform DML query on primary DB and validate data on primary and cloned DB copy.